MISTAKE #6: YOU USE INCOMPLETE OR OVERLAPPINGÂ MULTIPLE CHOICE SELECTIONS
I was recently participating in an online Survey when I encountered the following multiple choice question:
On average, approximately how much do you pay in US DOLLARS for a pint of beer at a pub in your neighborhood? (select one)
A) Less than $2
B) $2 – $2.99
C) $3 – $3.99
D) $4 – $4.99
E) $5 – $6
F)Â $6 or more
At first glance, the question seems well-constructed. The units are clearly defined, the grammar and spelling are correct, and the selections are in a logical order. The author had clearly had some formal Survey training – and yet, they made two HUGE mistakes (and one minor one) on this question.
Do you see them?
The first one is a bit subtle, but should be rather obvious considering the title of this post: the response selections overlap. Specifically, selections E and F both include $6. Since this is a multiple choice exclusive (“select one”) question, if the respondent wanted to answer $6 they would be forced to choose between two equally valid options.
Response selections for multiple choice exclusive questions  should be completely unique from one another in order to avoid logical traps – and, consequently, incorrect data – like the one above. As a rule of thumb, remember the word “exclusive”: multiple choice exclusive questions should have mutually exclusive responses.
The second error is far less obvious, but represents a classic example of assumption bias (probably the most common form of bias present in Survey research). If you paid attention to the title, you might be thinking it has something to do with “incomplete” multiple choice question selections, and indeed you would be correct. Response selections should cover all possible scenarios, and yet in this case the author has failed to do so.
“But wait, Wyatt!” you say. “The responses provided do cover all possible values from zero dollars to infinite dollars.” You are correct, they do cover all possible values… and yet, they do not actually cover all possible scenarios. For example,
What if I don’t have any pubs in my neighborhood?!?!
The author has simply assumed that I have at least one neighborhood watering hole, yet it is entirely possible that I do not. In fact, although I currently live in an area (San Francisco) with many pubs, I grew up in a part of the United States with many “dry” towns and counties… and, therefore, no pubs! It is not unreasonable to expect that someone taking a Survey on beer might live in a dry town, just as it is not unreasonable to expect that some people in Canada don’t like the cold and that some New Yorkers are actually very nice people.
You might think I’m splitting hairs, but this is a very real problem with Surveys that often goes unnoticed until it is too late – i.e. when frustrated respondents start complaining or, worse, quit the Survey altogether. Besides, the solution to this is so simple and easy that there is really no reason to ever make this mistake. Drum roll, please…
None of the Above
Memorize these four little words and use them – or some relevant variant thereof – on every multiple choice question. I call this the “catch-all response” – in other words, the extra answer that will catch all of the other people who, for reasons I can not predict, might take exception to the responses I have provided. The almighty catch-all response will save you every time.
So what’s the minor mistake? Well, you’ll just have to wait for Mistake #7 to find out!
