You Think Sample Size Matters – One of the Top 10 Mistakes Survey Writers Make

Editor’s Note: This is the ninth in a series of posts on common Survey writing mistakes. Click here to see the previous item and stay tuned for more!

MISTAKE #9: YOU THINK SAMPLE SIZE MATTERS

Yes, I have just committed Survey heresy. I can already hear the statisticians, data scientists, and demographic pollsters screaming. “What the heck are you talking about?” they holler from on high. “OF COURSE sample size matters! How can you get statistically significant results if you don’t have the right sample size? How do you know your margin of error? What about confidence intervals and second-order relationships and…” And on, and on, and on.

Before they start lighting matches, allow me to clarify. For quantitative opinion polls constructed by professional Survey writers, I agree – sample size is important. Seriously, who would believe a presidential poll of 25 people? (Hehehe…) It’s just that most Surveys are not written by professionals, are not intended for mass distribution, and are so fraught with mistakes that all a bigger sample size will do is yield more bad data.

As my boss at Applied Materials used to say to me early in my career,

It is better to be approximately right than exactly wrong.

In other words, do not waste a lot of time, money, and effort trying to achieve perfection when in most situations what matters most is that you get the right idea (a concept that has life implications far beyond how many responses to seek out for a Survey!). This idea is especially true for qualitative expertise Surveys in which the professional credentials of the respondents carry tremendous weight. Quality matters far more than quantity.

We live in a