4 Reasons Why I Didn't Finish Filling Out Your Survey

User research is hard! Please don't make it harder on yourself by giving me the following reasons to abandon your survey:

1. It’s too boring

Let's say you're a water bottle company and you want to know why people buy not only water bottles but reusable coffee mugs and other beverage containers/accessories. So you come up with a list of about 20 questions to reuse for every. single. subcategory. in your industry. Water bottles for everyday use! Water bottles for athletic use! Coffee mugs for home! Coffee mugs for travel! Wine containers! Beer containers! Let's get answers for them all!

Here's the thing though- at some point, I will stop taking care to answer your questions accurately because I cannot bear to look at another set of the same questions about a slightly different product than the five you just asked me about. At that point, I’ll probably just quit, but if I finish it’s with the intent to get this over thing with as fast as possible and not to give you good answers. Thus, a boring survey is a lying survey. 

However, if you had just asked me about one of those six marginally different products, I would be more likely to have finished the survey and given you more thoughtful answers. I'm no statistician, but I'm pretty sure 100 good responses about one product are more valuable than 600 useless responses about six products.

Don't do this to me

2. You’re asking me to recall too much

You want to learn how people choose where to get gas for their car, so you ask something like “when you get gas for your car, what factors contribute to your choice of a gas station?” This seems like an easy enough question, but you’re actually asking me about an estimated 500 different events in my life at this point. Not only was each motivated by different contributing factors according to context and need, and not only do I barely remember any of them, but I also don’t know how my inaccurate generalizing about the overall experience is useful to you.

What I can tell you about is the last time I bought gas and why I chose that particular gas station. Not only is this an easier question for me to answer, but it’s also an easier question for a larger number of people to answer accurately. This means that you will not get an in-depth view of my gas station history, but you weren’t going to get that anyway (see 3 and 4). But hopefully what you will get is more answers that give you a clearer snapshot of the current trends regarding gas station choices. 

3. You’re asking too much of me (and you too)

Regardless of how much I like you or want your project to be successful, I cannot write a whole essay for you about my feelings about the Spotify app. I could barely get myself to start a paper in grad school when I was interested in the topic and I wanted to graduate. But even if I could write you a whole essay about Spotify, how many other respondents are willing to do the same? And even if you got a whole bunch of Spotify essays, wouldn’t this mean that your respondents tend to skew toward the far ends of the love/hate spectrum?

100% percent of survey respondents found these pugs "adorable."

And how are you going to synthesize that data? See, it’s not just that you’re asking me to do too much, you’re also going to have to weed through all of that for some useful insights. Insights that you may have gotten more easily if your survey had simple, easy-to-answer questions. And if you're trying to get insights that you can't get through simple, easy-to-answer questions, I'm sorry but:

4. What you’re trying to get through this survey is what interviews are for

Tools work best when they are being used for what they are good at. Hatchets are great at cutting branches, but they aren’t great at giving haircuts. Surveys are great at getting a somewhat shallow view of your topic from a large number of people. Research interviews are great at getting an in-depth view of your topic from a smaller number of people. But surveys are not great at getting an in-depth view of your topic from a large number of people because again, no one wants to write you an essay about their feelings about the Spotify app, and it wouldn’t be worth your time to read and synthesize those essays anyway. You can only get that in-depth view from conducting research interviews, which means you'll be able to do things like ask follow-up questions and leave pauses to prompt your interviewee to continue answering- things that surveys are just not designed for. And just like using a hatchet for a haircut, using surveys for the wrong job can lead to inaccurate and potentially dangerous results. 

Does this feel like a good idea?