The Problem With Online Surveys

Ketchup Questions and Online Surveys

“On a scale of one to ten, how do you rate the color of your current ketchup as compared to the color of all other tomato-based condiments you have tasted in the past 90 days?”

That’s a “ketchup question,” quoted verbatim from a telephone survey conducted by a research firm for a ketchup maker. It’s written like that to provide a quantitative answer that can be analyzed with a statistical analysis package. Ketchup marketers need to know that 52.665% of Venezuelan heads of household rate the color of their ketchup better than the color of competitive tomato-based condiments. Nobody knows why.

This video shows an automated tool for writing ketchup questions (from SurveyMonkey):

Online surveys are often promoted as a quick and easy way to learn about customers. That may be true for ketchup (although we doubt it). But B2B startups that need to validate the assumptions in their plan or identify the best Beachhead Market typically can’t get the information they need from online surveys.

The reason is that busy executives seldom participate in online surveys. The less busy employees who do fill them out seldom provide detailed answers. They may click a multiple-choice box, or type in a few words, but that’s it. No detail, no nuance, nothing to help you understand the Pain that causes them to lose sleep at night.

SurveyMonkey Best Practices:

Here’s an example of a SurveyMonkey survey using “certified questions.” (from SurveyMonkey)

Can you identify nuanced differences among customers using questions like these? Probably not. To adequately define your Target User Persona, you need to understand what problems they struggle with, and more importantly, WHY those problems are so difficult. You need to be able to discuss “WHY?”

How Can You Identify Customer Pain?

Here’s one simple technique you can use in conversation with a customer. Each time a customer answers a question with a declarative answer, explore that statement with them. Here’s a common example:

“We use Acme.” “Why did you choose Acme?”

“Acme integrates with our Bravo system.” “How well does Acme solve the problem you described earlier?”

“It solves about 60%.” “What is the impact of that other 40% not being solved?”

“Finishing that takes three of us two days every month. One of us is leaving next month.”

Even if they clicked the “Acme” checkbox in the online survey, you could not have learned any of those subsequent details. Complex products require detailed customer feedback about customer Pain, which means you have to talk to customers.

This is covered in detail in the course Stage 1: Validate Your Plan.