Surveys or open questions – What works better for initial product validation

Over the last few weeks, the new batch (fourth) of 16 companies at the Microsoft accelerator has been getting started with customer development. Some companies are fairly advanced, doing hundreds of thousands of dollars in revenue, but most are early stage. Last week our CEO-in-residence from the Israel accelerator, Hanan Lavy, came by to lead them through our customer development framework. The first thing I gathered from many entrepreneurs after that session, was that they were surprised at how it helped them revisit some of the assumptions they had made when they had the first idea about their product.

There’s an old saying that good sales folks are used to quoting “Always be Qualifying” (as opposed to the more popular ABC – Always be closing”, which never quite works, but is popular). The “lean startup” generation has its own version of that at the early stages of the startup – Always be validating – your assumptions, your plan, your pricing, your offering, etc.

Customer validations, early on, start by asking questions of customers, mostly in face-to-face meetings and then “graduate degenerate” to emails and phone conversations when entrepreneurs are unable to scale. I dont think there’s only one way to validate though – a good product manager uses all techniques to get in front of her customers / users as often as possible.

There are pros and cons to each of the techniques to validate your idea and assumptions, so rather than focus on all of them and their efficacy, I thought I’d take some time to share what I learned from 5 of the startup founders who have been trying 2 techniques over the last week with both Indian and US customers to validate their problem statements, ideas and positioning.

Think of this as A/B testing the format of communication as opposed to the medium or the message.

The medium most of them chose was email, given that they had to provide a quick turnaround back to Hanan (they were given 2 days to speak / connect with at least 10 customers. They could have chosen face-to-face meetings or focus groups using Webex / Skype, in app questions or real-time in-app chat, but they all chose to email their potential and few existing customers.

Now that most chose email, the next question I asked them was how many of the sent customers open-ended questions versus an objective survey with 3/4 choices for answers.

Turns out 2 of them used an online survey tool with 5 questions and 3/4 choices per question and 3 of them chose to send and email with 4-5 open-ended questions. Response rates varied from 40% to 60% I was told (fairly high given that their potential customers had only 24 hours to response). The survey’s got more responses than open questions.

What I did learn was that for companies that were earlier (had started building product, but did not have a prototype) the survey format worked better since they were able to get specific answers to questions and make decisions on 3 features they had to drop so they could ship quicker and gain more feedback quickly.

The open questions format worked for those that had worked longer with their customers and prospects since they got good qualitative feedback and a suggestion or two, which they had not considered before.

I have a personal bias against survey questions, since the choices are predetermined. Survey’s tend to be much better when you want a quick pulse to make feature decisions, not direction decisions. Surveys also work when you have a large pool of responses. Open questions on the other hand work just as well with 5 people as 50 – but at 50 people you have a hard time collating the responses. Open questions also requires you have a better relationship with the folks responding since their commitment of time is more.

What I also learned was that while there are pro’s and cons to both mechanisms, the decision you are trying should guide your choice of format, not the speed of the responses.

There are many types of decisions one takes at the early stages of the startup. Product direction decisions are rarely going to be resolved with surveys or email. Those are the type that many people leave to gut, data and lots of soul searching.

On the other hand, validating assumptions is always better with open questions is what I have learned.


2 thoughts on “Surveys or open questions – What works better for initial product validation”

  1. I would say “always be learning”, not “always be validating”.

    If you want to learn from an expert, do you guess at what might be true and then ask very specific questions to “validate” it? Or do you ask open-ended questions?

    Also, this obsession with “validating” often leads to confirmation bias – experiments crafted to confirm biases rather than truly test them, or prejudiced interpretations of results.

    To most effectively learn, we should be asking open-ended questions, probing deeply into the answers, and attempting to falsify our hypotheses, not “validate” them.

Comments are closed.