Tag Archives: Learning Objectives

Going beyond “I have an idea” – a rigorous design of your experiment

The first step to your experiment is usually an idea. Either you or someone at your startup has an idea, which you want to determine is a good one or not. Having tried a framework to experiment your ideas, I’d say the crucial firs step is designing the experiment.

Brainstorming the idea over a casual conversation is usually the first step. The tendency among most startup oriented people is to complicate it and think of all the edge cases and 100% coverage, which is why documenting all the ideas is important.

This documentation need not be long drawn out or format, but needs to be written down.

The best way to do this is to continue using your multi-user messaging solution as before – Slack or HipChat, or Google docs suffices.

To design your experiment, you need to write down the 3 most important aspects of it.

a) Hypothesis b) Outcomes, c) Learning Objectives and d) Time frame

I mention hypothesis vs. assumptions for a reason.

Something that has yet not been proved to classify as a theory but believed to be true is a hypothesis.

An assumption is any statement that is believed to be true.

Until you know your idea will work or not, it is an experiment which has certain items you have to prove.

For example: If one of your customer service people have an idea – they believe as a startup you should send a summary email of the usage of the product every month so the customer understand how valuable your product is to them.

They believe this will reduce churn – the outcome you desire from this experiment.

Your hypothesis may be many all or some of which might be proven, yet others may be false – a) email summary reduces churn b) customers would like the email summary c) your email summary captures key pieces of information about the usage that customers care about and others.

Finally the learning objectives – this is important especially for failed experiments, but it will be useful if it succeeds as well – It helps plan future experiments and creates filters for ideas that come next.

Designing your experiment involves documenting these 3 important items, usually by the person who has the idea. Even an email will suffice, but documenting it is key – even an email will suffice.

Finally in designing the experiment, the time frame how long you are going to run it is important. First so you dont have too many experiments running at the same time, which might skew results, but second so that you can give each experiment the due time to determine if it truly failed or succeeded.

I was thinking of an acronym – HOTEL – Hypothesis, Outcomes, Time Frame, Experiment Design and Learning objectives, but it does not quite fit. I’d love some suggestions on if you can come up with a better acronym of if you believe I have missed documenting anything.

How to design Experiments
How to design Experiments