The Art of Disciplined Experimentation

Being a hobbyist is an awesome way to keep learning and test “theories” you have. Most cases, when I have a theory I’d like to prove or disprove, I’ve found the best way is to just try it out. That applies to a new product idea, new marketing technique or a new sales strategy that I have either a hunch for or have overheard from someone else.

The key part that I have learned from my experiments, is that you need a framework (or a model) to clearly outline what you intend to learn from it, what assumptions you made, what steps you took and what you learned from the experiment.

If you dont have a framework, you end up with a lot of experiments whose results might suit you at a later date, but you “forget” about those experiments.

The thing about experiments is you have to understand clearly why they succeeded or failed. 

If you do that and internalize the learning, it becomes a part of your decision making for the future. Experiments without learning is just wasting time – which is also a valid reason to experiment in itself, but you have to be clear about that upfront.

To be disciplined in my experimenting, I have found that doing one at a time suits me best. I found out from a expert in SEO about a much simpler way to track the keywords you want to rank for and a quicker ethical way (than the usual 2-3 months) to appear on the first search engine results page.

My immediate thought process was “that’s just not right” and “wont work all the time”. But it was right and it works, and the only way I would be convinced of it, was if I did it myself.

I also put a time frame for my experiment, to determine if its worth the result. Many of these experiments take several months, so doing nothing but that one experiment during that time, is hard. The results from that learning better make up for more than the time and effort.

Which is why I developed for myself a list of questions so I can be disciplined about my experimenting. These questions are fairly straightforward, but my lens for the questions is based on three criteria:

a) Will it be fun?

b) Will I learn something I dont already know?

c) What new things will I learn and where can I use the learning from the experiment?