Patterns Blog BetterUI Review Coaching Courses Datastories

Overcoming Fears Of Your First Experiment

Tagged As: Optimization

Those who run A/B tests sometimes forget how scary it might have been in getting started with the first online experiment. Such fears, big or small, can irrationally discourage us from discovering more effective design ideas. I’m going to identify and address some of these fears along with ways to overcome them as I think that anyone is better off testing than not. The experimental world of seeking effects (both positive and negative) is a very exciting and rewarding one.

The Fear Of Uncertainty

Low Certainty
Human beings like certainty and by not running any tests you may feel as if your most recent version is your best one – even if that is only an illusion. The moment your experiment is launched, you are automatically placed into a state of not knowing which of your variations are better. To make things worse, the early days or weeks of an experiment can especially be uncertainty as they are ruled by chance. Experiments typically start off with illusionary winners flipping into losers back and forth chaotically before the true effects begin to stabilize. Such uncertainty may be uncomfortable for some. Don’t let this stop you, because:

The Fear Of Having To Implement An Undesired Win

The Drop
Some experiments test variations which we might already believe to be better for some specific business reason by some specific measure. We might be asked to run such experiments where we don’t want certain variations to win. In these situations, running the experiment may feel uncomfortable as it may feel like we have to implement an undesired outcome. Don’t let this stop you, because:

The Fear Of Being Wrong

The Winning Streak
We all make predictions some of which we might announce to our peers. In such cases experiments might feel as if we are the ones being tested. Now what if that variation we said will win does not win? Being wrong can feel frightening. Don’t let this stop you, because:

The Fear Of Losing Profits

The Bump
If you want to run experiments that impact the bottom line, you will need to run tests that affect business metrics such as sales and revenue. This can feel frightening when combined with the possibility of negative effects. We’ve observed quite a few times where a client with access to a test link would ask to stop a test with a -10% or -20% drop in sales. Don’t let this stop you, because:

The Fear Of Setting Up The Experiment

The Bump
When setting up an experiment for the first time, the technical aspects may overwhelm some people. Will the data be measured accurately? Is it difficult to getting it started? Won’t the experiment break something else? Don’t let this stop you, because:

Know Of Other Fears? Share Them Right Here

Please add any other fears which you think might exist around running a/b tests.

Posted by Jakub Linowski on Dec 19, 2016

Leave Your Reply

Be the First to Comment!