GoodUI BLOG - Back To All Posts

Agile A/B Testing: Using Stop Rules To Minimize Losses & Time Wasted

Tagged As: Process

Elements Of A Successful Optimization Program

Time and money are precious, among other things. So when we seek gains while minimizing losses through risky a/b tests, we are also faced with an interesting problem of making the most optimal decisions, in the most optimal time frame. The reality is that tests will have variations which will hurt the business by exposing it to loses and waste its time with insignificant changes. This challenges the classic approach to experimentation of estimating the sample size, setting and forgetting. I’m beginning to believe that a more effective approach to optimization is one where stop rules are combined with good enough data. By not committing to run tests to their fullest estimated duration, being agile, winning tests should have greater chances to emerge.

Wasting Time With Insignificant Tests – Problem One

Here is the first problem. Let’s say that you estimate your a/b test to detect a +20% increase off your magical 10% purchase rate, to realize you need over 7,200 visitors (3,600 per variation x 2) to detect it. Great. You design the A/B test, set it and the wizards will warn you not to peek or touch anything until the 7,200 sample size has been fully exposed. The hardcore statisticians do have a valid point as repeated significance testing (checking on the test) introduces your chances of detecting a false positive rate. But that aside for now, remind yourself that for some online businesses the wait time associated with a 7,200 sample size might be hours and for others, possibly months. And let’s say that you do glance at the test mid way through and see a highly insignificant +3% increase with a humongous ±22% margin of error. This is as grey as it gets and here the problem begins to shine through – your chance of turning the test around and getting B to jump from a +3% increase to over +20% are way smaller than a measly false positive rate from checking. When you made the estimate to detect the +20% increase, that was just a guess and now you have more concrete data. If you continue running the test longer, most likely you will continue exposing the test with a change that has no effect – essentially wasting time.

Wasting Money With Losing Variations – An Even Bigger Problem

There is an even bigger problem than the above which happens to the best of us. Building on the example above, let’s say that you peek at the test mid way through and you see a frightening -23% drop to sales. Now you are faced with the problem of exposing your sales funnel to losses and the question arises of how bad does it have to get for you to pull out – a very valid question indeed. If you continue running the test, the chances for the test to turn around, diminish, and your loses widen.

Setting Stop Rules

To address the above problem, we are exploring the idea of using stop rules for variations when they match particular criteria (we might actually perform and share simulations of these in a follow up post). The problem of introducing false positives from peeking to stop variations early is closely tied to how the stopping rule is defined. The most typical stop rule that is often criticized is when someone looks to stop the test as soon as a p-value of 0.05 is reached. We agree that this is probably a bad stopping rule as it has been observed to generate a false positive rate of as high as 30% or so. With that, here are our latest thoughts about more conservative stopping rules which combine a minimum conversion threshold, an effect magnitude, the p-value, and the p-value’s persistence into the test.

Stop Rule For Losing Variations (Proposed)

Disable any tested variation for revenue sensitive tests, as soon as the following are met:

Stop Rule For Insignificant Variations (Proposed)

Disable any tested variation as soon as the following are met:

What About You?

Please share your comments below on what you do to avoid losing variations or wasting time with insignificant changes (aside of increasing your chances of generating winning tests – a different topic).

Or Hire Us To Optimize Your Site While Minimizing Your Losses

Tweet about this on TwitterShare on Facebook0Share on Google+1Share on LinkedIn0Buffer this pageEmail this to someone

Posted by Jakub Linowski on Mar 17, 2016

Leave Your Reply

2 Comments on "Agile A/B Testing: Using Stop Rules To Minimize Losses & Time Wasted"

Sort by:   newest | oldest | most voted
My ideas on stopping rules are basically formulated around the premise that statistics is a tool to aid reasoning not a substitute for it. As such I tend to think that generic stopping rules are a poor substitute for reasoning about the results. Most of my general advice can be found in this post I did for the KISSmetrics blog: The basic point is that it’s okay to stop early, or at a low confidence if you have another idea lined up and ready to test. In Computer Science talk this is the trade off between exploration and exploitation.… Read more »
Jakub Linowski

Will. Thanks for the comment.

My point is what / if any rules do people apply to avoid waiting till the very end of an estimated experiment if it shows strong weakness or insignificance. Let’s say this is what week 6 of 8 looked like: Would you wait 2 more week to most likely end with an insignificant result or would you rather stop it and put something better into testing? If so … what triggers / rules would you use to remove such weak variations?

Hire Us To Optimize Your Site
We can lift your metrics and prove it.