Implement Or Discard?

Make your call when you're ready to reveal the true effect. Early results are unreliable, but waiting too long takes time away from running other tests. Can you tell when to stop?

When Should You Stop A Test?

Conversions Required

An effect becomes statistically credible when you get the right number of conversions:

Observed Effect2%5%10%15%20%
Conversions Needed Per Variation17,0002,500800400200
Conditions for a statistically significant result assuming a base conversion rate <20%. For base conversion rates 20-40%, you'd need 25% fewer conversions.

However, these are typically NOT rules for when to stop.

Full Weeks

It's recommened to run tests for full weeks, even if your traffic is high. See our Better Data for more advice.

Not Too Early

Don't stop a test as soon as you see a statistically significant result. This applies to Bayesian tools too. A false positive is likely to happen just by chance at some point in a test, usually early on. Finish running the full week and, if you can afford it, wait another week. Note that it can take days or weeks for a fake effect to disappear, depending on your traffic.

In the end, you'll need to balance quick decision making with the risk of a false result.

Stopping involves many factors, including the ideal duration for the kind of effect we are aiming for. I recommend using Evan Miller's test duration calculator or my reverse test planning calculator, whichever you find more intuitive. Find a duration that makes sense for your traffic, conversion rate, and reasonable goals. The closer you get to that duration, the more you can believe any results you're seeing. If you stop half-way, you can better gauge your uncertainty.

Testing tools like VWO or Optimizely aren't aware of past tests or user research. If your tests are based on patterns that worked in the past and you're seeing similar results, you might be able to stop your test a bit earlier, taking that information into consideration. Subscribe to GoodUI Datastories to get the latest patterns and analysis to guide your decisions. But be careful not to stop your tests too early. What worked for others may not work for you, even if initially the results look promising. Proper test duration estimation is still recommended.

More Games