Make your call when you're ready to reveal the true effect. Early results are unreliable, but waiting too long takes time away from running other tests. Can you tell when to stop?
An effect becomes statistically credible when you get the right number of conversions:
|Conversions Needed Per Variation||17,000||2,500||800||400||200|
However, these are typically NOT rules for when to stop.
Don't stop a test as soon as you see a statistically significant result. This applies to Bayesian tools too. A false positive is likely to happen just by chance at some point in a test, usually early on. Finish running the full week and, if you can afford it, wait another week. Note that it can take days or weeks for a fake effect to disappear, depending on your traffic.In the end, you'll need to balance quick decision making with the risk of a false result.
Stopping involves many factors, including the ideal duration for the kind of effect we are aiming for. I recommend using Evan Miller's test duration calculator or my reverse test planning calculator, whichever you find more intuitive. Find a duration that makes sense for your traffic, conversion rate, and reasonable goals. The closer you get to that duration, the more you can believe any results you're seeing. If you stop half-way, you can better gauge your uncertainty.
Testing tools like VWO or Optimizely aren't aware of past tests or user research. If your tests are based on patterns that worked in the past and you're seeing similar results, you might be able to stop your test a bit earlier, taking that information into consideration. Subscribe to GoodUI Datastories to get the latest patterns and analysis to guide your decisions. But be careful not to stop your tests too early. What worked for others may not work for you, even if initially the results look promising. Proper test duration estimation is still recommended.