Requires the New VWO
You'll need a browser plugin to run the Addon script:
Firefox users,
Install Greesemonkey (FREE)
Chrome users,
Install Tampermonkey (FREE)
Copy the Addon script to your clipboard.
See p-values and margin of error right on Overview. Use the dotted guide line to gauge statistical significance. If the margin of error for a variation is completely above the horizontal guide line, then the result is statistically significant at greater than 95% level (VWO's Detailed Report is based on < 80% level).
The biggest problem in A/B testing is not estimating the sample size required for an experiment and relying instead on the A/B tool to declare the winner. Our tools give you more trustworthy results:
Start Using It Now Version 1.1.4Our tool increases the trustworthiness of the results you see in VWO and mitigates the risks due to running tests improperly. Here are the ideal uses of our tool: (1) use it to set or change your sample size goal early in the test (e.g., if your traffic or conversion rate is different than expected), but we suggest running a "Conversion Tracking" test beforehand, (2) use it sparingly (ideally just once) to adjust duration midstream, and (3) see how long it would hypothetically take confirm the observed effect, so you can guage if a retest is even feasible.
We found VWO's chance to beat to be unreliable and possibly buggy. We tell our clients to completely disregard the chance to beat. VWO wants to keep you motivated about testing. So they want you to see stronger results and get more winners. The good thing about that is it reduces your risk of missing true effects. The problem is you'll see many false positives and inflated effect sizes, leading to less confidence in the tool. We don't believe in sugar-coating results.
VWO uses about 75%. Scientists use 95% (alpha of 0.05). We want the truth, so we can weigh the pros and cons of our decisions. If you make decisions based on low confidence, it means you might retest a winner and get completely different results. To mitigate that risk, we want to give you truer confidence. The range of improvement is shown with 99% confidence.
Even the best A/B testers break many statistical rules, which can invalidate their conclusions. Moreover, the web is a more complex testing environment than what standard statistical methods assume. Setting a higher standard helps compensate for these deficiencies.
We give you a p-value for reference. However, it is not enough that a result be statistically significant. We want to ensure it's not negative. If you have a case where the p-value is low, it could still be that, given the margin of error, the true effect is tiny or even negative and may not hold up to a retest. In contrast, when our tool gives you 95% Confidence, it means we are much more certain that it's statistically significant and greater than zero.
We use 85% power and 0.05 alpha, because we want to maximize your chances of finding a solid result. We also want to compensate for the fact that you're re-evaluating the duration midstream. You may not always reach the recommended duration, but it always tells you the truth: how long you should have planned this test for to confirm the results you are seeing. Technically, you should then stop and retest. Most A/B testers won't do that, so our Confidence Guide tries to mitigate some of that risk by using higher standards.
Try refreshing the page first. Address all comments to the developer @vladmalik. If a bug is preventing your work, just disable to addon temporarily. This addon hacks VWO's dashboard and modifies it using jQuery using the information presented on a given screen. Whenever VWO changes things on their end, features may break temporarily. We use this tool daily for all our projects, so we fix bugs continuously. Bugs and updates will be posted on Twitter.
No. Revenue analysis requires transaction data for each user. However, analysis based on conversion rate is still useful. You'll have to analyze the revenue data manually.
To disable the addon temporarily, just turn off Greasemonkey or Tampermonkey and refresh.