GoodUI Analytics Addon for VWO

Make better decisions right from your VWO dashboard

Start Using It Now Version 1.1.6

Requires the New VWO

Step 1: Install Greesemonkey or Tampermonkey

You'll need a browser plugin to run the Addon script:

Firefox users,
Install Greesemonkey (FREE)

Chrome users,
Install Tampermonkey (FREE)

Step 2: Copy our script

Copy the Addon script to your clipboard.

Get Version 1.1 From GitHub

Step 3: Paste the script

  1. Open Greesemonkey
  2. Click "New User Script"
  3. Click "Use Script From Clipboard" button
    (or just hit OK and paste the script on the next page)
  4. Enable the script
  5. Reload your VWO dashboard
  1. Open Tampermonkey
  2. Click "Add a new script" option
  3. Paste the script on the next page
  4. In the header of the script, replace @include with @match
  5. Save
  6. Reload your VWO dashboard

Results with 95% Confidence on Overview Page

See p-values and margin of error right on Overview. Use the dotted guide line to gauge statistical significance. If the margin of error for a variation is completely above the horizontal guide line, then the result is statistically significant at greater than 95% level (VWO's Detailed Report is based on < 80% level).

Significant!

Samples Size and Confidence Analytics on Dashboard

The biggest problem in A/B testing is not estimating the sample size required for an experiment and relying instead on the A/B tool to declare the winner. Our tools give you more trustworthy results:

Sample Size Guide suggests a duration to confirm the observed effect, which is always longer than VWO recommends
Confidence Guide tells you how trustworthy an effect is based on non-overlapping 95% Confidence Intervals, minimum duration/conversion rules, and current effect size
A range for degree of improvement at the 99% confidence level gives a truer sense of the potential size of the effect
Useful stats at a glance
Start Using It Now Version 1.1.4

Frequently Asked Questinos

Are your methods statistically valid?

Our tool increases the trustworthiness of the results you see in VWO and mitigates the risks due to running tests improperly. Here are the ideal uses of our tool: (1) use it to set or change your sample size goal early in the test (e.g., if your traffic or conversion rate is different than expected), but we suggest running a "Conversion Tracking" test beforehand, (2) use it sparingly (ideally just once) to adjust duration midstream, and (3) see how long it would hypothetically take confirm the observed effect, so you can guage if a retest is even feasible.

Why get rid of "Chance to Beat"?

We found VWO's chance to beat to be unreliable and possibly buggy. We tell our clients to completely disregard the chance to beat. VWO wants to keep you motivated about testing. So they want you to see stronger results and get more winners. The good thing about that is it reduces your risk of missing true effects. The problem is you'll see many false positives and inflated effect sizes, leading to less confidence in the tool. We don't believe in sugar-coating results.


Why 95% Confidence?

VWO uses about 75%. Scientists use 95% (alpha of 0.05). We want the truth, so we can weigh the pros and cons of our decisions. If you make decisions based on low confidence, it means you might retest a winner and get completely different results. To mitigate that risk, we want to give you truer confidence. The range of improvement is shown with 99% confidence.

Even the best A/B testers break many statistical rules, which can invalidate their conclusions. Moreover, the web is a more complex testing environment than what standard statistical methods assume. Setting a higher standard helps compensate for these deficiencies.

Non-overlapping intervals or p-value?

We give you a p-value for reference. However, it is not enough that a result be statistically significant. We want to ensure it's not negative. If you have a case where the p-value is low, it could still be that, given the margin of error, the true effect is tiny or even negative and may not hold up to a retest. In contrast, when our tool gives you 95% Confidence, it means we are much more certain that it's statistically significant and greater than zero.


What power is used for sample estimates?

We use 85% power and 0.05 alpha, because we want to maximize your chances of finding a solid result. We also want to compensate for the fact that you're re-evaluating the duration midstream. You may not always reach the recommended duration, but it always tells you the truth: how long you should have planned this test for to confirm the results you are seeing. Technically, you should then stop and retest. Most A/B testers won't do that, so our Confidence Guide tries to mitigate some of that risk by using higher standards.

What if I find a bug?

Try refreshing the page first. Address all comments to the developer @vladmalik. If a bug is preventing your work, just disable to addon temporarily. This addon hacks VWO's dashboard and modifies it using jQuery using the information presented on a given screen. Whenever VWO changes things on their end, features may break temporarily. We use this tool daily for all our projects, so we fix bugs continuously. Bugs and updates will be posted on Twitter.


Does it work for revenue tests?

No. Revenue analysis requires transaction data for each user. However, analysis based on conversion rate is still useful. You'll have to analyze the revenue data manually.

How do I temporarily disable the addon?

To disable the addon temporarily, just turn off Greasemonkey or Tampermonkey and refresh.