Meta-Analysis Calculator For A/B Tests

apples

For meta-analyzing and combining two or more a/b tests together.
Good for apples, red, green, orange-like, pink or simply honey crisps. Maybe even oranges or fruit as well? Your call.

A/B Test 1: / vs /
Effect P Z

A/B Test 2: / vs /
Effect P Z
A/B Test 3: / vs /
Effect P Z
A/B Test 4: / vs /
Effect P Z
A/B Test 5: / vs /
Effect P Z
QUICK SUMMARY OF APPROACH
  1. We take an inverse variance-weighted (precision-weighted) average of the lifts from each experiment to estimate the "true" lift that's assumed common to all experiments. This is also the maximum-likelihood estimate.
  2. Because we're working with RELATIVE lifts, we estimate the variance of the lift using the standard approach derived using the delta method.

CREDITS
A big thanks to the following indivduals for making this little project possible:
Ron Kohavi for guidance and input on weighing of p-values and inspring this project with his own meta-analysis spreadsheet.
Simon Dhala from whom I took the cumnormdist PHP function.
Tyler Buffington for multiple discussions and input about various meta analysis approaches.
Ryan Thomas for finding that the initial p-values were not being properly calculating upon the initial release.

Thanks,
Jakub Linowski