Pattern #85: Benefit Button

Pattern #85  Tested 5 timesFirst tested by Andrei Zakhareuski Recently tested by Herman Klein on Aug 13, 2019

Based on 5 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
Benefit Button (Variant A) Benefit Button (Variant B)

Expected Median Effects Of B

X.X%

Progression

(5 tests)

X.X%

Leads

(1 tests)

-

Signups

-

Engagement

X.X%

Sales

(3 tests)

-

Revenue

-

Retention

-

Referrals

X.X%

ANY PRIMARY

(5 tests)

All 5 tests meta-analyzed: +X.X% combined effect (p-val X.XXXX)

25.8% of 90% cumulative power target at 2% MDE from 5 tests Help replicate this with an A/B test

90%
↓
3% 5.5% 4.3% 5.2% 10.8%
Global 2 tests: X%
Home & Landing 1 test: X%
Signup 1 test: X%
Product 1 test: X%

Tested on

Tests

Pattern #85: Benefit Button
Was Tested On Runrun.it by Franklin Valadares

Replaced

Isolated

Test # 6 on Runrun.it by Franklin ValadaresFranklin Valadares Sep 15, 2015 Test link

Find Out How It Did With 9,618 Visitors

  • Measured by visits to signup screen

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Benefit Button - Variant A A
Benefit Button - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.8%1%
5.5%2%
14.3%5%
42.5%10%
74.8%15%
80%16%
93.4%20%

Get Access To See The Test Results

The Same Pattern Was Also Tested Here

Replaced

Isolated

Test # 40 on Adoramapix.com by Herman KleinHerman Klein Aug 13, 2019 Test link

Find Out How It Did With 13,454 Visitors

  • Measured by order starts

  • Measured by successful orders

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Benefit Button - Variant A A
Benefit Button - Variant B B
Product
Statistical Power by Minimum Detectable Effect
4%0.5%
5.5%1%
10.8%2%
43.7%5%
80%7.8%
94.7%10%
99.9%15%
99.9%20%

In this experiment only the button label changed. The control had a more immediate (next step-oriented) button label of "Upload Your Photos Now". The variation tried a more benefit-oriented button of "Order Now" (hinting at paying and obtaining the product).

Get Access To See The Test Results

Replaced

Isolated

Test # 213 on Mt.com by Vito MediavillaVito Mediavilla Dec 04, 2018 Test link

Find Out How It Did With 26,639 Visitors

  • Measured by lead form visits

  • Measured by form submits

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Benefit Button - Variant A A
Benefit Button - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.7%1%
5.2%2%
12.8%5%
37.3%10%
68.1%15%
80%17.4%
89.4%20%

In this variation, a longer button label with a clearer benefit was tested against a shorter one.

Get Access To See The Test Results

Replaced

Isolated

Test # 204 on Kenhub.com by Niels HapkeNiels Hapke Oct 11, 2018 Test link

Find Out How It Did With 171,904 Visitors

  • Measured by clicks on "help" link

  • Measured by subscriptions started

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Benefit Button - Variant A A
Benefit Button - Variant B B
Global
Statistical Power by Minimum Detectable Effect
3%0.5%
3.3%1%
4.3%2%
8.5%5%
21.3%10%
40.7%15%
62.3%20%
80%24.9%

In the variation, the "Help" section was reworded to "How to study".

Get Access To See The Test Results

Replaced

Isolated

Test # 203 on Driving-tests.or... by Andrei ZakhareuskiAndrei Zakhareuski Oct 10, 2018 Test link

Find Out How It Did With 67,451 Visitors

  • Measured by visits to premium landing page

  • Measured by actual sales

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Benefit Button - Variant A A
Benefit Button - Variant B B
Global
Statistical Power by Minimum Detectable Effect
3%0.5%
2.7%1%
3%2%
3.9%5%
5.7%10%
8.1%15%
11.1%20%
80%87.6%

In the variation, the button label "Upgrade To Premium" was changed to one with a clearer benefit of "Pass The First Time".

Get Access To See The Test Results

Leaks

Leak #19 from Airbnb.com   |   Aug 13, 2019 Product

Airbnb A/B Tested These 3 Button Labels And They All Failed

A - Jul 22, 2019
D - Jul 22, 2019 REJECTED

Button label tests are easy to setup, including this one that Airbnb ran on their property listing pages. In total we detected that they a/b tested 3 variations against the original control. One month later, all treatment variations were rejected suggesting that the "Reserve" button defended its superior performance. View Leak

+0.5 Evidence

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.