Pattern #52: How It Works

Pattern #52  Tested 4 timesFirst tested by Justin Baer Recently tested by Michael McSweeney on Nov 28, 2023

Based on 4 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
How It Works (Variant A) How It Works (Variant B)

Expected Median Effects Of B

X.X%

Progression

(1 tests)

X.X%

Leads

(1 tests)

X.X%

Signups

(1 tests)

-

Engagement

X.X%

Sales

(3 tests)

-

Revenue

-

Retention

-

Referrals

X.X%

ANY PRIMARY

(4 tests)

All 4 tests meta-analyzed: +X.X% combined effect (p-val X.XXXX)

17.4% of 90% cumulative power target at 2% MDE from 4 tests Help replicate this with an A/B test

90%
↓
4.3% 4% 5.8% 4.6%
Home & Landing 2 tests: X%
Product 1 test: X%
Signup 1 test: X%

Tested on

Tests

Pattern #52: How It Works
Was Tested by Jakub Linowski

Added

Isolated

Test # 395 by Jakub LinowskiJakub Linowski Jan 31, 2022 Test link

Find Out How It Did With 17,099 Visitors

  • Measured by adds to cart

  • Measured by completed sales

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
How It Works - Variant A A
How It Works - Variant B B
Product
Statistical Power by Minimum Detectable Effect
3%0.5%
3.9%1%
5.8%2%
15.8%5%
47.3%10%
80.2%15%
80%15%
96%20%

In this experiment, a How It Works section was appended on a product detail page just above product reviews.

Get Access To See The Test Results

The Same Pattern Was Also Tested Here

Replaced

Isolated

Test # 505 on Shmoodyapp.com by Michael McSweeneyMichael McSweeney Nov 28, 2023 Test link

Find Out How It Did With 8,231 Visitors

  • Measured by transactions (7 days later)

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
How It Works - Variant A A
How It Works - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.4%1%
4.6%2%
10.3%5%
27.9%10%
53.2%15%
76.8%20%
80%20.9%

In this experiment, new paywall copy and messaging was used to encourage more users to signup and pay for access to a mental health app. The variation reinforced that users would be receiving a full featured application, with reminders about the upcoming billing. It reinforced that users will have a full week to decide and potentially cancel their application. Impact on lagging +7 day later transactions was measured.

Get Access To See The Test Results

Removed

Isolated

Test # 200 on Trydesignlab.com by Will AndersonWill Anderson Sep 14, 2018 Test link

Find Out How It Did With 33,802 Visitors

  • Measured by completed signups

  • Measured by course enrolments (paid)

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
How It Works - Variant A A
How It Works - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.2%1%
4%2%
7.4%5%
17.2%10%
32.4%15%
50.9%20%
80%28.8%

In this experiment, a "How It Works" content section was removed.

Get Access To See The Test Results

Added

Isolated

Test # 23 on Checkineasy.com by Justin BaerJustin Baer Oct 01, 2015 Test link

Find Out How It Did With 4,743 Visitors

  • Measured by visits to post-signup

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
How It Works - Variant A A
How It Works - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.3%1%
4.3%2%
8.7%5%
21.8%10%
41.7%15%
63.5%20%
80%24.6%

Get Access To See The Test Results

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.