Pattern #121: Free Shipping

Pattern #121  Tested 6 timesFirst tested by Jakub Linowski Recently tested by Melina Hess on Aug 16, 2022

Based on 6 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
Free Shipping (Variant A) Free Shipping (Variant B)

Expected Median Effects Of B

X.X%

Progression

(4 tests)

-

Leads

-

Signups

-

Engagement

X.X%

Sales

(6 tests)

X.X%

Revenue

(1 tests)

-

Retention

-

Referrals

X.X%

ANY PRIMARY

(6 tests)

All 5 tests meta-analyzed: +X.X% combined effect (p-val X.XXXX)

84.1% of 90% cumulative power target at 2% MDE from 5 tests Help replicate this with an A/B test

90%
↓
21.8% 21.9% 22% 14% 61.2%
Product 3 tests: X%
Shopping Cart 2 tests: X%

Tested on

Tests

Pattern #121: Free Shipping
Was Tested by Jakub Linowski

Test # 332 by Jakub LinowskiJakub Linowski Dec 30, 2020 Test link

Find Out How It Did With 97,277 Visitors

  • Measured by adds to cart

  • Measured by total sales

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Product Added Isolated
Statistical Power by Minimum Detectable Effect
5%0.5%
8.6%1%
21.9%2%
80%4.8%
83.7%5%
99.9%10%+

Added

Isolated

In this experiment, an extra "Free Shipping" message was added on a product page - at the top of the buy box with an add-to-cart call to action. It's prominence was increased by using white copy on a darker blue background. Impact on adds-to-cart and total sales was measured.

Get Access To See The Test Results

The Same Pattern Was Also Tested Here

Test # 429 on Snocks.com by Melina HessMelina Hess Aug 16, 2022 Test link

Find Out How It Did With 56,067 Visitors

  • Measured by completed transactions

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Product Replaced Isolated
Statistical Power by Minimum Detectable Effect
8%0.5%
20.1%1%
61.2%2%
80%2.5%
99.9%5%+

Replaced

Isolated

Upon clicking the Add-To-Cart button, the button label changed to a cheering message which congratulated the user on his choice and ensuring that shipping is free. Translation: "Good choice. Your shipping and returns are free."

Get Access To See The Test Results

Test # 398 on Adoramapix.com by Herman KleinHerman Klein Feb 25, 2022 Test link

Find Out How It Did With 3,123 Visitors

  • Measured by visits to checkout

  • Measured by completed sales

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Shopping Cart Added Isolated
Statistical Power by Minimum Detectable Effect
4%0.5%
6.4%1%
14%2%
60.5%5%
80%6.3%
99.5%10%
99.9%15%+

Added

Isolated

In this experiment, customers were eligible for free shipping (with cart amounts of $39 or more) were presented with an earned free shipping message. The variation also showed the value of the earned free shipping - (for example $10). Impact on progression to checkout and sales was measured.

Get Access To See The Test Results

Test # 397 on Snocks.com by Melina HessMelina Hess Feb 24, 2022 Test link

Find Out How It Did With 92,187 Visitors

  • Measured by completed transactions

  • Measured by Revenue Per User

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Product Added Isolated

Added

Isolated

In this experiment a "Free Shipping & Returns" message was added just below the price information. 

Get Access To See The Test Results

Test # 377 on Adoramapix.com by Jakub LinowskiJakub Linowski Sep 30, 2021 Test link

Find Out How It Did With 4,422 Visitors

  • Measured by visits checkout

  • Measured by completed transactions

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Shopping Cart Added Isolated
Statistical Power by Minimum Detectable Effect
5%0.5%
8.5%1%
22%2%
80%4.7%
85.5%5%
99.9%10%+

Added

Isolated

In this experiment, a big "free shipping" badge was added and defaulted to when available. Impact on progression to checkouts and completed sales was measured.

Get Access To See The Test Results

Test # 331 by Jakub LinowskiJakub Linowski Dec 30, 2020 Test link

Find Out How It Did With 96,962 Visitors

  • Measured by adds to cart

  • Measured by total sales

  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • A: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
  • B: XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Free Shipping - Variant A A
Free Shipping - Variant B B
Product Added Isolated
Statistical Power by Minimum Detectable Effect
5%0.5%
8.6%1%
21.8%2%
80%4.8%
83.6%5%
99.9%10%+

Added

Isolated

In this little experiment, an extra "Free Shipping" message was added on a product page. It's prominence was increased by using white copy on a darker blue background. Impact on adds-to-cart and total sales was measured.

Get Access To See The Test Results

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.