Pattern #8: Natural Language Forms

Pattern #8  Tested 5 timesFirst tested by Greg Van Horn Recently tested by Patrick McKenzie on Aug 26, 2016

Based on 5 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
Natural Language Forms (Variant A) Natural Language Forms (Variant B)

Expected Median Effects Of B

X.X%

Progression

(2 tests)

X.X%

Leads

(4 tests)

X.X%

Signups

(1 tests)

-

Engagement

-

Sales

-

Revenue

-

Retention

-

Referrals

X.X%

ANY PRIMARY

(5 tests)

All 5 tests meta-analyzed: +X.X% combined effect (p-val X.XXXX)

40% of 90% cumulative power target at 2% MDE from 5 tests Help replicate this with an A/B test

90%
↓
4.3% 4.1% 14.2% 20.9% 3.7%
Home & Landing 5 tests: X%

Tested on

Tests

Pattern #8: Natural Language Forms
Was Tested On Prizegrab.com by Greg Van Horn

Replaced

Isolated

Test # 14 on Prizegrab.com by Greg Van HornGreg Van Horn Oct 20, 2016 Test link

Find Out How It Did With 3,133 Visitors

  • Measured by form submits

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Natural Language Forms - Variant A A
Natural Language Forms - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.3%1%
4.3%2%
8.8%5%
22.5%10%
43.2%15%
65.7%20%
80%23.9%

Get Access To See The Test Results

The Same Pattern Was Also Tested Here

Replaced

Isolated

Test # 59 by Someone Someone Oct 01, 2016 Test link

Find Out How It Did With 3,716 Visitors

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Natural Language Forms - Variant A A
Natural Language Forms - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.2%1%
4.1%2%
7.7%5%
18.5%10%
35.1%15%
54.8%20%
80%27.3%

Get Access To See The Test Results

Replaced

Isolated

Test # 58 on Bingocardcreator... by Patrick McKenziePatrick McKenzie Aug 26, 2016 Test link

Find Out How It Did With 736 Visitors

  • Measured by visits to post-signup

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Natural Language Forms - Variant A A
Natural Language Forms - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
3%0.5%
3.1%1%
3.7%2%
6.3%5%
13.5%10%
24.6%15%
39.3%20%
80%33.8%

Get Access To See The Test Results

Replaced

Isolated

Test # 60 on Embracepetinsura... by Matt AsterMatt Aster Sep 30, 2015 Test link

Find Out How It Did With 12,102 Visitors

  • Measured by policy selection start

  • Measured by visits to quote screen

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Natural Language Forms - Variant A A
Natural Language Forms - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
4%0.5%
6.5%1%
14.2%2%
60.3%5%
80%6.3%
99.3%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

Replaced

Isolated

Test # 15 on Embracepetinsura... by Matt AsterMatt Aster Aug 28, 2015 Test link

Find Out How It Did With 18,638 Visitors

  • Measured by policy selection start

  • Measured by visits to quote screen

  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
  • XX.X% (XXX successes out of XXX,XXX visitors)
Zoom In
Natural Language Forms - Variant A A
Natural Language Forms - Variant B B
Home & Landing
Statistical Power by Minimum Detectable Effect
5%0.5%
8.3%1%
20.9%2%
80%4.9%
81.8%5%
99.9%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

Leaks

Leak #68 from Airbnb.com   |   Apr 20, 2021 Home & Landing

Airbnb A/B Tests And Rejects A Natural Language Form

A - Mar 18, 2021
B - Mar 18, 2021 REJECTED

Airbnb just completed an interesting experiment on their Become A Host landing page. They replaced a simple headline with a more dynamic natural language form that provided a sense of how much money someone could make from hosting their place. View Leak

-0.5 Evidence

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.