Pattern #20: Canned Response

Pattern #20  Tested 11 timesFirst tested by Mark Freedle Recently tested by Ardit Veliu on Sep 30, 2022

Based on 11 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
Canned Response (Variant A) Canned Response (Variant B)

Expected Median Effects Of B

X.X%

Progression

(3 tests)

X.X%

Leads

(9 tests)

-

Signups

-

Engagement

X.X%

Sales

(2 tests)

-

Revenue

-

Retention

-

Referrals

X.X%

ANY PRIMARY

(11 tests)

All 10 tests meta-analyzed: +X.X% combined effect (p-val X.XXXX)

58.7% of 90% cumulative power target at 2% MDE from 10 tests Help replicate this with an A/B test

90%
↓
6.8% 10.8% 12.6% 13.9% 3.9% 16.1% 3.7% 6.7% 3.8% 5.3%
Signup 7 tests: X%
Product 3 tests: X%

Tested on

Tests

Pattern #20: Canned Response
Was Tested On Glass.net by Mark Freedle

Added

Isolated

Test # 112 on Glass.net by Mark FreedleMark Freedle Jul 14, 2017 Test link

Find Out How It Did With 13,197 Visitors

  • Measured by visits to next screen

  • Measured by visits to thank you page

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
4.2%1%
6.8%2%
21.2%5%
63.3%10%
80%12.2%
92.7%15%
99.5%20%

Get Access To See The Test Results

The Same Pattern Was Also Tested Here

Added

Isolated

Test # 433 on Expertinstitute.... by Ardit VeliuArdit Veliu Sep 30, 2022 Test link

Find Out How It Did With 14,490 Visitors

  • Measured by form submissions

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.7%1%
5.3%2%
13.3%5%
38.7%10%
69.9%15%
80%17%
90.6%20%

In this experiment a dynamic canned response was added to a lead form. The contents of the copy reflected a handful of user choices from other form field selections. Impact on leads / form submissions was measured.

Get Access To See The Test Results

Replaced

Isolated

Test # 409 on Expertinstitute.... by Ardit VeliuArdit Veliu Apr 30, 2022 Test link

Find Out How It Did With 2,831 Visitors

  • Measured by completed forms

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.1%1%
3.8%2%
6.9%5%
15.3%10%
28.4%15%
44.8%20%
80%31.3%

In this experiment, the copy of an input textarea on a lead form was used to summarize a user's choices. Instead of showing using a generic "Message" statement, the variation use the following formula: "I'm a [Lead Type] located in [State] looking for [Expert Type]. Looking forward to hearing from you today. Thank You." Impact of leads was measured.

Get Access To See The Test Results

Added

Isolated

Test # 249 on Autoscout24.com by Optimizely Optimizely Jul 17, 2019 Test link

Find Out How It Did

  • Measured by successful form completions

Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Product

AutoScout24 is Europe’s largest online marketplace for new and used cars. As published by Optimizely, they tested a form pre-filled with text that read: ‘Hello. I am interested in your vehicle. Please contact me. Kind regards.’ 

Get Access To See The Test Results

Replaced

Isolated

Test # 248 on Volders.de by Alexander KriegerAlexander Krieger Jul 16, 2019 Test link

Find Out How It Did With 13,878 Visitors

  • Measured by total contract cancellations

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
4.2%1%
6.7%2%
20.5%5%
61.4%10%
80%12.5%
91.7%15%
99.3%20%

In this experiment, an editable contract cancellation letter was tested against a non-editable one. The editable letter first appeared in a text state, which required a click on a link in order for it to be transformed into an editable textarea field.

Get Access To See The Test Results

Added

Isolated

Test # 231 on Glass.net by Mark FreedleMark Freedle Mar 14, 2019 Test link

Find Out How It Did With 1,342 Visitors

  • Measured by form submit to next step (validation)

  • Measured by total form leads

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.1%1%
3.7%2%
6.4%5%
13.8%10%
25.4%15%
40.2%20%
80%33.5%

Get Access To See The Test Results

Replaced

Isolated

Test # 147 on Reverb.com by Nicholas EvansNicholas Evans Jan 25, 2018 Test link

Find Out How It Did With 40,229 Visitors

  • Measured by offers started

  • Measured by sales from offers & direct buys

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Product
Statistical Power by Minimum Detectable Effect
4%0.5%
7%1%
16.1%2%
67.2%5%
80%5.8%
99.8%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

Added

Isolated

Test # 128 on Acousticalsurfac... by Nina BayattiNina Bayatti Dec 02, 2017 Test link

Find Out How It Did With 2,951 Visitors

  • Measured by successful form submissions

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
3%0.5%
3.1%1%
3.9%2%
7.1%5%
16.1%10%
30.2%15%
47.7%20%
80%30%

Get Access To See The Test Results

Added

Isolated

Test # 100 on Vivareal.com.br by Rodrigo MauésRodrigo Maués Aug 01, 2017 Test link

Find Out How It Did With 203,704 Visitors

  • Measured by form submits

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Product
Statistical Power by Minimum Detectable Effect
4%0.5%
6%1%
12.6%2%
52.6%5%
80%7%
97.9%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

Added

Isolated

Test # 109 on 3dhubs.com by Rob DraaijerRob Draaijer Jun 01, 2017 Test link

Find Out How It Did With 4,005 Visitors

  • Measured by placed quotes

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Signup
Statistical Power by Minimum Detectable Effect
4%0.5%
6.4%1%
13.9%2%
59.9%5%
80%6.3%
99.4%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

Removed

Isolated

Test # 85 on Vivareal.com.br by Rodrigo MauésRodrigo Maués Mar 01, 2017 Test link

Find Out How It Did With 102,331 Visitors

  • Measured by lead form submits

  • XX.X% (XXX successes out of XXX,XXX visits)
  • XX.X% (XXX successes out of XXX,XXX visits)
Zoom In
Canned Response - Variant A A
Canned Response - Variant B B
Product
Statistical Power by Minimum Detectable Effect
4%0.5%
5.8%1%
11.9%2%
49.3%5%
80%7.3%
97%10%
99.9%15%
99.9%20%

Get Access To See The Test Results

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.