Pattern #43: Long Titles

Pattern #43  Tested 5 timesFirst tested by Ronny Kohavi Recently tested by Marika Francisco on Jul 15, 2022

Based on 5 Tests, Members See How Likely Version B Wins Or Loses And By How Much

LOSSES
-5
-4
-3
-2
-1
FLAT
+1
+2
+3
+4
+5
WINS

Measured by the sum of negative and positive tests.

A B
Long Titles (Variant A) Long Titles (Variant B)

Expected Median Effects Of B

?

Progression

(2 tests)

?

Leads

(1 tests)

-

Signups

-

Engagement

?

Sales

(3 tests)

?

Revenue

(2 tests)

-

Retention

-

Referrals

?

ANY PRIMARY

(5 tests)

Tested on

Tests

Added

Isolated

Test # 133 on Bing.com by $conducted_test->test->user_->first_name . ' ' . $conducted_test->test->user_->last_name Ronny Kohavi    Dec 13, 2017 Test link

It Worked Here

Zoom In
A
B

Source: exp-platform.com/2017abtestingtutorial/

Listing

In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert.

HBR, September–October 2017 Issue, https://hbr.org/2017/09/the-surprising-power-of-online-experiments

Note: This experiment was a solid success and replicated multiple times over a period of months. It worked at Bing and had a profound influence. The only reason why we atributed a 0.25 point (a "Maybe") was because we don't have the exact sample size and conversion data.

 

Added

Isolated

Test # 314 on Zapimoveis.com.b... by $conducted_test->test->user_->first_name . ' ' . $conducted_test->test->user_->last_name V. Barros Peixoto    Aug 21, 2020 Test link

Find Out How It Did With 6,533,516 Visitors

  • Measured by filling out any lead form

Zoom In
  • ?% (? successes out of ? visits)
A
  • ?% (? successes out of ? visits)
B
Product

In this experiment, a dynamic page title was generated and added at the top of the screen. The first few words from a property description were used to dynamically generate these titles. The effect on leads was measured.

Get Access To See The Test Results

Added

Isolated

Test # 359 on Snocks.com by $conducted_test->test->user_->first_name . ' ' . $conducted_test->test->user_->last_name Samuel Hess    Jun 11, 2021 Test link

Find Out How It Did With 129,030 Visitors

  • Measured by adds to cart

  • Measured by unique sales

Zoom In
  • ?% (? successes out of ? visits)
  • ?% (? successes out of ? visits)
A
  • ?% (? successes out of ? visits)
  • ?% (? successes out of ? visits)
B
Product

In this experiment, product titles were extended with descriptive copy. Instead of just showing the product name, "with organic cotton" was appended on product and category/listing pages. Impact to adds to cart and sales was measured.

Get Access To See The Test Results

Replaced

Isolated

Test # 382 on Snocks.com by $conducted_test->test->user_->first_name . ' ' . $conducted_test->test->user_->last_name Samuel Hess    Oct 31, 2021 Test link

Find Out How It Did With 140,756 Visitors

  • Measured by adds to cart

  • Measured by completed sales

Zoom In
  • ?% (? successes out of ? visits)
  • ?% (? successes out of ? visits)
A
  • ?% (? successes out of ? visits)
  • ?% (? successes out of ? visits)
B
Product

In this experiment, the variation contained a simple text change that described the quality of the product. According to Google Translate the text changed from "tight men's underpants" (control) to "are so comfortable that you don't feel them when you wear them" (variation).

Get Access To See The Test Results

Replaced

Isolated

Test # 421 on Amazon.com by $conducted_test->test->user_->first_name . ' ' . $conducted_test->test->user_->last_name Marika Francisco    Jul 15, 2022 Test link

Find Out How It Did With 3,727,581 Visitors

  • Measured by revenue

  • Measured by completed transactions

Zoom In
  • ?% (? successes out of ? visits)
A
  • ?% (? successes out of ? visits)
B
Product

Fluke is an industrial manufacterer of measurement and calibration devices and they ran a test on their Amazon store for a series of their prodcts. Instead of using short product names, they tested longer and more descriptive ones. Impact was measured on sales.

Get Access To See The Test Results

Leaks

Leak #74 from Etsy.com   |   Dec 22, 2021 Product

Etsy A/B Tests Smaller & Shorter Product Titles

A - Oct 26, 2021
B - Oct 26, 2021 REJECTED

Etsy just ran a simple little experiment that changed their product titles into smaller and truncated ones. The outcome looks like it has been rejected so something didn't work out here. View Leak

+0.5 Evidence

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.