Pattern #43: Long Ad Titles
Was Tested On Bing.com

Pattern Author: Ronny Kohavi - General Manager, Analysis and Experimentation @ Microsoft

See Version A & The Result

Learn How Often This Pattern Won Or Lost, By How Much And Where.

Members get access to all test data for a given pattern along with: repeatability, detailed effects, sample size, and conversion rates for multiple metrics.

Long Ad Titles

  1. 1. Replace Short Link Titles With Longer Ones

  • Test 133
  • Tested on   Bing.com   by Ron Kohavi
  • Dec 13, 2017

Maybe It Worked Here

Screen: Search Results

Source: exp-platform.com/2017abtestingtutorial/

In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert.

HBR, September–October 2017 Issue, https://hbr.org/2017/09/the-surprising-power-of-online-experiments

Note: This experiment was a solid success and replicated multiple times over a period of months. It worked at Bing and had a profound influence. The only reason why we atributed a 0.25 point (a "Maybe") was because we don't have the exact sample size and conversion data.

 


For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.

Avoid What Failed & Repeat What Worked With
Access to All 44 Patterns Based on 75 Tests

Total sample size across all tests is 11,689,452 visitors - that's a lot of testing to do on your own :)

  • See the exact implementation of each test with Version B screenshots
  • See the effect from each test with shallow & deep metrics
  • Predict the impact of each pattern with summary median effects and repeatability
  • See what other reader's interpretations as to why these tests worked or didn't
  • Get new patterns and tests each month as we publish new results
  • Billed Annually
    2 Months Free
  • Billed Monthly
    30 Day Money Back
Company

License to share and use the data with everyone at your company.

$200 / month
$2400 paid upfront

Get Access

Team

License to share and use the data with up to 5 people.

$100 / month
$1200 paid upfront

Get Access

Solo

License to share and use the data for 1 individual.

$50 / month
$600 paid upfront

Get Access

Company Wide Access + Coaching

We'll personally guide you with: test design + idea roadmapping + weekly coaching
+ License to share and use the data with everyone at your company

Get Access With Coaching $2900 / month  |  Learn More