All Latest 650 A/B Tests

In this experiment from Conversionrate.store, the framing of the registration message was changed from a generic account creation one to a specific image selected by the user. The experiment started on a listing page of a stock photography / illustration site. The control showed a more generic message with benefits for signing up and making the purchase. Whereas the variation repeated the actual image that customers clicked on from listing pages - establishing continuity as well as providing a reason for signing up. Impact on sales was measured.

In this simple experiment, a highly visible coupon field was replaced with a less visible (but clickable) link in the variation. Clicking on the link would show the coupon field. Impact on sales and revenue was measured.

In this experiment, the discounted price was shown along with an active promotion. The control only showed that the relative -8% discount was present with the pre-discounted price.

This experiment tested the order of purchase plans. The control version sorted the purchase options by the least expensive while the variation sorted them by the most expensive first. Impact on sales and revenue was measured.

In this experiment, a 3 plan vs 2 plan pricing page was shown to potential customers. Impact on sales and revenue were measured.

In this experiment, the order of pricing plans was rearranged as to show the most expensive one first.

In this experiment, side-by-side plan features were aligned and changed to a comparison table with checkmarks for easier comparison.

In this experiment, all three prices of each plan were shifted higher for greater visibility.

In this experiment, the three pricing plans were condensed into a single recommended plan (annual), with a secondary option to choose the monthly plan.

In this experiment, 4 things were adjusted in the variation: the highest pricing plan was shifted to the left, it was set as the default one, the recommendation was also adjusted to point to the highest plan, and one benefit from the lowest plan was removed (customer support).

In this experiment, two pricing plans were inverted to show the most expensive plan first (in the variation).

This test deprioritized the free option (kostenlos) of cancelling a contract. It did so by placing it under the paid options as small text link / radio option.

In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert.

HBR, September–October 2017 Issue, https://hbr.org/2017/09/the-surprising-power-of-online-experiments

Note: This experiment was a solid success and replicated multiple times over a period of months. It worked at Bing and had a profound influence. The only reason why we atributed a 0.25 point (a "Maybe") was because we don't have the exact sample size and conversion data.

In this experiment, the plans were sorted by the most expensive first, left to right (variation).

This test simply ordered the pricing plans from highest to most expensive. Overall sales decreased by an insignificant -1% with a possible +14% increase to the most expensive plan. The net effect was an insignificant 0.9% increase the the premium plans.

VWO.com published this test which removed two coupon fields on a shopping cart: a gift card code and a special offer code.

Kyle's team changed a donation form for the Barack Obama 2012 campaign from a single step to a 4 step one. The 4 steps were: amount, personal information, billing information and occupation/employer.

"Our plan was to separate the field groups into four smaller steps so that users did not feel overwhelmed by the length of the form." - Kyle Rush