Bol Tests 8 Add-To-Cart Buttons With 1 Winner
Bol, a leading online retailer in the Netherlands, just completed a beautiful experiment with 8 add-to-cart button variations. Two potentially leading variations (C & G) were then retested in a followup A/B test. Today, we have finally detected that the C version ("In winkelwagen" translated to "Add to cart") has been implemented on all product pages. Here are my comments about some of these variations.
Highlighted UI Changes From This Leak
Icon Only Version
Here we can see the icon only button which also turns out to be the smallest one. I have a feeling this might have been the control version although I'm not fully certain (as this is one of the first experiments captured on this screen). Overall, given the amount of evidence from past experiments, I'm not surprised that the smallest and icon-only version didn't make it.
Add To Cart
This version with "Add to cart" label looks like it was implemented It could have been the label, or it could have also been the increased width of the button itself.
Interestingly this more direct and end-goal oriented label (Buy Now) didn't manage to outperform the more subtle "Add to cart" label. Perhaps an buy or order now is more suited at the end of the purchase process as a final action?
Add To My Cart
The G variation is comparable to C, in that it has the extra word "my" in "Add to my cart". This makes the label wider (which based on past patterns should contribute to a better result). However, when C and G were tested in a follow up experiment, compared to C, the increase size of the G button and the additional "my" did not deliver enough signal to outperform C.
Yes, I Want This
This button label starts with a strong affirmative "Yes". This version did not manage to deliver the best result and was also rejected. The most comparable experiment to this variation is this test with a similar label change (but soon we'll turn this into a separate pattern as we have more expeirments like this in the pipeline). So far, it doesn't look like startign with "Yes ... " is a good idea.
On Confounding Variables - Towards A Better Experiment Design
If we look more closely at this experiment we can see one little problem - it's not just a label change. In fact, all variations in Bol's experiment introduced two variables: label changes AND button size changes. So unfortunately, if we truly wish to infer anything about button labels or button sizes, we have a classic case of confounding here. Assuming we wanted to learn about these variables, a slightly better setup would have been to isolate one change per variant (ex: with a fixed button width).
Grouping multiple changes together is still a good strategy when you already have past data and are designing a bigger experiment to maximize gains.