Amazon's Beautifully Designed And Failed Three vs. Two Column Layout Experiment
Amazon found the courage to run a beautiful a/b test where they put their old three column product page layout against a new two column one. Although the two column layout was arguably more beautiful with the addition of white space, margins and shadows, it was nevertheless rejected. Before it was removed we decided to leak it here to ensure its beautiful failure continues to teach us well into the future. Finally, you might also notice that this leap experiment introduced multiple changes. Given that we track the win/lose rates of some of these changes as patterns, we can infer that Amazon introduced a mixed bag of both negative and positive changes at the same time - the exact mistake that often destroys such well-intentioned and beautiful redesigns. Here are some of them:
Highlighted UI Changes From This Leak
Wider & Centralized Product Image - Column One
In the A version, the product photo is more tightly squeezed into the left column (of the three column layout). While the B variation clearly increased the white space around the main product image and also made it more central. This key change begins the shift toward a two column layout.
Wider Product Selection And Add To Cart Box - Column Two
As the product / add-to-cart selection box widened in variation B, it also absorbed a number of other elements. Price information, product size selection and availability information all shifted into the right column above the add to cart button. Was this a positive, flat or negative change? I have no evidence on this, so I do not know.
Lower And Wider Calls To Action
Look at the A version which has more narrow and higher up buttons compared to B. These two changes are a perfect example of very subtle changes which we do have accumulating evidence for. We're starting to understand that lowering buttons is most likely worse, while increasing their size is most likely better - and Amazon has combined such positive and negative changes together, potentially canceling them out.
Lowered Product Description ... And Everything Else
The B variation lowered the product description below the product photo. More so, to minimize lowering all other information (ex: upsells) a partial collapse was introduced with an expandable "More" interaction. Nevertheless all other content below was pushed down. I'll stay neutral on this change as I'm really not sure what kind of impact this might have had, if any.
In the B version, around 15px to 20px of margins were added for better or for worse.
Orange To Black Section Titles
The A version had orange section titles which were changed to black. More contrast? Maybe, maybe not.
Fewer Product Titles
Remember those wider margins? Well, there is no such thing as a free lunch and something had to give. The A version had 8 product titles, whereas the B version lost one. Interestingly we started tracking whether showing more results is better or worse as an isolated pattern.
How To Do Leap Experiments A Little Better?
I hope I didn't discourage anyone from running multiple change experiments. In fact, I strongly believe that companies should run such experiments. The key thing of course is to make a solid assessment of what you know and what you don't know. Afterwards only group the positive probability changes into your big leap variation, while pushing all unknowns into separate variations that can be tested in the future. And how can you tell what has positive or negative probability? Keep track of your winning and losing experiments or use the evidence-based patterns we already track for you from numerous companies that are nice enough to share.