A +58,782% More Annoying Modal Popup
During a recent a/b test, we discovered our highest effect to date and it wasn't pretty. The record breaking increase of +58782% appeared for the number of people who clicked "No Thanks" and/or "Close" on a modal popup between these two tested variations.
Coming back down to earth, the astronomical +58782% effect really translated into an absolute rate of 72% of people who clearly did not wish to see the modal popup - perhaps that's a little easier to chew on. 72% is still high and suggests a major irritation. Part of the explanation was that our modal window variation was shown after a fixed 6 second delay. More so, the popup window was floating, so no amount of scrolling would have removed it (although I'd say that's pretty common). These two factors combined easily could have explained the large amount of people who might have expressed their annoyance.
Relatively speaking, was this a fair comparison? Probably not. The control variation actually did not have a "No Thanks" button and only a smaller "X" to close its floating footer. Nevertheless, the footer in the control only had an absolute close rate of around 0.1%.
Did We Learn Anything From Any Of The 5 Variations?
Yes. For one, we think there might be a more intelligent way to time modal windows based on behavior (ex: scroll or mouse movement rules). Secondly, we also discovered a way better interaction pattern with a slide element positioned at the very top of the screen which we'll be falling back on from now on. Here are the exact 5 variations that were tested, while focusing on the primary metric of trying to increase the "signup" rate.
Get the full story described in Issue #21 to learn everything we learned from this amazing test.