GoodUI BLOG - Back To All Posts

Why Did Our Best Variation Fail?

Tagged As: Question

We just completed a three test project for www.checkineasy.com – an event planning & management platform. The tool allows event greeters to check in their guests very quickly using one or more mobile devices. The platform requires a free application download, a free account, and guest credits which can be purchased later. When we set out to optimize the existing signup screen for more signups (of course), our initial reaction was that stylistically it was quite busy. We started asking ourselves if by simplifying the user interface, and applying a handful of best practices, higher signup rates could be reached. It wasn’t long before our first variation launched and flopped. Here is what we’ve done in variation B:

The Two Signup Pages That We Tested In Test 1 Of 3

Changes

1. Giving A Gift

The 50 Guest Credits were emphasized as something users would receive. (GoodUI #2).

2. Explaining

The pricing and credit structure was explained in depth (replacing 3 existing benefits). (GoodUI #68).

3. Top Aligned Labels

Form labels were placed above the form fields.

4. Fewer Form Fields

The username field was removed (with the email address being passed on instead). (GoodUI #13).

5. Bigger Click Areas

The size of all buttons and form elements was increased. (GoodUI #38).

6. Keeping Focus

The standard footer with additional links was removed. (GoodUI #16).

7. Social Proof

Logos of existing customers have been shown in the footer. (GoodUI #4).

So Why Did All Of These Great Ideas Fail? You Tell Us First!

Yes, variation B which contained a number of our supposedly “best practices” resulted in a -21% drop to signups. It is not uncommon for what we think will perform well, to have an opposite effect – that’s why we tested it. The key take away of course is that it’s important not to give up on the first try. That’s exactly what we did with the project as we designed two more tests to answer this question. But before we share them with you, we’d love to hear what you think as to what the cause might have been. Please share in the comments section.

Update [Nov 5]: One More Variation To Provide Additional Clues

Like many of the reader comments have already pointed out, we also started speculating that the control’s headline might have been more effective for various reasons. The 50 guest credit gift might have been irrelevant or even causing anxiety. More so the control’s reference to “trial” instead of “account”, might have carried more value. Finally, variation B also might have been showing a conflicting message between receiving a “FREE Account” and then showing overwhelming amount of pricing related messaging. Reverting variation C slightly back to the control, we then turned it into a test. As the data started coming in (and we were acting on thin data in the interest of time), the new variation C started performing better at an insignificant -11% (ranging widely between -32% to 11% with a p-value of 0.43). This was good news as the needle started shifting closer towards the control. It began confirming that some of the changes in variation B were in fact hurting and we were beginning to identify why. We did not stop here. We ran two more variations, one of which finally generated a significant win.

Changes C


Read The Complete Story In Issue #19 + With 2 More Variations

Be sure to grab the full story where we show in detail how we managed to score a +15% signup rate with two more follow up tests. The issue also includes: additional screens that were tested, the testing strategy used (Making Big Leaps), our reflections on the process, transparent data, and a blueprint template. Issue #19 now also comes with day snapshots which show exactly what we did from one day to another:

Day Stamps


Tweet about this on TwitterShare on Facebook0Share on Google+1Share on LinkedIn0Buffer this pageEmail this to someone

Posted by Jakub Linowski on Nov 3, 2015

Leave Your Reply

29 Comments on "Why Did Our Best Variation Fail?"

avatar
Sort by:   newest | oldest | most voted
Andrius | Adguns

Free credits … Buy credits. Confuses. Let’s try to remove the Buy thing or soften explanation about Credits.
And there is no explanation about the product, what is it, why to use it, not clear the product itself.

Michael Facchinello

The content is not nearly as good in variation 2. If you were to test variation 2 with variation 1’s copy (specifically UI 1 & 2 as labeled on variation 2), it would win.

vicente

#1 is giving me a free trial [for a possibly fantastic check in app. Worth the try]

#2 is giving me a free account [for what? again?]

No short explanation of the service on #2 in my opinion. Seems like no one reads further than a bullet list on internet..

My 2 cents

Vicente Ocana

Matheus

The B test looks awesome and I can only guess what could drop the signups.

The A test has an arrow in a blue area, very attractive by the way. That arrow draw attention to the form on the left side of the screen, making users signup.
On the other side, the B test has only the big blue button “Create Your Account” on the bottom side and a text area describing the benefits. Maybe people are reading the full text and giving up to complete the form.

This is only a guess.

Vlad Malik

I always thought that blue arrow box looked like an ad. And the arrow inside is inverted on a dark background with a box boundary separating it from the form it’s pointing to. I assumed people would ignore it. A few commenters liked the arrow… curious if it’s post-hoc analysis on our part or if new visitors really did notice it.

Peter Murray

I suspect the failure is due to users not understanding what they are signing up for. “Create your free account and receive 50 guest credits” for what?

The A version has the content “ENTER HERE for an awesome check-in experience” which tells the user why they are registering.

Bill

Call to Action is ‘Create an Account’. Who wants to do that? I’d rather have an awesome experience than another account.

Adam Szabo

I personally like the changes made except number 2 (explaining).
Too much text might distract visitors from the main goal.

I think the four headlines would have been enough without the explaining text.
I would have also put the signup form in the center.

David
Signing up for a free trial gives the impression that you will get the full range of features available, but for a limited time. It allows users to experience & evaluate the full value of the service before committing any money to it. In Test B, this changes completely. A free account is available, but the scope of whats possible with it are limited by credits. If I was contemplating signing up for this service, I’d like to see whats possible with 50 credits, what features are available, and how much they cost (and I’d want to know all of… Read more »
Michael Sallander

Like many others also pointed out on variation B #1 and #2 are likely hurting it. If you kept everything else about b the same but reverted the copy change from #1 and #2 back to the original, I think you would see better results.

Emils_W

Probably it has something to do with ”free trial” vs ”free account and 50 guest credits”, the core offer.

Maybe the users didn’t find the gift (50 guest credit) valuable at all, just confusing.

But that’s just a guess, I would do some user research and find is this out for sure.

Sara
1. Commitment: creating an account, even when it’s free, is somehow more overwhelming than starting a free trial. 1.b 50 credits wiil be enough for me? What does 50 credits mean? What can I do with them? Stating a limited amount of credits brings more confusion than clarity. Sign up pages are for signing up, not for formulating new questions. 2. Too much info. People here want to be reassured. 3 bullets points that summarize what they found in the landing page/website should be enough. 3. Organization name: what if I don’t want to involve my organization at this time?… Read more »
Nenad S.
Point 1 – Clients who need this software don’t care about the money as much as reliability, so offering free credits makes them feel you’re selling an app, not reliability. Plus, offering free stuff isn’t in line with premium services and offers. If their clientele is of that type, that’s an additional turn-off. Point 2 – There’s too much text, it takes away the focus from the form that needs filling; the users don’t want to read at that point, but go through the process. A better solution would probably have been to provide plain links or snippets BELOW the… Read more »
Delmania

When I look at variation A I feel like I need to sign up. The big arrow pointing the sign-up form, the short lists of benefits almost compels me to register. Variation B just seems more passive to me.

Vlad Malik

“Passive” is a neat qualifier for this variation. I do think it’s more passive, because there is too much happening. It’s a signup form trying to be a landing page. Sometimes that works though, because people really can land on a from directly, and some landing pages are just glorified forms (“Buy my book” etc). But I think it’s best avoided. Forms are functional. We tried replacing this form with an inline form on the home page in one test. Jakub, did we write about that somewhere? Seems relevant.

Richard Vaughan

I think the mention of buying credits and showing the pricing structure a on Var B was the thing that affected signups. It would be interesting to see what the upsell and/or churn was on the variations, it may in the long run not be a negative thing to deter those who are scared away from the prospect of spending money.

John Henry Müller

As a new user, I don’t have context to what “guest credit” means. So a gift of 50 doesn’t seem like a gift at all. If fact, this new currency promotes an unknown and causes anxiety.

I know what “Start your free trial” means. Try it, then like it or leave it. No risk.

Dave Mulder

I agree with the comments about the account vs free trial language, as well as the surprising introduction of “guest credits”. It’s a gift, but it’s like trying to incentivize someone with a mystery box.

I think there’s also something to the checkmark bullets in the original version. “30-second signup”, “lightning fast check-ins”, and “amazing first impression”. They reinforce the idea that there’s something remarkable worth signing up for.

Victor Bejar

Definitely there is more cognitive load for the user. “Free account, with 50 GUEST credits” is hard to understand. “Free Trial” is easy to understand. I would have to read the description to really understand how things work. Option A benefits are clear and easy to ready. Option A form is more conventional (username, password, email).

Jakub Linowski

By the way … a huge thanks so far for sharing all these amazing reasons why it flopped. Update still coming in a few days.

Vahur Metsala
I’d go with everyone suggesting the #1 and #2 failure. This is a very personal and subjective opinion, but “credits” in general seem a complicated thing. It’s like an online game or gambling thing and I wouldn’t want to deal with credits VS a simple monthly plan… Seems like one of those instances where the company jargon and user benefits have been confused.. i.e. 50 credits is a lot from the company’s perspective but what does that mean to a user? 50 free check-in’s? 100 free guests? As a user, I would want to know the benefit in my terminology…… Read more »
Andy

To me, the first one is a hands down winner, even if it was “busy”. Even with all of the business, it was very clear to

1.Start your free trial
2. Enter here following the left pointing arrow to the form.

The second variation was a page of sameness with nothing standing out giving me direction. There was no apparent flow. I did not even read it. Repeat, I did not even read it.

Now with both, I am not sure what you are offering, but I probably would have known if I was directed via a good ad, search or sale funnel.

Andy

Now that I am thinking about it instead of giving you a knee jerk reaction, I need to add, it just might be the headline. It confused me totally, and when I looked at the rest of the page, all of the things that I stated above became true.

Peter

Visually the form too big. Hard to take it In all at once. Looks like a screaming used car-dealer ad.
While button was bigger, it was too wide and didn’t look like a button but yet another ad.

Alex

I wonder if it has something to do with the page being split into 2 columns. Perhaps that’s one too many!

Mia

Perhaps if the CTA button were more button-y and less advertising banner-y it would tweak the results…?

Jason
Interesting post and seems like good information on best practices. I just wish that you would use them on your own site! The bottom of the article has a call to action – literally a big red link – that says “Read The Complete Story In Issue #19”. However, when I follow that link I don’t get to read the complete story in issue 19. I don’t even get to issue 19. Instead I go to a page that seems entirely unrelated promoting something called “DataStories” that I’ve never heard of before. Its only after scrolling 2 page lengths and… Read more »
wpDiscuz
Hire Us To Optimize Your Site
We can lift your metrics and prove it.