Are Visible Cancellation Graphs Too Much - As Airbnb Learned In Their Failed Experiment?

The cancellation message in the control (A) frames the cancellation as something positive with a lower commitment to act: It reassures people that they have 48 hours for a full refund. Flip to the variant (B) and the story is inversed with people seeing the risk of losing 50% of their deposit within 48 hours. If people are loss averse, then this alone could have dragged the whole experiment down frightening a portion of users away.

The graph shows information with two reference points which is ripe for confusion. First, readers are asked to evaluate "their eligible scenario" from the present time of booking (see the "Reservation confirmed" reference on the left with the "48 hours later".). Then, at the same time information is also shown relative to the future check-in time (see "Check-in" on the right). This creates immense ambiguity. For example, if a check-in date is tomorrow but the reservation was made today, are users eligible for a full refund or no refund at all?

A better approach would have been to pick a single time frame and display all information accordingly to avoid such confusion.

When information is presented relatively as 14 or 7 day references, users need to do the intense calculations themselves. It would have been more beneficial if specific dates and times were added to clarify the key date points. For example: "7 day before check-in" could also have appended a specific date and time such as "Fri June 7th, 14pm+". I suspect that hour references in this contexts could be important to clarify.

Notice that the time labels are shown as points along a line and yet the actual eligibility is a time frame. More so, the above variation also doesn't follow a consistent application of the labels (the "48 hours" label applies leftwards and the "7 days" label applies rightwards). This is a case of informational mismatch. I suspect that the labels should be aligned closer together with their corresponding descriptions for even less ambiguity.

Overall, given that experiments run for weeks, it probably is worth to fine tune them a bit more before launch. This can increase their odds of success and minimize failures (easier said than done). With some adjustments perhaps this experiment might still have some remaining potential for a greener outcome. :)