× Patterns Blog BetterUI Review Coaching Courses Datastories

How Certain Are You That Your Idea Will Win?

Tagged As: Optimization

All optimization work starts with you and your degree of certainty about particular ideas. This opens up the door for immediate implementation of ideas when we are truly certain (and accepts that not everything has to be tested). Here is an emerging certainty scale we are beginning to use on projects.

The Optimization Certainty Scale

Certainty Scale
We are starting to use this scale to help us prioritize ideas, as well help to make implementation decisions. Half of the scale is reserved for an initial subjective expression, and the other half requires quality tests to be used to back up a higher certainty claim. The highest certainty mark I’ll ever make may be a 9, which requires three strong supporting winning test, and 1 losing test. This is so, because I believe that even more certainty can be extracted from avoiding something that a relevant losing test uncovered. And why avoid a 10? Because such high certainty closes too many doors. :)

How Do You View Your Own Certainty?

Please share your own thoughts.

Share this story

Posted by Jakub Linowski on Jul 12, 2016

Leave Your Reply

3 Comment threads
2 Thread replies
Most reacted comment
Hottest comment thread
4 Comment authors
Jakub LinowskiScott BarnettbryandavidkPatrick Boens Recent comment authors
newest oldest most voted
Patrick Boens

Interesting, Jakub. However I don’t agree with it, or only partially. I believe that things cannot be tested alone (idea 1, 2 and 3 in your case): they must be tested in a context. I would therefore suggest that the scale you use gets adapted to include that extra dimension.

By the way this resembles Prediction machines. ????

PS: I read all your posts … Keep up with the good work


This linear description could be a great summary of A/B test results

Scott Barnett
Scott Barnett

I think there should be at least 2 dimensions used to determine whether something is tested: 1) certainty, 2) importance.

You’ve covered certainty here and left out importance. I recommend adding it to this conceptual model because for low value items sometimes it’s just best to guess and not both testing. For very high value items, it’s important to test even if you’re 95% it’ll work out because the 5% chance of being wrong is so costly that it’s worth the time/effort to run the test.