Running Your First A/B Test - A Draft Course Outline
We have moved forward with a draft outline for the optimization course we're preparing and we'd love to hear your feedback. Last week some of you already shared some questions to steer us and then we also validated the need for such the course itself. Thanks for that. This week, here is another update.
Draft Course Title: Running Your First A/B Test
And we already have a really cool company which we'll work with to optimize their site as the core subject. :)
The Tools We'll Use In The Course
Some of the tools we're thinking of using in our course include: VWO for A/B testing, Evan Miller's Sample Size Calculator, Thumbtack's ABBA Split Test Calculator, Google Docs for idea prioritization, GoodUI Evidence for testing inspiration, and Trello for test management. That's at least what we have been using for the last few years while a/b testing.
The Course Outline
-
Basic Considerations
- Good Optimization Work Defined
- Certainty - The Currency Of Optimization
- Understanding Chance & Variance
-
Day 1 - Setting Focus
- Setting Primary Metrics & Targets
- Optimizing For Lift Or Insight
- AVOID: Shallow Metrics
-
Day 2 - Tracing The Funnel
- Starting a Tracer With A Tiny Change
- AVOID: Testing Downtime
-
Day 3 - Identifying Testing Opportunities From 3 Sources
- Using Existing Analytics
- Using Customer Insights
- Using Past A/B Tests
-
Day 4 - Estimation & Prioritizing Opportunities
- Certainty, Effect, Effort, Duration
- Checking Past A/B Tests / Historical Data
-
Day 5 - Concepting & Designing The Test Structure
- Sketching Ideas
- Anatomy Of An A/B Test
- Testing Strategies
- AVOID: The First Idea
-
Day 6 - Test Setup
- VWO
- Secondary Goals
- Screenshots & Managing Tests
-
Day 7 - QA - Before It’s Live
- QA Checklist
- Browser Testing
-
Day 8 - QA - After It’s Live
- Signs Of Trouble
-
Day 30 - Stopping The Test
- Dynamic Stop Rules
- Checking Margins Of Error
- Checking Secondary Metrics For Consistency
- Checking Time Graphs For Consistency
- AVOID: Stopping With p-values Alone
-
Day 32 - Extracting Findings
- Identifying Findings Or New Hypothesis
- Dealing With Weak Results
-
Day 33 - Post Test Considerations
- Implementation
- Healing Broken URLs
Please Tell Us Your Thoughts About The Outline
What do you think about the above outline? Do you wish for us to focus on anything else during the course? Now is the time to let us know. (The "days" are just used for reference. We might decide to refer to actual days from the real project.)
Posted by 
Jakub Linowski on Mar 24, 2016
Comments
Carolina Avila Coral 8 years ago ↑0↓0
Hi Jakub:
I think it will be an amazing course.
I'm not native English speaker, so, I hope you save the class in a video so I can watch the class many times until understand, in addition, I will appreciate a lot if the videos could have subtitles or a closed caption.
Thanks a lot.
Carolina
Reply
Jennifer McKay 8 years ago ↑0↓0
Jakub, I'm really impressed with what you're doing here (with goodui). This class structure sounds really strong and I'm very interested in taking this. I really appreciate the community here.
Reply
Thomas Tonder 8 years ago ↑1↓0
Looks really great !
Would love to see something about the tools you use to work as a team during the process (ideas backlog, bugs/correction tracking, global project management, reporting of results...)
Thanks for sharing with so much passion :)
Reply
Ruggero 8 years ago ↑0↓0
Great plan!
I'd like to deep the tools sections: how to use them and how to correctly read the results of the tests.
Reply
David Frahm 8 years ago ↑0↓0
I would want this to touch on how to know when to stop. Might be a bit in the Setting Focus as far as what type of gains you are looking for, and then in the post analysis how to know whether to go again/deeper on the same test or move on. Basically discussing incremental vs exponential gains.
Reply
Jakub Linowski 8 years ago ↑0↓0
Agreed, that this could be interesting. We are also actively researching this and have tried to stir a discussion on stopping rules right here: http://goodui.org/blog/agile-ab-testing-using-stop-rules-to-minimize-losses-time-wasted/ Care to share your stopping rules if you have any?
Reply
Michael Facchinello 8 years ago ↑0↓0
I am confused how it jumps from day 8 to day 30. A bit of clarification around that could be good.
Reply
Jakub Linowski 8 years ago ↑1↓0
Hi Micahel. I see how that can be confusing. I think I should have added that we will be using a real life optimization project to show exactly how/what we optimize ... and hence we'll be using a day/day breakdown. By skipping to day 30, I think I wanted to imply that there would be some waiting time for the test to run and gather data. We actually don't know how long the test will take yet at this point. This will be a new project. Does this help?
Reply
Joshua 8 years ago ↑0↓0
Looks amazing Jakub! I really hope you consider LifterLMS when you go to build it :-) I'd love to make a custom theme for you :-)
Reply
Robert Brown 8 years ago ↑1↓0
When will you do this and how much will it cost? Will it be a webinar course? We actually have a web application product.
Reply
Dominick Branigan 8 years ago ↑0↓0
I think this is going to be great and I look forward to the course. I would love to see specific A/B testing such as, visitor-to-lead conversion rate, maybe landing page, email, or call-to-action A/B test. As I very new to A/B testing it would also be cool to see commonly tested elements
Reply
Stacy Sutton Williams 8 years ago ↑0↓0
I would love to have my team take this. How much will it cost?
Reply
Jakub Linowski 8 years ago ↑0↓0
Hi Stacy. We still haven't decided on the pricing. It might be offered as part of Datastories, is one idea.
Reply