Running Your First A/B Test - A Draft Course Outline

We have moved forward with a draft outline for the optimization course we're preparing and we'd love to hear your feedback. Last week some of you already shared some questions to steer us and then we also validated the need for such the course itself. Thanks for that. This week, here is another update.

Draft Course Title: Running Your First A/B Test

And we already have a really cool company which we'll work with to optimize their site as the core subject. :)

The Tools We'll Use In The Course

Some of the tools we're thinking of using in our course include: VWO for A/B testing, Evan Miller's Sample Size Calculator, Thumbtack's ABBA Split Test Calculator, Google Docs for idea prioritization, GoodUI Evidence for testing inspiration, and Trello for test management. That's at least what we have been using for the last few years while a/b testing.

The Course Outline

  • Basic Considerations
    • Good Optimization Work Defined
    • Certainty - The Currency Of Optimization
    • Understanding Chance & Variance
  • Day 1 - Setting Focus
    • Setting Primary Metrics & Targets
    • Optimizing For Lift Or Insight
    • AVOID: Shallow Metrics
  • Day 2 - Tracing The Funnel
    • Starting a Tracer With A Tiny Change
    • AVOID: Testing Downtime
  • Day 3 - Identifying Testing Opportunities From 3 Sources
    • Using Existing Analytics
    • Using Customer Insights
    • Using Past A/B Tests
  • Day 4 - Estimation & Prioritizing Opportunities
    • Certainty, Effect, Effort, Duration
    • Checking Past A/B Tests / Historical Data
  • Day 5 - Concepting & Designing The Test Structure
    • Sketching Ideas
    • Anatomy Of An A/B Test
    • Testing Strategies
    • AVOID: The First Idea
  • Day 6 - Test Setup
    • VWO
    • Secondary Goals
    • Screenshots & Managing Tests
  • Day 7 - QA - Before It’s Live
    • QA Checklist
    • Browser Testing
  • Day 8 - QA - After It’s Live
    • Signs Of Trouble
  • Day 30 - Stopping The Test
    • Dynamic Stop Rules
    • Checking Margins Of Error
    • Checking Secondary Metrics For Consistency
    • Checking Time Graphs For Consistency
    • AVOID: Stopping With p-values Alone
  • Day 32 - Extracting Findings
    • Identifying Findings Or New Hypothesis
    • Dealing With Weak Results
  • Day 33 - Post Test Considerations
    • Implementation
    • Healing Broken URLs

Please Tell Us Your Thoughts About The Outline

What do you think about the above outline? Do you wish for us to focus on anything else during the course? Now is the time to let us know. (The "days" are just used for reference. We might decide to refer to actual days from the real project.)


  • Carolina Avila Coral

    Carolina Avila Coral 8 years ago 00

    Hi Jakub:

    I think it will be an amazing course.
    I'm not native English speaker, so, I hope you save the class in a video so I can watch the class many times until understand, in addition, I will appreciate a lot if the videos could have subtitles or a closed caption.

    Thanks a lot.

  • Jennifer McKay

    Jennifer McKay 8 years ago 00

    Jakub, I'm really impressed with what you're doing here (with goodui). This class structure sounds really strong and I'm very interested in taking this. I really appreciate the community here.

  • Thomas Tonder

    Thomas Tonder 8 years ago 10

    Looks really great !
    Would love to see something about the tools you use to work as a team during the process (ideas backlog, bugs/correction tracking, global project management, reporting of results...)
    Thanks for sharing with so much passion :)

  • Ruggero

    Ruggero 8 years ago 00

    Great plan!
    I'd like to deep the tools sections: how to use them and how to correctly read the results of the tests.

  • David Frahm

    David Frahm 8 years ago 00

    I would want this to touch on how to know when to stop. Might be a bit in the Setting Focus as far as what type of gains you are looking for, and then in the post analysis how to know whether to go again/deeper on the same test or move on. Basically discussing incremental vs exponential gains.

    • Jakub Linowski

      Jakub Linowski 8 years ago 00

      Agreed, that this could be interesting. We are also actively researching this and have tried to stir a discussion on stopping rules right here: Care to share your stopping rules if you have any?

  • Michael Facchinello

    Michael Facchinello 8 years ago 00

    I am confused how it jumps from day 8 to day 30. A bit of clarification around that could be good.

    • Jakub Linowski

      Jakub Linowski 8 years ago 10

      Hi Micahel. I see how that can be confusing. I think I should have added that we will be using a real life optimization project to show exactly how/what we optimize ... and hence we'll be using a day/day breakdown. By skipping to day 30, I think I wanted to imply that there would be some waiting time for the test to run and gather data. We actually don't know how long the test will take yet at this point. This will be a new project. Does this help?

  • Joshua

    Joshua 8 years ago 00

    Looks amazing Jakub! I really hope you consider LifterLMS when you go to build it :-) I'd love to make a custom theme for you :-)

  • Robert Brown

    Robert Brown 8 years ago 10

    When will you do this and how much will it cost? Will it be a webinar course? We actually have a web application product.

  • Dominick Branigan

    Dominick Branigan 8 years ago 00

    I think this is going to be great and I look forward to the course. I would love to see specific A/B testing such as, visitor-to-lead conversion rate, maybe landing page, email, or call-to-action A/B test. As I very new to A/B testing it would also be cool to see commonly tested elements

  • Stacy Sutton Williams

    Stacy Sutton Williams 8 years ago 00

    I would love to have my team take this. How much will it cost?

    • Jakub Linowski

      Jakub Linowski 8 years ago 00

      Hi Stacy. We still haven't decided on the pricing. It might be offered as part of Datastories, is one idea.