Visualizing A/B Test Effect Trends
A first sketch of an effect-trend visualizer: instead of a single end-of-test number, each test's daily effect is shown as it actually moved — noisy readings, a fitted trend line, and an oversimplified "shape" guess on top, so patterns like a fading novelty effect or a slow ramp-up are visible at a glance.
The Prototype
Stable
Effect holds steady around the same lift for the whole test.
Novelty effect
Strong at launch, fades as the new-thing appeal wears off, ends flat or negative.
Gradual growth
Starts weak or negative, strengthens as users adapt or word spreads.
Gradual decay
Starts positive, steadily erodes toward flat or negative — no early spike like a novelty effect.
Level shift
Holds at one level, then jumps to a new one — often a sign something external changed mid-test.
Reversal
Crosses from one direction to the other and back — no stable sign to read a verdict from.
Where This Came From
This started as a notebook sketch while thinking through how a test's effect can look wildly different depending on when you check it — stable, a fading novelty effect, a slow ramp-up, or a level shift are all "the same" single end-of-test number today, but very different stories about what's actually happening.
Posted by 
Jakub Linowski on Sep 19, 2026
Comments