A/B test
An A/B test is a controlled experiment in which users are randomly assigned to two or more versions of a product experience, and a metric chosen in advance is compared between the groups. Because assignment is random, a difference large enough to rule out chance can be attributed to the change itself, which a before-and-after comparison cannot do. Version A is usually the current experience, called the control, and B the variant.
Why it matters for a PM
A/B tests let PMs settle debates with evidence from real behavior and measure the actual size of an effect. They also have costs and limits: they need enough traffic, take time, and answer only the question they were designed for. Knowing when not to run one is part of the skill.
Example
An online store tests a shorter checkout form. Visitors are split at random between the current form and the new one, the primary metric is completed orders per visitor, and refund rate is a guardrail. The sample size calls for three weeks, and the team commits to waiting until then.
Key points
- Before launch, put in writing the hypothesis, the main success measure and the metrics that must not degrade.
- Choose the randomization unit (user, account, session) so that one group's behavior cannot spill over into the other.
- Compute the sample size from the smallest effect worth detecting, then run the test to its planned end.
- Decide with rules written in advance, including what happens when the result is inconclusive.
Common mistakes
- Stopping the test as soon as the result looks significant, a practice known as peeking.
- Testing with too little traffic to detect any realistic effect.
- Tracking dozens of metrics and celebrating the one that moved.
- Ignoring a sample ratio mismatch, a sign that assignment or tracking is broken.
Go further with Module
The courses and lessons that cover this concept: