Skip to content
Module
Module 2 of 5Lesson 1 of 2~14 min

What a controlled test proves, and what an analysis does not

A metric that improves after a launch does not prove your feature caused it. This lesson explains what a controlled test establishes that everyday comparisons cannot, then sets up a shared vocabulary with data teams. You can then challenge a shaky impact claim before it steers the roadmap.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to explain why a pre/post comparison or a comparison between users and non-users does not measure the effect of a change, what randomization brings, and to use the vocabulary of an A/B test correctly (control, variant, unit, exposure).

Questions this lesson answers

  • Why doesn't a before and after comparison prove a product change caused the result?
  • What does a controlled A/B test prove that ordinary comparisons cannot?
  • What do control, variant and exposure mean in an A/B test?

Topics covered

  • A/B testing
  • causality
  • randomization
  • before-after comparison
  • control group

In the glossary

Full glossary

Where it fits

Should you test?

What does an A/B test prove that an analysis does not, and when is it better to do without one?

Lessons in this module

  1. What a controlled test proves, and what an analysis does not (this lesson)
  2. When not to test, and what to do instead

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.

Related courses