Skip to content
Module
Module 4 of 5Lesson 2 of 3~20 min

Analysis pitfalls: peeking, multiple comparisons, novelty, SRM, Simpson

A well-designed test can still lead to a false conclusion if its analysis falls into one of the classic traps. The lesson teaches you to spot them in a result or a write-up, check how units were split between groups and pick the right remedy. You avoid shipping a variant that only won by chance.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to spot peeking, multiple comparisons, a novelty effect, a sample ratio mismatch or a Simpson's paradox in a test or its report, to check a mismatch by calculation, and to choose the right countermeasure.

Questions this lesson answers

  • Why shouldn't you stop an A/B test as soon as it becomes significant?
  • What is a sample ratio mismatch in an A/B test?
  • How do you spot a novelty effect or Simpson's paradox in test results?

Topics covered

  • peeking
  • multiple comparisons
  • novelty effect
  • sample ratio mismatch
  • Simpson's paradox

In the glossary

Full glossary

Where it fits

Analyze the result

How do I read a test result without over-interpreting it or missing a problem?

Lessons in this module

  1. Read a p-value and a confidence interval
  2. Analysis pitfalls: peeking, multiple comparisons, novelty, SRM, Simpson (this lesson)
  3. Bayesian or frequentist, and which tool to choose

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).

Related courses