Module
Module 3 of 5Lesson 3 of 3~24 min

Compute the sample size and the duration

Knowing how many units and weeks an A/B test requires keeps you from launching a test that is doomed from the start or stopping it too early. You learn to estimate the sample size for a rate-based test, by hand and then with a free calculator, and to derive a realistic duration. You also know which options remain when the math exceeds your volume.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to compute by hand and with a free calculator the sample size of a test on a rate, from the baseline rate, the minimum detectable effect, the significance level and the power, to derive a duration in full weeks, and to know what to do if it is out of reach.

Topics covered

  • sample size
  • A/B test duration
  • minimum detectable effect
  • statistical power
  • A/B test calculator

Where it fits

Write the protocol

What must be settled before launching the test so that its result is readable and credible?

Lessons in this module

  1. Hypothesis, primary metric and guardrails
  2. Choose the randomization unit
  3. Compute the sample size and the duration (this lesson)

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses