Module
Module 5 of 5Lesson 1 of 2~14 min

Decide by written rules and document the experiment

Without a rule set in advance, every A/B test result ends up validating the team's initial hunch. The lesson shows you how to decide whatever the outcome, even an inconclusive one, and how to plan what comes next. You record each experiment in a short memo the whole organization can reuse for its next tests.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to apply a decision rule written before the test to a conclusive, negative or inconclusive result, to plan what comes next (rollout, long-term holdout, new iteration), and to write a reusable experiment memo.

Topics covered

  • decision rule
  • inconclusive result
  • experiment memo
  • holdout group
  • test documentation

Where it fits

Decide and capitalize

How do I turn a result into a decision, and the test into lasting learning?

Lessons in this module

  1. Decide by written rules and document the experiment (this lesson)
  2. Final workshop: your experiment's protocol, analysis and memo

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses