Skip to content
Module
Module 5 of 5Lesson 1 of 2~14 min

Decide by written rules and document the experiment

Without a rule set in advance, every A/B test result ends up validating the team's initial hunch. The lesson shows you how to decide whatever the outcome, even an inconclusive one, and how to plan what comes next. You record each experiment in a short memo the whole organization can reuse for its next tests.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to apply a decision rule written before the test to a conclusive, negative or inconclusive result, to plan what comes next (rollout, long-term holdout, new iteration), and to write a reusable experiment memo.

Questions this lesson answers

  • How do you decide after an inconclusive A/B test result?
  • How do you write a decision memo after an A/B test?
  • Why keep a holdout group after rolling out an A/B test winner?

Topics covered

  • decision rule
  • inconclusive result
  • experiment memo
  • holdout group
  • test documentation

In the glossary

Full glossary

Where it fits

Decide and capitalize

How do I turn a result into a decision, and the test into lasting learning?

Lessons in this module

  1. Decide by written rules and document the experiment (this lesson)
  2. Final workshop: your experiment's protocol, analysis and memo

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses