Module
Module 3 of 5Lesson 1 of 3~17 min

Hypothesis, primary metric and guardrails

A test whose metric is picked after the fact almost always ends up confirming what the team wanted to hear. This lesson teaches you to frame a testable hypothesis, choose the metric that will drive the decision and set guardrails before launch. You also agree with engineers on what to log so the analysis can be trusted.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to write a testable, quantified hypothesis, choose a sensitive and attributable primary metric, guardrails with their threshold, and specify the exposure event to log.

Topics covered

  • A/B test hypothesis
  • primary metric
  • guardrail metrics
  • exposure event
  • experiment protocol

Where it fits

Write the protocol

What must be settled before launching the test so that its result is readable and credible?

Lessons in this module

  1. Hypothesis, primary metric and guardrails (this lesson)
  2. Choose the randomization unit
  3. Compute the sample size and the duration

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses