Skip to content
Module
Module 3 of 5Lesson 1 of 3~17 min

Hypothesis, primary metric and guardrails

A test whose metric is picked after the fact almost always ends up confirming what the team wanted to hear. This lesson teaches you to frame a testable hypothesis, choose the metric that will drive the decision and set guardrails before launch. You also agree with engineers on what to log so the analysis can be trusted.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to write a testable, quantified hypothesis, choose a sensitive and attributable primary metric, guardrails with their threshold, and specify the exposure event to log.

Questions this lesson answers

  • How do you write a testable hypothesis for an A/B test?
  • How do you choose the primary metric of an A/B test?
  • What are guardrail metrics in an A/B test and how do you set them?

Topics covered

  • A/B test hypothesis
  • primary metric
  • guardrail metrics
  • exposure event
  • experiment protocol

In the glossary

Full glossary

Where it fits

Write the protocol

What must be settled before launching the test so that its result is readable and credible?

Lessons in this module

  1. Hypothesis, primary metric and guardrails (this lesson)
  2. Choose the randomization unit
  3. Compute the sample size and the duration

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.

Related courses