Module
Module 1 of 5Lesson 1 of 1~4 min

Course overview

This course is for product managers who want to know, through A/B testing, whether a product change truly made a difference. You learn to decide whether to test, write a sound protocol, analyze the result and make the call in writing. The running case is a B2B product, where limited volume often makes the exercise harder.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this overview, you will know what you are going to produce (the protocol of an experiment on your product, its analysis and a decision memo), which two questions Atelio will settle throughout the course and which four modules take you there.

Topics covered

  • A/B testing
  • product experimentation
  • experiment protocol
  • statistical analysis
  • product decision

Where it fits

Overview

What will I be able to do by the end of this course, and in what order?

Lessons in this module

  1. Course overview (this lesson)

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses