Module
Module 5 of 5Lesson 2 of 2~22 min

Final workshop: your experiment's protocol, analysis and memo

The final workshop applies the approach to your own product, from the experiment protocol through to the decision memo. If volume falls short, a reasoned alternative plan takes the place of the test, and a rubric helps you judge how solid the whole package is before you share it with engineers and the data team. You leave with a reusable kit and a plan to launch your first test.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will have written the protocol of an experiment on your product (or a reasoned alternative plan), analyzed a real result or the practice dataset provided, written the decision memo, self-assessed the whole with a grid, and you will leave with an exit kit and a 7-day and 30-day plan.

Topics covered

  • A/B testing workshop
  • experiment protocol
  • decision memo
  • self-assessment rubric
  • experimentation kit

Where it fits

Decide and capitalize

How do I turn a result into a decision, and the test into lasting learning?

Lessons in this module

  1. Decide by written rules and document the experiment
  2. Final workshop: your experiment's protocol, analysis and memo (this lesson)

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses