Module
Module 3 of 5Lesson 2 of 3~14 min

Choose the randomization unit

Randomizing by user, account or session affects a test's validity as much as its power. The lesson helps you weigh a coherent experience for every user, the risk of interference between groups and the number of units available. A critical choice in B2B and marketplaces, best discussed early with the engineering team.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to choose the unit to randomize (user, account, session, business object, cluster) by weighing consistency of experience, interference and number of units, and to spot the B2B and marketplace cases where groups influence each other.

Topics covered

  • randomization unit
  • interference
  • B2B A/B testing
  • marketplace
  • cluster randomization

Where it fits

Write the protocol

What must be settled before launching the test so that its result is readable and credible?

Lessons in this module

  1. Hypothesis, primary metric and guardrails
  2. Choose the randomization unit (this lesson)
  3. Compute the sample size and the duration

What you will learn in the course

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.
  • Write a testable hypothesis, choose a sensitive, attributable primary metric, guardrails and the exposure event to log.
  • Choose the randomization unit and prevent interference between groups, including in B2B.
  • Compute a sample size and a duration from the baseline rate, the minimum detectable effect, the significance level and the power.
  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).
  • Decide based on rules written before the test and document the experiment in a reusable memo.

Related courses