Skip to content
Module
Module 2 of 5Lesson 2 of 2~18 min

When not to test, and what to do instead

Running an A/B test without enough volume means spending weeks on a result that settles nothing. The lesson helps you judge whether a test is feasible, recognize situations where it doesn't belong and choose another method, especially in B2B. You still make the call, knowing what your approach demonstrates and what lies beyond it.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to estimate whether the available volume allows an A/B test (minimum detectable effect), to recognize the other cases where a test is not the right method, and to choose a suitable alternative, especially in B2B.

Questions this lesson answers

  • When should a Product Manager decide not to run an A/B test?
  • How do you know if you have enough volume for a B2B A/B test?
  • What alternatives exist when an A/B test isn't possible?

Topics covered

  • minimum detectable effect
  • statistical power
  • A/B testing alternatives
  • B2B experimentation
  • progressive rollout

Where it fits

Should you test?

What does an A/B test prove that an analysis does not, and when is it better to do without one?

Lessons in this module

  1. What a controlled test proves, and what an analysis does not
  2. When not to test, and what to do instead (this lesson)

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Decide whether an A/B test is the right method (volume, B2B, reversibility, time, ethics) and choose a suitable alternative otherwise.

Related courses