Skip to content
Module
Module 4 of 5Lesson 3 of 3~15 min

Bayesian or frequentist, and which tool to choose

Your experimentation tool shows either a probability of winning or a p-value, and the team isn't always sure which one to trust. This lesson explains what each framework really says, how to read their outputs and how to pick a tool based on your volume, data and budget. A free option is covered so you can start without commitment.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to explain to a team the difference between a frequentist and a Bayesian reading of a test, to read a tool's output in both frameworks, and to choose an experimentation tool suited to your constraints, with a free path.

Questions this lesson answers

  • What's the difference between a Bayesian and a frequentist A/B test?
  • Which free A/B testing tool should a small product team choose?
  • What is variance reduction (CUPED) in A/B testing?

Topics covered

  • Bayesian A/B testing
  • frequentist approach
  • experimentation tools
  • CUPED
  • variance reduction

In the glossary

Full glossary

Where it fits

Analyze the result

How do I read a test result without over-interpreting it or missing a problem?

Lessons in this module

  1. Read a p-value and a confidence interval
  2. Analysis pitfalls: peeking, multiple comparisons, novelty, SRM, Simpson
  3. Bayesian or frequentist, and which tool to choose (this lesson)

What you will learn in this module

This lesson is part of the course Design and analyze an A/B test

  • Analyze a result (p-value, confidence interval, practical significance, guardrails) and read a tool's Bayesian or frequentist output.
  • Detect and avoid analysis pitfalls (peeking, multiple comparisons, novelty effect, sample ratio mismatch, Simpson's paradox).

Related courses