Module 3 of 6Lesson 1 of 2~22 min
Build a representative test set
How to build a test set that looks like your users' real requests, including rare cases and attempts to misuse the feature. It is the foundation of every decision about an AI feature: a poorly built set yields reassuring scores on cases that don't matter, and its size drives the effort you negotiate with the data team.
Lesson objective
By the end of this lesson, you will be able to compose a test set from real traffic, edge cases and adversarial cases, set its starting size and justify its split.
Questions this lesson answers
- How do you build a test set that mirrors real user traffic?
- Why include edge cases and adversarial cases in an AI test set?
Topics covered
- AI test set
- golden dataset
- stratified sampling
- edge cases
- adversarial cases
- synthetic data
In the glossary
Where it fits
Build the test set
Which cases do you test on, and how do you keep the set reliable over time?
Lessons in this module
- Build a representative test set (this lesson)
- Label, version and protect the test set
What you will learn in this module
This lesson is part of the course Evaluate an AI feature: test sets, metrics and LLM judges
- Build a representative test set from real traffic, edge cases and adversarial cases, and justify its composition.
- Label the test set with a guide, measure inter-annotator agreement, version it and protect it from overfitting.
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductExpertAdvanced level: for deeper mastery and specialization.
Ship and monitor an AI feature in production
3 hr