Track quality in production
An AI feature that passed your evaluations before launch can degrade when it meets real usage. This lesson teaches you to combine several quality signals in production, notice when users' requests start to shift, and turn the failures you observe into new test cases for your evaluation set.
Lesson objective
By the end of this lesson, you will be able to combine three quality signals in production (user feedback, sampled human review, online evaluations), spot input drift and turn failures into new cases in the evaluation set.
Topics covered
- production quality
- online evaluations
- data drift
- user feedback
In the glossary
Where it fits
Measure quality, cost and latency
How do you know, every week, whether the feature answers well, what it costs and whether it is fast enough?
Lessons in this module
- Track quality in production (this lesson)
- Manage cost per request and per outcome
- Control perceived latency
What you will learn in this module
This lesson is part of the course Ship and monitor an AI feature in production
- Set up quality signals in production (user feedback, sampled human review, online evaluations, drift) and connect them to the evaluation set.
- Manage cost per request and per useful outcome, and perceived latency (budgets, alerts, caching, streaming, response length, model choice).
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Evaluate an AI feature: test sets, metrics and LLM judges
3 hr