Module 4 of 6Lesson 2 of 2~24 min
Design and calibrate an LLM judge
How to specify a model judge, spot its biases and check that it grades like your experts before trusting it with thousands of answers. A poorly tuned judge produces flattering scores that can get a faulty version approved; knowing how to audit it lets you ask the right questions of the team that built it.
Lesson objective
By the end of this lesson, you will be able to specify an LLM judge (rubric, grading format, choice of model), identify its biases and calibrate it against human grades before trusting it with evaluation at scale.
Questions this lesson answers
- How do you design a rubric for a reliable LLM judge?
- What biases affect an LLM judge, and how do you counter them?
- How do you calibrate an LLM judge against human grades?
Topics covered
- LLM as a judge
- model judge
- judge bias
- judge calibration
- evaluation rubric
In the glossary
Where it fits
Grade the answers
How do you grade each criterion reliably, and when can you trust an LLM judge?
Lessons in this module
- Choose a grading method per criterion
- Design and calibrate an LLM judge (this lesson)
What you will learn in this module
This lesson is part of the course Evaluate an AI feature: test sets, metrics and LLM judges
- Choose, for each criterion, a grading method (code, human, LLM judge) and justify the trade-off between cost, speed and reliability.
- Design an LLM judge (rubric, format, different model), identify its biases and calibrate it against human grades.
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductExpertAdvanced level: for deeper mastery and specialization.
Ship and monitor an AI feature in production
3 hr