Skip to content
Module
Module 4 of 6Lesson 1 of 2~18 min

Choose a grading method per criterion

Code, human review or a model judge: how to choose the way each criterion of an AI feature gets graded. The choice shapes the cost, speed and reliability of your evals, and it is a concrete conversation to have with engineers before they build the grading pipeline.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to choose, for each criterion, between code-based grading, human grading and an LLM judge, and to justify the choice by the nature of the criterion, cost, speed and reliability.

Questions this lesson answers

  • How do you choose between code-based, human and LLM judge grading?
  • When should a human grade an AI answer instead of a model?

Topics covered

  • grading methods
  • automated grading
  • human evaluation
  • LLM as a judge
  • grading rubric

In the glossary

Full glossary

Where it fits

Grade the answers

How do you grade each criterion reliably, and when can you trust an LLM judge?

Lessons in this module

  1. Choose a grading method per criterion (this lesson)
  2. Design and calibrate an LLM judge

What you will learn in this module

This lesson is part of the course Evaluate an AI feature: test sets, metrics and LLM judges

  • Choose, for each criterion, a grading method (code, human, LLM judge) and justify the trade-off between cost, speed and reliability.
  • Design an LLM judge (rubric, format, different model), identify its biases and calibrate it against human grades.

Related courses