Skip to content
Module
Module 6 of 6Lesson 2 of 2~18 min

Assemble your feature's evaluation plan

The course's final workshop: you bring your choices together into a complete evaluation plan for your own AI feature. That document becomes the shared reference with engineering and the business for making a release decision, along with a rollout schedule for the weeks that follow.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to assemble your AI feature's evaluation plan, with its versioned test set and release thresholds, and plan its rollout at 7 and 30 days.

Questions this lesson answers

  • What should a complete AI feature evaluation plan include?
  • How do you get an AI evaluation plan approved by the business team?

Topics covered

  • evaluation plan
  • LLM evals
  • versioned test set
  • release thresholds
  • AI evaluation workshop

Where it fits

Decide on release

When is a version ready, and how do you keep a change from degrading it?

Lessons in this module

  1. Set release thresholds and the regression rule
  2. Assemble your feature's evaluation plan (this lesson)

What you will learn in this module

This lesson is part of the course Evaluate an AI feature: test sets, metrics and LLM judges

  • Turn an AI feature's goal into specific, measurable, achievable and relevant success criteria, taking error severity into account.
  • Build a representative test set from real traffic, edge cases and adversarial cases, and justify its composition.
  • Label the test set with a guide, measure inter-annotator agreement, version it and protect it from overfitting.
  • Choose, for each criterion, a grading method (code, human, LLM judge) and justify the trade-off between cost, speed and reliability.
  • Design an LLM judge (rubric, format, different model), identify its biases and calibrate it against human grades.
  • Choose and interpret the right metrics for a classification and for a RAG system, and infer which stage to fix.
  • Define the regression rule, how offline and online evaluations fit together, and release thresholds set before seeing results.

Related courses