Module 6 of 7Lesson 2 of 2~19 min
Evaluate an agent on tasks, trials and trajectories
An agent never answers the same way twice and can reach the right result through a dangerous path, which makes the usual tests insufficient. This lesson teaches you to design an agent evaluation from real cases, with criteria and thresholds set before measuring.
Lesson objective
By the end of this lesson, you will be able to design an agent evaluation: tasks drawn from real cases, graders on the final state, trajectory invariants, multiple trials, pass@k and pass^k metrics, and thresholds set before measuring.
Topics covered
- agent evaluation
- evals
- pass@k
- AI agent testing
- quality thresholds
In the glossary
Where it fits
Observe and evaluate an agent
How do I reconstruct what the agent did, and how do I know it is reliable?
Lessons in this module
- Specify a run trace and its metrics
- Evaluate an agent on tasks, trials and trajectories (this lesson)
What you will learn in this module
This lesson is part of the course Design reliable agents and tool calling
- Specify the trace of a run and the evaluation of an agent (task success, trajectory, cost, latency, consistency over several trials).
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Evaluate an AI feature: test sets, metrics and LLM judges
3 hr