Module 6 of 7Lesson 1 of 2~16 min
Specify a run trace and its metrics
When an agent goes wrong in production, you need to understand what it saw, decided and executed, step by step. This lesson teaches you to specify what a run trace must keep and to choose the indicators that tell you whether the agent delivers on cost, duration and success.
Lesson objective
By the end of this lesson, you will be able to specify what a run trace must record to reconstruct what an agent did, and choose the run metrics to track (success, steps, cost, duration, stops, rejected approvals).
Topics covered
- agent observability
- LLM tracing
- agent metrics
- AI monitoring
- AI agents
Where it fits
Observe and evaluate an agent
How do I reconstruct what the agent did, and how do I know it is reliable?
Lessons in this module
- Specify a run trace and its metrics (this lesson)
- Evaluate an agent on tasks, trials and trajectories
What you will learn in this module
This lesson is part of the course Design reliable agents and tool calling
- Specify the trace of a run and the evaluation of an agent (task success, trajectory, cost, latency, consistency over several trials).
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Evaluate an AI feature: test sets, metrics and LLM judges
3 hr