Final workshop: your dashboard and your procedure
The final workshop applies the course to your own AI feature: a focused dashboard for quality, cost and latency, with its alerts and owners, and your procedure for changing models. You leave with the exit kit and an application plan to put both into service in the weeks ahead.
Lesson objective
By the end of this workshop, you will have produced, for your own AI feature, a quality, cost, latency dashboard (eight metrics at most, three quantified alerts, one owner per metric) and a model change procedure, with the kit and the action plan to put them into service.
Topics covered
- AI dashboard
- workshop
- exit kit
- model change procedure
Where it fits
Tooling and reporting
Which tools do you use to track all this, and how do you report on it to get decisions?
Lessons in this module
- Choose your observability tools
- Report to stakeholders
- Final workshop: your dashboard and your procedure (this lesson)
What you will learn in this module
This lesson is part of the course Ship and monitor an AI feature in production
- Design the staged launch of an AI feature (feature flag, shadow, canary, percentages) with quantified promotion criteria and a tested kill switch.
- Define what you log and trace for each model call, what you mask or exclude (personal data), how long you keep it, and with which tool.
- Set up quality signals in production (user feedback, sampled human review, online evaluations, drift) and connect them to the evaluation set.
- Manage cost per request and per useful outcome, and perceived latency (budgets, alerts, caching, streaming, response length, model choice).
- Diagnose AI-specific incidents (drift, provider outage, deprecation, call loop, cost spike) and plan the right fallbacks.
- Run a model or prompt change with regression evaluations, shadow comparison, staged rollout and a rollback plan.
- Build the quality, cost and latency dashboard of an AI feature and report to stakeholders with decisions to make.
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Evaluate an AI feature: test sets, metrics and LLM judges
3 hr