Control perceived latency
A correct answer that arrives too slowly can still make users give up. This lesson teaches you to choose the latency measures that truly reflect the experience you are aiming for, beyond the average, then to propose the right levers, from progressive display to background processing.
Lesson objective
By the end of this lesson, you will be able to choose the right latency measures (time to first token, total time, percentiles) for the intended experience, and to propose the right levers (streaming, length, model, steps, deferred processing).
Topics covered
- LLM latency
- time to first token
- percentiles
- streaming
- user experience
Where it fits
Measure quality, cost and latency
How do you know, every week, whether the feature answers well, what it costs and whether it is fast enough?
Lessons in this module
- Track quality in production
- Manage cost per request and per outcome
- Control perceived latency (this lesson)
What you will learn in this module
This lesson is part of the course Ship and monitor an AI feature in production
- Set up quality signals in production (user feedback, sampled human review, online evaluations, drift) and connect them to the evaluation set.
- Manage cost per request and per useful outcome, and perceived latency (budgets, alerts, caching, streaming, response length, model choice).
Related courses
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Build an AI assistant for your product
3 hr
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Design a RAG architecture that fits your product
3 hr 30 min
- AI & ProductAdvancedRegular practice: you already use the tools or work on this topic.
Evaluate an AI feature: test sets, metrics and LLM judges
3 hr