How to evaluate an AI feature: LLM evals explained for PMs
AI evals show whether an LLM feature is ready to ship: test cases, rubric, LLM judge, thresholds and production checks. A PM method with a free template.
Guide written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.
In short
An AI eval is a repeatable test that measures whether an LLM-powered feature answers well, on a chosen set of cases and against criteria written in advance. For a product manager, evals replace “it looks good” with a decision you can defend: ship, fix or drop. They rest on three things: real and difficult cases, a grading rubric, and thresholds set before anyone looks at the results.
What is an AI eval?
An AI eval is a set of test cases, each paired with a success criterion, that you run against an LLM-powered feature to measure its quality. It produces a score you can compare across versions, instead of an impression formed from a handful of tries.
AI evals do for an AI feature what tests do for regular code: they tell you whether the product does what it should, and whether a change broke it. The difference lies in what gets checked. A regular test compares an output with an exact value. An eval judges a natural language answer that can be right in a thousand different wordings.
In practice, an eval brings together three things:
- A case set: representative inputs (user questions, documents, tickets), with the expected answer when there is one, or the points it must contain.
- Criteria: what makes an answer acceptable. Accuracy, format, tone, no made-up facts, refusing when it should.
- A grading method: an automated check, a human review, or a model that grades against a rubric.
Anthropic’s documentation on building evaluations describes this approach: define specific, measurable success criteria, then build evaluations that check them, automating the grading where you can. The PM does not write the eval code. The PM owns its content: which cases matter, what a good answer looks like, and what result is good enough to ship.
Why is testing an LLM different from testing code?
Testing an LLM differs from testing code because an LLM can give different answers to the same input, several answers can be correct, and its mistakes sound convincing. A few manual tries therefore say very little about how the feature behaves on real user traffic.
- Variability. Send the same prompt twice and you may get two different answers. A case that passed once is not a case you control.
- Many right answers. A correct summary can be written in countless ways. Word-for-word comparison with a reference answer fails; you have to judge the substance.
- Believable mistakes. An LLM hallucination looks like a good answer: fluent, confident, wrong. It easily survives a quick read.
- Side effects everywhere. Changing one sentence of the prompt, the model or the source documents can improve one kind of case and degrade another, with nothing visibly broken.
This is why a successful demo is a classic trap. It shows the feature can answer well, not that it answers well often enough. Evals measure the second thing, on enough cases for the result to mean something.
How do you build a test set to evaluate an AI feature?
A good test set for an AI feature mixes three families: real cases drawn from actual usage, hard cases where the model is likely to fail, and edge or adversarial cases. Each case states what a good answer must include or must avoid.
- Start from real usage. Support questions, searches, documents users upload: they reflect your traffic best. Before launch, ask the support or sales team to reconstruct typical requests.
- Add hard cases. Ambiguous questions, information missing from the sources, off-topic requests, clumsy phrasing. This is where two versions start to differ.
- Add edge and adversarial cases. Empty or very long input, another language, attempts to steer the assistant away from its instructions. These attacks get a full treatment in the AI product security course.
- Write down what you expect. Next to every case, note a model answer or, more often, the list of what it must contain and what it must never say.
- Freeze a version. Version the test set like any product document, so you know which version backed which decision.
Anthropic recommends favoring volume: many automatically graded cases often tell you more than a small batch scored by hand. Start small, then add a case every time you find a failure. The lesson on building a test set shows how to balance these families for your own feature.
Which criteria and rubric should you use to grade answers?
Eval criteria turn the product goal into checkable requirements: accuracy, faithfulness to sources, correct format, suitable tone, appropriate refusals. A rubric spells out, for each criterion, what separates an acceptable answer from a failed one, with an example on each side.
A useful criterion can be checked without debate. “Relevant answer” lets everyone judge in their own way; “the answer only uses information found in the provided documentation” can be settled with a yes or a no.
- One criterion, one question. Grade accuracy and tone separately, so a polite but wrong answer cannot earn an average score that hides the error.
- Pass or fail rather than a score out of 10. A binary verdict is easier to apply consistently, by a person or by a model.
- Weight by severity. An awkward phrase and a harmful piece of advice do not weigh the same. Separate blocking criteria from nice-to-have ones.
- Examples inside the rubric. One accepted and one rejected answer per criterion beat an abstract definition.
These criteria work much like acceptance criteria: the PM owns them, because they encode what users expect. If your feature must return a precise format, the guide on writing a spec for an AI agent shows how to state that kind of requirement without ambiguity.
Human, automated or LLM-as-a-judge evaluation: which one?
Pick the grading method criterion by criterion: code-based checks when the rule is exact, an LLM judge when you need to assess meaning across many cases, and human review to calibrate the judge and settle sensitive cases. All three can live in the same eval.
- Code-based checks. Is the format valid, is the label right, does a banned word appear? Fast, free, fully repeatable, but limited to exact rules.
- Human review. The most flexible and highest-quality way to judge an open-ended answer, but slow and costly. Use it mainly to build references and to check the other methods.
- LLM judge. A second model grades the answers against your rubric. It reads meaning like a person, at machine speed, but it has biases of its own.
An LLM judge, often called “LLM as a judge”, needs a few safeguards that Anthropic’s documentation sums up: a detailed rubric, a constrained verdict (correct or incorrect, or a 1 to 5 scale), reasoning requested before the verdict, and preferably another model than the one whose answers are being graded. Research by Zheng et al. (2023) also describes known biases, such as favoring longer answers or, most often, the first answer shown in a side-by-side comparison.
Above all, do not take a judge at its word. Check its verdicts against people’s verdicts on a sample first, and only hand it large-scale grading once agreement is good enough. The lesson on designing and calibrating an LLM judge walks through that calibration.
What thresholds should you set before shipping?
Set one threshold per criterion before running the eval, never after seeing the scores. Criteria tied to serious harm block the release, usually with almost no tolerance, and every new version must also do at least as well as the version already in production.
A threshold answers one question: what result are we willing to ship with? Agree on it with the people who carry the risk, such as support, legal and the business owner. The PM makes the call and writes it down.
- Absolute thresholds. For example: no answer that invents a commercial policy, and a large majority of answers graded accurate. The exact values depend on your context; what matters is writing them first.
- Comparison with what is live. A candidate version is also measured against the production version. An average gain that hides a regression on an important category is not a gain.
- A written decision rule. What happens if a nice-to-have threshold is narrowly missed? If the answer is “we lower the threshold”, the eval no longer protects anything.
OpenAI’s evaluation best practices recommend continuous evaluation: rerun your evals on every change, including a model switch, since a newer model is not automatically better for your use case. See the lesson on release thresholds and regression rules.
How do you monitor quality in production and catch regressions?
In production, you track an AI feature’s quality by sampling real answers, grading them with the same rubric, and watching usage signals. Any change to the prompt, model or data reruns the full test set before it is deployed.
Pre-launch evals tell you whether a version can ship. They cannot see the questions nobody thought of. Production can.
- Sample and grade. Each week, a share of real conversations goes through the rubric, graded by the judge and spot-checked by people.
- Watch weak signals. Thumbs down, rephrased questions, drop-offs, escalations to support: each one points to cases worth a look.
- Feed the test set. Every real failure becomes a new test case. The set grows with the product, and the same mistake cannot come back unnoticed.
- Rerun on every change. Prompt, model, indexed documents, settings: every change goes through the full eval, like a regression test suite.
The course on running AI features in production covers the logging, gradual rollout and incident handling that complete this monitoring.
How do evals relate to the prompt?
Evals and the prompt form a loop: the prompt is a hypothesis about how to get good answers, and the eval tests it. Without evals, every prompt tweak fixes the case in front of you and quietly breaks others that nobody is looking at.
Prompt engineering without measurement is guesswork. With a test set, every prompt version gets a score, and you keep the one that improves without regressing elsewhere. Anthropic presents defining criteria and tests as a step that comes before prompt work.
- Write the criteria and a first test set before optimizing the prompt.
- Measure the current version: that is your baseline.
- Change one thing at a time, then rerun the eval.
- Read the failures, not just the score: they tell you what to fix.
Evals also show when the prompt is the wrong lever. If answers fail because the information is missing, the problem usually sits in retrieval. The guide RAG vs fine-tuning helps you choose the right lever, and the prompting course for PMs covers the basics of writing the prompt itself.
What mistakes should you avoid when evaluating an AI feature?
The most common mistakes are deciding on a demo, testing only easy cases, grading with vague criteria, trusting an uncalibrated LLM judge, setting thresholds after the fact, and no longer evaluating once the feature has launched.
- Deciding on a few tries. Ten well-chosen questions in a meeting do not represent real traffic.
- A test set that is too clean. If every case is easy, every version looks great and the eval separates nothing.
- One overall score. An average hides severe failures. Keep one result per criterion.
- An uncalibrated judge. An LLM judge never compared with people may be measuring something other than what you think.
- Optimizing for the test set. Tweak the prompt long enough to pass the same cases and you end up passing the eval without improving the product. Keep some cases aside and refresh them.
- Stopping at launch. Usage shifts and models change: an eval that is no longer run no longer protects you.
To apply this method to your own feature and leave with its evaluation plan, test set and thresholds, the AI feature evaluation course guides you step by step, without writing code.
Free template: PLAN-EVALS.md
Fill in this plan for an AI feature before you change its prompt or model, then share it with the team that will build the eval. Keep it versioned next to the feature.
Download PLAN-EVALS.md# Evaluation plan: [feature name]
Owner: [name]
Test set version: v1
Last updated: [date]
## Feature goal
[Which users, which task, what outcome. 2 to 3 sentences.]
## Criteria
| Criterion | Definition (pass if...) | Severity | Grading method |
|---|---|---|---|
| Accuracy | ... | blocking | LLM judge + human check |
| Faithfulness to sources | ... | blocking | LLM judge |
| Format | ... | nice-to-have | code |
| Tone | ... | nice-to-have | LLM judge |
## Test set
- Real cases (source: ...): [count]
- Hard cases (ambiguous, missing info, off-topic): [count]
- Edge and adversarial cases: [count]
- Held-out cases (never used to tune the prompt): [count]
For each case: input, expected (must include / must never say), category.
## LLM judge
- Judge model (different from the evaluated model): ...
- Rubric: [link]
- Calibration: [count] cases graded by people, observed agreement: ...
## Release thresholds (set before results)
- Blocking criteria: ...
- Nice-to-have criteria: ...
- Against the production version: no regression on ...
## Decision rule
- All thresholds met: ship.
- A blocking criterion fails: do not ship.
- A nice-to-have criterion fails: [who decides, based on what]
## Production monitoring
- Sample graded every [week]: [size]
- Signals tracked: negative feedback, rephrasing, escalations
- Every real failure becomes a test case
## History
- [date]: [version tested], [result], [decision]
Put it into practice
The “Evaluate an AI feature: test sets, metrics and LLM judges” course walks you through it step by step, on your own product, all the way to the deliverable.
See the courseFrequently asked questions
Do you need to code to set up AI evals?
Not to design them, yes to automate them. The PM defines the criteria, cases, rubric and thresholds; a developer or an evaluation tool runs them. A spreadsheet is enough for a first manual eval.
How many cases should a test set have?
There is no universal number. Start with enough cases to cover every usage category and every type of risk, then add cases each time you observe a failure. The more critical a category, the more cases it deserves.
What does “LLM as a judge” mean?
“LLM as a judge”, or LLM judge, means using a language model to grade another system’s answers against a rubric. It helps evaluate large volumes of open-ended answers, as long as the judge is calibrated against human grades.
Do evals replace user testing?
No. Evals measure answer quality on known cases; user testing and usage signals tell you whether the feature meets the need. They complement each other, and user feedback often supplies new eval cases.
Are evals enough to make an AI feature safe?
No. They measure failures, they do not prevent them. You also need guardrails in production: filters, limits on actions, handoff to a person. For an assistant built into your product, see the course on designing an in-product AI assistant.