LLM hallucination
An LLM hallucination is an output that sounds plausible but is false, unsupported or invented: a citation that does not exist, a wrong figure, a feature the company never shipped. It happens because a language model generates a likely continuation of a text, not a verified fact. Fluent, confident wording says nothing about accuracy.
Why it matters for a PM
In a product, a hallucination is a defect with a cost: a wrong refund policy given to a customer, a made-up clause in a contract summary, an invented parameter in generated code. PMs cannot remove the risk entirely, so they decide where it is acceptable, how it gets detected, and what the interface does to keep users from acting on an unchecked answer.
Example
A travel app's assistant tells a customer that their fare includes free cancellation, because similar fares often do. The booking terms say otherwise. The fix is not a better tone of voice: the assistant should read the actual fare rules through a tool and quote them, or decline to answer.
Key points
- Hallucinations are more frequent on rare facts, precise numbers, recent events and topics poorly covered by the training data.
- Giving the model the source material (RAG, attached documents, tool results) reduces them but does not eliminate them.
- Letting the model answer “I don't know”, or leave a field empty, is a design choice that has to be specified and tested.
- Rates come from a test set rather than from impressions: the same prompt can be right on Monday and wrong on Tuesday.
- Sycophancy, the habit of siding with whatever the user suggests, is a related failure that makes leading questions risky.
Common mistakes
- Assuming a newer or larger model no longer hallucinates.
- Asking the model to check its own answer and treating the reply as proof.
- Hiding sources in the interface, which leaves users no way to verify.
- Trusting a confidence level the model states about itself as if it were a calibrated probability.
Go further with Module
The courses and lessons that cover this concept: