Skip to content
Module
AI

What is an LLM hallucination? Causes and safeguards

An LLM hallucination is an output that sounds plausible but is false, unsupported or invented: a citation that does not exist, a wrong figure, a feature the company never shipped. It happens because a language model generates a likely continuation of a text, not a verified fact. Fluent, confident wording says nothing about accuracy.

Definition written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.

Why it matters for a PM

In a product, a hallucination is a defect with a cost: a wrong refund policy given to a customer, a made-up clause in a contract summary, an invented parameter in generated code. PMs cannot remove the risk entirely, so they decide where it is acceptable, how it gets detected, and what the interface does to keep users from acting on an unchecked answer.

Example

A travel app's assistant tells a customer that their fare includes free cancellation, because similar fares often do. The booking terms say otherwise. The fix is not a better tone of voice: the assistant should read the actual fare rules through a tool and quote them, or decline to answer.

Why do LLMs hallucinate?

A language model learns to produce a plausible continuation, and its training often rewards a confident answer over an admission of ignorance. When information is missing or fuzzy in its training data, it fills the gap with something that has every outward sign of a correct answer: a credible title for a research paper, a legal reference in the right format, a library function that could plausibly exist.

Two families are worth separating. A factual hallucination contradicts the real world. A faithfulness hallucination contradicts the source the model was given: a summary adds a figure that is not in the document, or an answer attributes to an article something it never says. The second kind is often the more damaging in a product, because users believe the answer has been checked.

Models keep improving, but the problem does not vanish: it shifts toward niche questions, long sources and chained tasks, where a small early error spreads through everything that follows. An agent that relies on an invented fact at step two can build ten coherent actions on a false premise.

How can a product reduce hallucinations?

The levers add up. Provide the source and ask the model to quote the exact passage behind each claim. Narrow the task: extracting or classifying invites less invention than open-ended writing. Hand precise facts (prices, stock levels, dates) to tools that query the system of record. Design an honest way out, in the prompt and in the interface, for when the information is missing.

On the interface side, show sources next to the answer, label generated content, and add a human approval step before any action with real consequences. An automated check can also compare each claim against the passages provided and block those that are not supported.

How do you measure a hallucination rate?

Build a set of questions whose correct answers you know, including a share of questions that cannot be answered from the available information. For each output, check whether the claims are accurate and whether they are supported by the source. A second model can handle part of that checking at scale, provided its judgments have been validated on a sample reviewed by people.

Track the result over time and by question type: a low overall rate can hide an entire category where the model invents an answer almost every time. Rerun the measurement whenever the model, the prompt or the retrieval setup changes, since any of the three can shift the rate.

Key points

  • Hallucinations are more frequent on rare facts, precise numbers, recent events and topics poorly covered by the training data.
  • Giving the model the source material (RAG, attached documents, tool results) reduces them but does not eliminate them.
  • Letting the model answer “I don't know”, or leave a field empty, is a design choice that has to be specified and tested.
  • Rates come from a test set rather than from impressions: the same prompt can be right on Monday and wrong on Tuesday.
  • Sycophancy, the habit of siding with whatever the user suggests, is a related failure that makes leading questions risky.

Common mistakes

  • Assuming a newer or larger model no longer hallucinates.
  • Asking the model to check its own answer and treating the reply as proof.
  • Hiding sources in the interface, which leaves users no way to verify.
  • Trusting a confidence level the model states about itself as if it were a calibrated probability.

Go further with Module

The courses and lessons that cover this concept:

Guides on this topic