What is RAG? Retrieval-augmented generation for PMs
Retrieval-augmented generation (RAG) is an architecture where an application first searches a document collection for the excerpts that best match a question, then sends those passages to a large language model along with the question. The model writes its answer from that supplied context rather than from memory alone. The term comes from a 2020 research paper by Lewis and colleagues at Facebook AI Research.
Definition written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.
Why it matters for a PM
Most AI assistants built on company knowledge (help centers, contracts, internal wikis, product catalogs) are RAG systems. As a PM you decide what the assistant may read, how fresh the content must be, who is allowed to see which documents, and what the product shows when nothing relevant is found. Those are product decisions, and they often weigh more on answer quality than the choice of model.
Example
A support assistant for an accounting app answers “Can I export invoices to my bank format?” by first retrieving the three most relevant help articles, then generating a short answer that links to them. If no article covers the bank in question, it says so and offers to open a ticket instead of guessing.
How does RAG work, step by step?
Upstream, the team prepares the index: documents are collected, cleaned and split into passages, then each passage is converted into a vector and stored with its metadata (source, date, access rights). This preparation runs again whenever a document changes, which calls for an update pipeline rather than a one-off import.
For each question, the system may rewrite the query, look up the closest passages, filter them by the user's permissions, optionally rerank them with a sharper model, and then insert the best ones into the prompt with an instruction to answer from them and cite them. The model writes the reply last. Each step can fail on its own, which is why it pays to inspect them one at a time.
RAG or fine-tuning: which should you choose?
RAG supplies knowledge at question time; fine-tuning changes the model itself through additional training. When the need is about facts (documentation, contracts, a catalog), RAG is almost always the right starting point: it stays current, it can cite sources and it respects access rights. Fine-tuning makes more sense for enforcing a format, a style or a very specific behavior, or for getting a small model to do what a larger one already handles.
A third option deserves a test first: if the corpus is small and stable, simply placing it in the context window may be enough, with no index and no pipeline to maintain. Context windows are now large enough for this to work with many internal handbooks or modest product catalogs, although cost per request rises with every page sent.
How do you evaluate a RAG system?
Evaluate the two halves separately. For retrieval: on a set of questions where you know the right document, does it appear among the top passages returned? For generation: does the answer rely only on the passages provided, does it actually address the question, and does each citation point to the right place?
Add questions whose answer is not in the corpus, to check that the system knows when to abstain, and questions asked by users with different permissions, to test filtering. When an answer is wrong, look at the retrieved passages first: if the information was not there, switching models will not fix it. Freshness deserves its own check too: edit a document, then measure how long it takes before answers reflect the change.
Key points
- RAG has two halves: retrieval (finding the right passages) and generation (writing an answer from them). Most quality problems start in retrieval.
- It suits knowledge that changes often or is private: updating an index is faster and cheaper than retraining a model.
- Documents are split into chunks and indexed, usually with embeddings for semantic search, often combined with keyword search.
- Good RAG products cite their sources and say plainly when the documents do not contain the answer.
- Access rights are applied during retrieval, so a user never gets an answer built from a document they cannot open.
Common mistakes
- Treating RAG as a cure for hallucinations: the model can still misread or ignore the passages it receives.
- Indexing everything without owners or freshness rules, so outdated pages keep surfacing in answers.
- Judging quality from a handful of demos instead of a test set of real questions with expected answers.
- Choosing fine-tuning to add knowledge, when fine-tuning mostly changes style and behavior.
Go further with Module
The courses and lessons that cover this concept:
Guide on this topic
- RAG vs fine-tuning: how to choose
What each approach solves, the decision criteria and how to evaluate, with a decision grid.