RAG or fine-tuning: picking the right approach for an AI feature
Use RAG for facts and documents that change, fine-tuning for a style, a format or a narrow task. Criteria, evaluation and a free decision grid.
Guide written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.
In short
Pick RAG when the model has to answer from facts or documents that change, and when you want to cite the source. Pick fine-tuning when you want to change behavior: tone, output format, a narrow task done over and over. In most products you start with a solid prompt, add RAG if knowledge is missing, and only consider fine-tuning after that, with measurements to back it.
RAG vs fine-tuning: which one should you choose?
RAG hands the model information at answer time; fine-tuning changes how the model behaves. A knowledge problem (the model does not know your data) calls for RAG. A behavior problem (the model does not answer the way you need) calls for fine-tuning.
The mix-up comes from a common belief: "for the AI to know our documents, we have to train it on them." In practice that is rarely the right path. A model tuned on your documentation does not memorize it reliably, cannot tell you where an answer came from, and goes stale as soon as the document changes.
- Facts, documents, data that evolve (catalog, help center, contracts, internal procedures): RAG.
- Tone, answer structure, strict format, high-volume classification or extraction: fine-tuning, when prompting alone falls short.
- Both at once: an assistant that cites your documents in a house format can combine RAG with a tuned model, but that usually comes late in a product’s life.
So the question for your team is not "which technique?" but "what is the model missing today to answer well: a piece of information, or a way of doing things?"
How does RAG work?
RAG (Retrieval-Augmented Generation) looks up the relevant passages in your documents when a question arrives, puts them in the prompt, and asks the model to answer from those passages, ideally citing them.
The term comes from a research paper by Lewis et al. (2020). In today’s products, the pipeline boils down to two phases.
- Prepare the corpus: documents are split into pieces (chunking), each piece is turned into a numeric vector (its embeddings) and stored in a vector database or a search index.
- Answer: for each question, the system retrieves the closest pieces, often mixing meaning-based and keyword search, then sends them to the model with the instruction to stick to them.
The underlying model stays untouched. Quality depends entirely on what you feed it: if retrieval surfaces the wrong passage, the answer will be confidently wrong. That is why you judge a RAG system on retrieval first, then on the answer.
Updating a document means reindexing it: no retraining, and the new version is live on the next question.
How does fine-tuning work?
Fine-tuning continues training an existing model on selected examples (an input and its expected output) so it adopts a specific behavior without being told again on every request. It produces a new model version that has to be served (by the provider or by you), evaluated and maintained, and that goes away with its base model.
The idea: you collect example/ideal-answer pairs (OpenAI suggests starting with about fifty well-crafted examples and scaling up if evaluation justifies it), the provider or your team adjusts the model’s parameters, and you get a dedicated model. Options vary by provider (supervised tuning, preference tuning, which models are eligible), so check your provider’s documentation before promising anything. The offering moves fast: OpenAI, for instance, has closed its fine-tuning platform to new users.
- What it learns well: a consistent style, a strict output format, a narrow repeated task (routing tickets, extracting fields), sometimes on a smaller and therefore faster model.
- What it learns poorly: precise, verifiable facts. It keeps an impression, not a copy, which can make hallucinations on details worse, as a 2024 study measured.
- What it costs: a clean example set, training runs, an evaluation for every version, and fresh work whenever the base model moves on.
OpenAI’s guide to optimizing LLM accuracy draws the same line: RAG addresses what the model needs to know to answer, fine-tuning addresses how it should behave.
What criteria decide between RAG and fine-tuning?
Six criteria separate RAG from fine-tuning: how fresh the information must be, whether you need to cite sources, total cost, acceptable latency, the data you actually have, and privacy. Run through them for each need, not for the whole product at once.
- Freshness: if the information changes every week, RAG keeps up by reindexing; a tuned model stays frozen at its training date.
- Traceability and citations: RAG can show the passage it used and link to the document. A tuned model cannot say where an answer came from. In legal, medical or customer support settings, this often settles the question.
- Cost: RAG costs mostly at run time (indexing, retrieval, longer prompts). Fine-tuning costs mostly up front (data, training, evaluation), then may shorten prompts. Compare on your real request volume.
- Latency: RAG adds a retrieval step and a longer prompt. A small tuned model can answer faster on a narrow task.
- Available data: RAG needs up-to-date, well-organized documents; fine-tuning needs many consistent examples of good answers. Plenty of teams have the former, few have the latter.
- Privacy and permissions: with RAG, each user must only retrieve documents they are allowed to see, which you enforce at search time. With fine-tuning, whatever goes into the model can come back out for anyone (researchers have extracted training data from a language model), so never train on data a given user should not be able to read.
This review often ends in a mixed answer: RAG for the help center, data passed in directly by the app for the user’s own account, and a plain prompt for tone.
Should you start with prompting before RAG or fine-tuning?
Yes, start with the prompt: clear instructions, examples, context supplied directly. It is the fastest option to test and to change. RAG comes in when the useful context grows too large or changes too fast; fine-tuning when the prompt can no longer hold a stable behavior.
Many needs people assume require fine-tuning are solved with two or three examples in the prompt and a well-described output format. Many needs people assume require RAG are solved by passing the model data the app already holds: the customer record, the current order.
When the corpus is small and stable, it may even fit entirely in the prompt. Anthropic says so in its post on contextual retrieval: below roughly 200,000 tokens of knowledge base (about 500 pages), putting all the content in the prompt, combined with prompt caching, can be enough. That shortcut stops working once the corpus grows, changes often, or depends on each user’s permissions.
- Prompt only: instructions, examples, expected format. Measure.
- Supplied context: add the data the app already has, or a small complete corpus. Measure.
- RAG: when the useful content outgrows what reasonably fits in the prompt or changes too fast. Measure.
- Fine-tuning: when a behavior gap remains that neither prompt nor context fixes, and you have the examples to train on.
To understand what a model can do on its own before adding architecture around it, the course on LLM capabilities and limits helps you map the limits of the bare model.
Can you combine RAG and fine-tuning?
Yes. RAG and fine-tuning combine well because they act on two different things: RAG supplies current facts, fine-tuning shapes how they are used. The combination is worth it once each building block has proven its value on its own.
Three combinations show up in real products:
- A model tuned to use retrieved passages well: it learns to stay within the supplied documents, to cite, and to admit it has no answer when retrieval brings back nothing useful.
- A model tuned for format: answers must follow a house structure (record, standardized summary, JSON fields) that the prompt alone does not hold reliably.
- A small tuned model inside the pipeline: a fast model rewrites the query or ranks passages before the main model answers.
Combining also doubles maintenance: an index to keep current and a model to retrain. Only adopt it if evaluation shows a clear gain over RAG alone. For an assistant built into your product, the course on designing an in-product assistant shows how these pieces fit together on the user-experience side.
How do you check that you made the right choice?
Check the choice with a set of real questions and their expected answers, run against each candidate: prompt only, RAG, tuned model. Compare correctness, source citations, latency and cost per request, then keep the simplest option that clears your bar.
- Collect real questions from support tickets, internal search logs and users. Include questions the documents cannot answer.
- Write the expected answer and, for RAG, the document that should be retrieved.
- Measure retrieval separately: is the right passage among those returned? If not, no model tweak will save the answer.
- Measure the answer: correct, faithful to sources, complete, in the right format. Flag invented answers.
- Compare cost and latency on a realistic volume, not on three tries.
Set the acceptance bar before running the tests; otherwise the favorite option always ends up "good enough." The general method is laid out in the guide AI evals for product managers, and the course on evaluating AI features puts it into practice.
What mistakes should you avoid with RAG and fine-tuning?
The most common mistakes are fine-tuning to teach facts, launching RAG without checking document quality, putting sensitive data into a tuned model, and choosing without an evaluation set. Each one costs weeks and shows up late.
- Fine-tuning to "learn the docs": the model blends things, invents details and ages with the docs. For facts, look them up at answer time.
- Indexing messy documents: conflicting versions, stale pages, unreadable PDFs. RAG will faithfully retrieve the wrong text. Curating sources is product work.
- Forgetting access rights: a RAG shared by all users can expose a confidential document to the right question. Filter at search time, per user.
- Judging from a demo: five well-chosen questions always pass. Only a set of real questions reveals the gaps.
- Skipping the prompt step: committing a team to fine-tuning before writing a careful prompt with examples.
- Believing a huge context window makes RAG obsolete: it fixes neither permissions, nor freshness, nor the per-request cost on a large corpus.
To scope a RAG system end to end (sources, chunking, retrieval, permissions, freshness, cost), the RAG architecture course walks you through it step by step, up to an architecture document you can share with engineering.
Free template: RAG-VS-FINE-TUNING.md
Fill in one grid per need in the feature (not one for the whole product), then share it with engineering before any development starts.
Download RAG-VS-FINE-TUNING.md# RAG or fine-tuning: [feature name]
Need under review: [e.g. answer questions about the return policy]
Date: [date]
## 1. What the model is missing today
- [ ] Information (facts, documents, data) → RAG or supplied context
- [ ] A way of doing things (tone, format, repeated task) → prompt, then fine-tuning
- Example of a bad answer today: ...
## 2. Criteria grid
| Criterion | Answer | Points to |
|---|---|---|
| Does the information change often? | yes / no | yes → RAG |
| Must answers cite the source? | yes / no | yes → RAG |
| Do answers depend on the user's permissions? | yes / no | yes → filtered RAG or app data |
| Does the whole corpus fit in the prompt? | yes / no | yes → supplied context |
| Does format or tone escape the prompt? | yes / no | yes → fine-tuning |
| Do we have at least several dozen well-crafted good-answer examples? | yes / no | no → no fine-tuning |
| Is latency critical? | yes / no | yes → small model, short prompt |
| Would sensitive data end up in training? | yes / no | yes → no fine-tuning |
## 3. Options tested
| Option | Correctness | Sources cited | Latency | Cost per request |
|---|---|---|---|---|
| Prompt only | | | | |
| Prompt + supplied context | | | | |
| RAG | | | | |
| Tuned model (if justified) | | | | |
Acceptance bar set before testing: ...
## 4. Decision
- Chosen option: ...
- Why: ...
- What would change our mind: ...
## 5. Checks
- [ ] RAG sources curated and current
- [ ] Access rights enforced at retrieval
- [ ] Evaluation set of real questions ready
- [ ] Owner named for updates (index or model)
Put it into practice
The “Design a RAG architecture that fits your product” course walks you through it step by step, on your own product, all the way to the deliverable.
See the courseFrequently asked questions
Does fine-tuning let an LLM know my documents?
Poorly. A tuned model keeps a general impression of the content, not a reliable copy, and cannot cite its source. For an assistant to answer from your documents, retrieve them at question time with RAG.
Is RAG cheaper than fine-tuning?
Not necessarily. RAG is cheap to start but lengthens every request; fine-tuning costs in preparation and upkeep but can shorten prompts. Compare on your real volume and include the team’s time.
When is fine-tuning actually worth it?
When a narrow task repeats at scale, the prompt cannot get a stable behavior, and you have plenty of good-answer examples. Without all three, the gain rarely covers the effort.
Does a model with a very large context window replace RAG?
For a small, stable corpus everyone can see, often yes. For a large, changing corpus or one with per-user permissions, no: you still have to choose what to send, and every request pays for the context it carries, even if prompt caching lowers the price.
Does a product manager need to know how to build a RAG system?
No, but they need to know how to scope one: which sources, for which users, with which permission and freshness rules, and how to judge quality. Those are product decisions engineering cannot make alone.