What is chunking? Splitting documents for RAG
Chunking is the step in a RAG system that splits each document into pieces, called chunks, before indexing them. Chunks are the units that search retrieves and that the model reads in order to answer. The split can follow a fixed size, the document's structure (headings, paragraphs, clauses) or the meaning of the text.
Definition written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.
Why it matters for a PM
How documents are split determines what the assistant can find. Chunks that are too small lose their context, like a sentence about a deadline without the name of the plan it applies to. Chunks that are too large bury the useful part and cost more to send to the model. When an assistant misses an answer that exists in the documents, chunking is one of the first suspects, well before switching models.
Example
An HR assistant indexes a company agreement cut every 500 characters. Asked "how many days off for a wedding?", it retrieves a chunk stating a number of days but not the event it covers. The team re-splits the agreement by clause and adds each clause title to its chunk, and the right answer comes back.
Key points
- A good chunk stands on its own: it keeps its heading, section or source document name.
- Following the natural structure of documents often beats a fixed size.
- A small overlap between chunks avoids cutting an idea in half.
- The right setting is found by testing retrieval on real questions, not in theory.
Common mistakes
- Using the same splitting rule for FAQs, contracts and tables.
- Losing useful metadata (date, author, access rights) during the split.
- Changing the split without reindexing and comparing quality before and after.
Go further with Module
The courses and lessons that cover this concept:
Guide on this topic
- RAG vs fine-tuning: how to choose
What each approach solves, the decision criteria and how to evaluate, with a decision grid.