Context window: what an LLM can read at once
The context window is the maximum amount of text, counted in tokens, that a language model can take into account in a single request: instructions, conversation history, documents and its own answer included. Tokens are chunks of text; in English one token averages roughly three quarters of a word. Whatever falls outside the window is invisible to the model for that request.
Definition written by Alex Dionisio, Senior Product Manager. Updated November 13, 2026.
Why it matters for a PM
The context window shapes many product choices: how much of a conversation an assistant remembers, whether a long contract can be analyzed in one pass, what each request costs and how long it takes. Large windows do not remove the need to select, because quality tends to drop when the relevant information is buried in a very long input.
Example
A legal assistant can accept a 300-page contract in one request, yet it misses a clause buried on page 180. The team changes the design: the contract is split by section, the assistant retrieves the sections relevant to each question and cites the page of every clause it mentions.
What fills the context window on each call?
The window fills up in layers: the system prompt, the descriptions of available tools, the conversation history, any documents or excerpts added, tool results, and finally the answer being written. The limit applies to the total. In an agent that chains tool calls, those results are often what takes up the most room. Tool descriptions are resent with every call as well, so an agent connected to dozens of tools can use a large share of its window before the user has typed a word.
Size is counted in tokens, not words or pages, and the ratio between the two depends on the language and the tokenizer: the same text can use more tokens in French than in English, for example. An estimate made on an English document does not necessarily hold for its translations.
Large context window or RAG: which should you choose?
Sending a whole document has real strengths: nothing is lost to chunking, and the model sees how the parts relate. It is often the right call for a single document of reasonable size, such as an interview transcript or a requirements document. RAG takes over when the corpus is larger than the window, changes often, mixes different access rights, or when cost per request matters.
The two approaches combine well: retrieval selects a few relevant documents, which are then passed in full rather than as small snippets. That combination often gets the best of both: a fast first sort, then a complete reading of the documents that survived it.
How do you handle a conversation that keeps growing?
Each technique comes with a trade-off. Truncating the history and keeping only recent turns is simple, but the start of the conversation is forgotten. Summarizing older turns keeps the gist at the cost of lost detail. Extracting durable facts (preferences, decisions) into a separate memory, fed back in when relevant, preserves what matters across sessions. Whatever the choice, tell users what the assistant remembers and what it does not, so they are not surprised when an early detail is gone.
Providers also offer prompt caching: when the system prompt and fixed documents stay unchanged between calls, their processing is reused, which cuts both cost and response time. Ask your team whether the feature takes advantage of it.
What questions should you ask about the context window?
How much of the window is used on average, and in the worst cases? What happens when the limit is reached: an error, a silent cut, an automatic summary? Has the behavior been tested on the longest real conversations? A limit crossed without a clear signal produces incoherent answers that are hard to diagnose after the fact. Ask too who watches these numbers after launch: prompts tend to grow with each release as instructions, tools and examples pile up.
Key points
- A model keeps no memory between requests: the application resends whatever it should remember.
- Cost and latency grow with the number of tokens sent, so long histories get summarized or trimmed.
- Details placed midway through a very long input get overlooked more often than those near its start or end.
- Retrieval (RAG) picks the relevant passages instead of sending everything.
- Coding agents manage their window actively: clearing it, compacting it or delegating to subagents.
Common mistakes
- Equating a million-token window with a million tokens of reliable understanding.
- Letting a long chat grow until early instructions are crowded out.
- Forgetting that the answer itself consumes part of the window and of the output budget.
Go further with Module
The courses and lessons that cover this concept:
Guides on this topic
- Claude Code for Product Managers
What a PM can do with Claude Code, the method and the guardrails, with a starter CLAUDE.md.
- RAG vs fine-tuning: how to choose
What each approach solves, the decision criteria and how to evaluate, with a decision grid.