Context is the information available to a model at the moment it generates a response, makes a decision, or chooses the next action. Concretely, and this is the meaning that matters operationally, it is the input your system assembled and sent on one specific call: instructions, conversation history, retrieved documents, tool definitions, tool results so far, and whatever else your code decided the model needed.
The word is used for three different things, which is the main reason conversations about it go wrong. Naming which one you mean is half the work. The other half is accepting that the model does not assemble its own context. Your application does, on every call, and what it includes or leaves out usually determines the output more than the choice of model does.
That is why so many failures filed as model failures are context failures. A model cannot reason over information it never received. A retrieval step that returned nothing useful, a summarizer that dropped the order ID, a history truncated at the wrong place: each produces a confident wrong answer from a model working correctly on the input it was given.
Key takeaways
- Context means three things: the window as a token budget, the assembled input sent on one call, and the retrieved passages in a RAG system. The assembled input is the primary meaning.
- Context is a view, not a store. It is built fresh every call from state, memory, retrieval, and instructions, and it disappears when the call returns.
- Retrieved context is a subset of assembled context, so retrieval scores say nothing about the instructions, history, or tool results around it.
- Most reported model failures are context failures: something needed was missing, stale, truncated, or buried among passages that were merely similar.
- Debugging requires recording the exact assembled input per call. Rebuilding it later from your assembly code is guesswork, because that code’s inputs have moved on.
Three things people mean by context
The context window. The maximum number of tokens a model can attend to in one call, counting input and output. A property of the model, and a budget rather than content. “We ran out of context” means this one.
The assembled context. The actual bytes sent on a given call: every message, every tool schema, every retrieved chunk, in the order they appear. A property of your application, different on every call, and the thing you inspect when debugging. “The agent lost the constraint” means this one.
Retrieved context. The passages a retriever returned for the current query, placed into the assembled context to ground the answer. A property of your retrieval system. “Context relevance dropped” means this one, and it is scored per chunk against the question, which says nothing about the rest of the input.
The three fail separately and have different owners. A window problem is fixed by compaction or a different model, an assembly problem in the code that builds the prompt, a retrieval problem in indexing, chunking, query rewriting, or ranking.
What is in an assembled context
For a production agent, one call typically contains:
- System instructions. Role, task framing, constraints, output format, and policy text.
- Conversation history. Prior turns, often partially summarized rather than verbatim.
- Retrieved documents. Passages from a search over your corpus, with any citation metadata.
- Tool definitions. The schema for every tool the model may call on this step, which costs real tokens and grows with the tool surface.
- Tool results. What previous calls in this run returned, in full or compacted.
- Task and progress information. The plan, which steps are done, counters and budgets, drawn from the run’s state.
- Application data. The current user, their entitlements, the record being discussed.
Every one of those is a decision your code made. Something chose the top_k, decided how many turns to keep verbatim, ordered the sections, and picked what to cut when the budget got tight. That assembly logic lives in the harness rather than in the model, and treating it as a component with its own tests is the core argument for agent harness evaluation and tracing.
Context, state, and memory
Three adjacent terms, one clean boundary between them.
State is the execution record for the current run. Context is the per-call view assembled from that state plus instructions and retrieval. Memory survives after the run and can be retrieved into a later context.
How context fails
Missing. The needed fact was never retrieved, never written to state, or written and never surfaced back. The model answers anyway, which is how a context gap becomes a hallucination. Most of the hallucination patterns worth cataloguing start as an input problem rather than a model defect.
Stale. Correct when cached, wrong by the time it was used. Long runs make this routine, and timestamps on cached values are the cheap defense.
Diluted. Ten retrieved chunks, one of which answers the question. Near-misses are not free: they consume budget and give the model plausible material to answer from.
Truncated. The window filled and something got dropped. What gets dropped is rarely chosen by importance, and summarizers do not know which identifier mattered.
Conflicting. A retrieved document contradicts the system prompt, or two documents contradict each other. The model resolves it silently and without telling you it had to choose.
None of these are visible from the final output alone. They are visible when the assembled input for each call is recorded alongside the output, which is why an agent observability setup stores prompts as span attributes rather than only latency and status.
FAQ
Is context the same as the context window?
No, and mixing them up leads to the wrong fix. The window is the token limit the model imposes. Context is the material you put inside that limit. A model with a two million token window can still be given the wrong context, and a larger window does nothing for an agent that was never handed the document it needed. Bigger windows let you defer compaction, at higher cost and latency per call, and filling one with everything you have tends to lower accuracy rather than raise it.
What is the difference between context and the prompt?
The prompt is usually understood as the instructions you wrote, the static part. Context is everything present on the call, including that prompt plus the dynamic material: history, retrieved passages, tool schemas, and tool output. In practice most of what determines the answer is the dynamic part, which is also the part not visible in your repository.
How do I debug a suspected context failure?
Get the exact input from the failing call. Read the assembled context and ask, in order: was the needed information present at all, was it accurate at that moment, was it findable among everything else, and did anything else in the input contradict it. If the answer was derivable from the input, it is a generation problem. If it was not, no amount of prompt tuning will fix it.
