Prompt caching is a model provider feature that reuses the processing of a repeated prompt prefix across requests. In AI engineering, it can make agents and long-context applications faster and, on many providers, cheaper, because they often resend the same instructions, tool definitions, and documents on every call.
For example, suppose a document-question agent sends a 6,000-token policy manual, a 500-token instruction block, and a 100-token user question with every request. If the manual and instructions come first and never change, later requests can reuse that 6,500-token prefix. Only the 100-token question needs full processing, so about 98% of each request’s input is eligible for reuse.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why prompt caching matters
Large language model (LLM) applications repeat a lot of input. Agents resend their system instructions and tool definitions on every step, and retrieval apps often resend the same reference material.
Prompt caching is useful for:
- Reducing input cost: On many providers, input tokens read from cache are billed differently from tokens processed in full.
- Reducing latency: The provider skips work it has already done for the shared prefix.
- Designing prompt layouts: The prefix rule gives you a clear reason to put stable content first and variable content last.
- Forecasting spend: Measured cache reuse tells you how much of a projected saving is real for your traffic.
Savings depend on how much of your traffic repeats the same prefix, not on whether the feature is switched on. Arize’s prompt caching benchmark and prompt caching analysis show why realized savings depend on prompt structure and traffic patterns.
How prompt caching works
LLM inference happens in two phases. The model first processes the input prompt, then generates output tokens one at a time.
When many requests start with the same material, that first phase repeats identical work. Common repeated material includes a long system instruction, tool definitions, a reference document, or few-shot examples.
Prompt caching lets the provider keep the result of processing that shared beginning. When the same beginning appears again, the provider reuses it instead of recomputing it. This happens within the provider’s eligibility and retention rules.
The unit that matters is the prefix. A prefix is the beginning of the prompt, in order, up to the first point where two requests differ.
Consider two requests that share everything except the user’s question at the end. The shared portion is a candidate for reuse.
Now suppose a timestamp or user name is placed at the very start. The two requests now differ in their first tokens. There may be little or no common prefix to reuse, even if most of the later text is identical.
The model still generates a fresh answer on every request. Prompt caching only reuses work on the input side.
For current eligibility, lifetime, and pricing rules, the provider’s own documentation is the authority. Check it before projecting savings for your workload.
How to structure prompts for caching
Put stable material first and variable material last. A common layout looks like this:
Cache-friendly ordering (illustrative)
[ stable system instruction ][ stable tool definitions ][ stable examples ]
[ session context that changes slowly ][ current user request <- varies ]
^______________ shared prefix likely reused ______________^
This order makes the shared prefix between consecutive requests as long as possible.
Reuse is commonly lost in a few ways:
- Early per-request values: A date, request ID, or user-specific text near the start breaks the prefix for everything after it.
- Unstable ordering: Tools or examples that appear in a different order between requests produce a different prefix.
- Formatting drift: Changes to whitespace or serialization in the stable section change the token sequence.
- Model changes: A cached prefix belongs to the model that processed it.
None of these is a reason to remove information the model needs. If the model genuinely needs a timestamp, provide it. Place it as late as the task allows.
Keep the serialization of the stable section deterministic. Generate it the same way, byte for byte, on every request where the provider requires an exact match.
How to measure whether prompt caching is working
Start with the cached-token share. This is the fraction of input tokens the provider reports as read from cache, taken from its usage fields for the model you call.
Look at that share by route or use case, not only in aggregate. One endpoint with a long stable prefix may benefit while another with short, variable prompts does not.
The following Python code groups hypothetical usage records by route. It reports the cached-token share and the fraction of requests that reported any cached tokens. Field names vary by provider, so map them from your own usage data.
from collections import defaultdict
records = [
{"route": "doc_qa", "input_tokens": 6600, "cached_tokens": 0},
{"route": "doc_qa", "input_tokens": 6600, "cached_tokens": 6500},
{"route": "doc_qa", "input_tokens": 6600, "cached_tokens": 6500},
{"route": "doc_qa", "input_tokens": 6600, "cached_tokens": 6500},
{"route": "chat", "input_tokens": 900, "cached_tokens": 0},
{"route": "chat", "input_tokens": 1100, "cached_tokens": 0},
]
totals = defaultdict(lambda: {"requests": 0, "hits": 0, "input": 0, "cached": 0})
for r in records:
t = totals[r["route"]]
t["requests"] += 1
t["hits"] += r["cached_tokens"] > 0
t["input"] += r["input_tokens"]
t["cached"] += r["cached_tokens"]
for route, t in totals.items():
share = t["cached"] / t["input"]
hit_rate = t["hits"] / t["requests"]
print(f"{route}: cached-token share {share:.1%}, "
f"requests with cache hits {hit_rate:.0%}")
Output:
doc_qa: cached-token share 73.9%, requests with cache hits 75%
chat: cached-token share 0.0%, requests with cache hits 0%
The document route reuses most of its input after the first request. The chat route reuses nothing, so caching does not help it as currently structured.
Then check the outcomes you care about: cost per request, cost per completed task, and latency. A high cached share on cheap, short requests may matter less than a moderate share on your longest, most frequent calls.
Tokenization matters here because providers count and match tokens, not characters. A change that looks trivial in text can still alter the token sequence.
Use traces and usage records to confirm which requests reported cached tokens. This is part of AI observability for the application. For more on the request-level metric, see cache hit rate.
Worked example: a document-question agent
Return to the agent from the opening. Each request contains a 6,000-token policy manual, a 500-token instruction block, and a 100-token user question. These sizes are illustrative.
| Request | Prefix reused | Tokens processed in full |
|---|---|---|
| First request | None | 6,600 |
| Later request, same manual | 6,500 | 100 |
| Request after the manual is edited | None | 6,600 |
| Request after the cached prefix expires | None | 6,600 |
Without caching, every request pays full input processing for about 6,600 tokens. With caching, later requests pay full processing only for the new question and any changed tail.
The saving per request is bounded by the shared prefix, not the whole prompt. It disappears for the first request, after the cached prefix expires, and whenever the manual is edited.
This means you cannot multiply a per-request saving by total traffic. First measure what fraction of real requests actually hit the cache.
Limitations of prompt caching
Prompt caching has clear boundaries:
- Output tokens: It does not reduce output-token cost.
- Unique prompts: It does not help prompts that are mostly unique or variable from the first tokens.
- Finite lifetime: Cache lifetime is provider-controlled, so low-frequency endpoints may find entries expired between calls.
- Provider terms: Some providers impose minimum prefix lengths or charge differently for cache writes and cache reads.
- Usage reporting: Provider usage fields can be hard to reconcile with your own logs.
- Data isolation: Do not assume a shared prefix isolates sensitive per-user content. Follow the provider’s stated isolation model and keep user-specific material in the variable tail.
Confirm the current terms for your provider and model rather than assuming they carry over from another one.
Prompt caching vs. response caching and the KV cache
Prompt caching is often confused with three related ideas.
Response caching stores a finished answer and returns it for a later identical request without calling the model at all. Semantic caching does the same for similar requests. Prompt caching still calls the model and generates a fresh answer.
That difference sets expectations. Response caching can skip model calls entirely, but it risks returning stale answers. Prompt caching cannot return a stale answer because generation still happens. It also cannot help if your prompts share no stable prefix.
The key-value (KV) cache is the internal mechanism that stores attention keys and values during generation. It avoids recomputing tokens within a request and, in some serving setups, across requests. Prompt caching is the externally visible provider policy built on related ideas. For more on the internal mechanism, see KV cache.
An LLM gateway may route, log, or apply policies across providers. A gateway does not by itself create provider-side prefix reuse. The model provider serving the request decides eligibility.
Frequently asked questions
Does prompt caching change the model’s answers?
It should not, because generation still runs on the full prompt. Prompt caching only reuses computation for the shared input prefix. If answers change after you restructure a prompt for caching, suspect the moved or edited content rather than the caching mechanism.
When does prompt caching save the most?
It saves the most when many requests share a long, stable prefix and differ only near the end. Typical examples are long instructions, tool definitions, reference documents, or examples followed by a short variable request. It saves little when prompts are short, mostly unique, or variable from the first tokens.
Why is my cached-token share low?
The usual cause is variable content placed early in the prompt. Other causes include unstable ordering or formatting of the stable section, and traffic too infrequent for entries to survive between calls. Prompts below a provider’s minimum cacheable length also miss. Compare the opening tokens of requests that did and did not report cached tokens.
Is prompt caching the same as caching answers?
No. Answer caching returns a stored response without a model call. Prompt caching always generates a new response and only avoids recomputing the shared input prefix. The two have different freshness and cost profiles, and you can use them independently.
Putting prompt caching to work
Order each prompt so stable content comes first, and make the stable section’s serialization deterministic. Then record cached tokens and total input tokens for every request.
Report the cached-token share by route, next to cost per completed task and latency. If a route shows near-zero reuse, inspect the first tokens of its requests for early variability before assuming the feature is unavailable.
Tracing tools such as Arize AX can capture token usage on each LLM call, so you can see which requests reused a prefix and which did not. For more on tracking input and output tokens, see token usage.
