The KV cache is memory where a large language model (LLM) stores the attention keys and values it has already computed for earlier tokens. Reusing those stored values keeps text generation fast, but the cache grows with context length and limits how many requests a server can handle at once.
For example, suppose a model stores the equivalent of 32,768 numbers per token for keys, and the same again for values, at 2 bytes per number. Each token then occupies about 131 kilobytes of cache. A single 32,000-token request needs roughly 4.2 gigabytes, and that memory is unavailable to any other request while it runs.
KV stands for key-value. The numbers in this example are illustrative, not the specification of any particular model.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why the KV cache matters for AI engineering
You rarely touch the KV cache directly through an API. Its effects still show up in your latency, your costs, and how much traffic your serving setup can handle.
Understanding the KV cache helps with:
- Explaining latency: See why the first token takes longer than the tokens that stream after it.
- Planning capacity: Estimate how context length and concurrency compete for the same memory.
- Pricing long context: Recognize that long prompts cost serving memory, not just tokens.
- Reading usage fields: Interpret “cached tokens” in a provider response without overreading it.
- Debugging slow agents: Tell whether a slowdown comes from resending history or from the serving side.
How the KV cache works
LLMs generate text one token at a time. To choose each next token, the attention mechanism relates the current position to every earlier position.
That comparison needs a key vector and a value vector for every earlier token, in every attention layer. Without storage, the model would recompute those vectors for the entire sequence each time it produced a token.
The work per token would keep growing with the sequence. Total cost would rise much faster than output length, and long generations would become impractically slow.
The KV cache removes that repeated work. Here is what happens step by step:
- The model computes keys and values for a token the first time it processes that token.
- It stores them in the cache.
- When the next token is generated, the model computes only that token’s own key and value and appends them.
- Attention for the new token reads the stored entries for all prior positions.
This is tied to how autoregressive models produce sequences. Each step conditions on everything before it, and the cache makes “everything before it” cheap to reuse.
For a step-by-step walkthrough of the mechanism, see Hugging Face’s KV cache documentation.
Prefill and decode
Because of the cache, a request has two phases that behave differently.
Prefill: The model processes the initial prompt. It computes and stores keys and values for every prompt token. This phase is compute-heavy.
Decode: The model generates output tokens one by one, reusing the cache. This phase is typically limited more by memory movement than by arithmetic.
This split explains why time to first token and streaming speed respond to different optimizations. A long prompt mostly affects prefill. Heavy serving load mostly affects decode.
How to estimate KV cache memory
Cache size grows with three things: how much the model stores per token, the precision of each stored number, and the number of tokens in the sequence.
Formula:
cache bytes per request = 2 × values_per_token × bytes_per_value × sequence_length
Where:
- 2 accounts for storing both keys and values.
- values_per_token is how many numbers the model stores for keys for one token, summed across all layers.
- bytes_per_value is the storage size of each number, such as 2 bytes at 16-bit precision.
- sequence_length is the number of tokens in the request, including the prompt and generated output.
Multiply by the number of concurrent requests to get the total memory the cache needs on a server.
Calculate KV cache memory in Python
The following script uses the illustrative numbers from the opening example. It also assumes, hypothetically, that 64 gigabytes of accelerator memory are left over for the cache, and it computes how many requests of each length fit at once.
values_per_token = 32_768 # keys only, summed across layers (illustrative)
bytes_per_value = 2 # 16-bit precision
cache_budget_gb = 64 # memory left for KV cache (hypothetical)
def cache_bytes(seq_len):
return 2 * values_per_token * bytes_per_value * seq_len
per_token = cache_bytes(1)
print(f"per token: {per_token / 1e3:.0f} KB")
for seq_len in (8_000, 16_000, 32_000):
per_request = cache_bytes(seq_len)
concurrent = int(cache_budget_gb * 1e9 // per_request)
print(f"{seq_len:>6} tokens: {per_request / 1e9:.2f} GB per request, "
f"{concurrent} concurrent requests")
Output:
per token: 131 KB
8000 tokens: 1.05 GB per request, 61 concurrent requests
16000 tokens: 2.10 GB per request, 30 concurrent requests
32000 tokens: 4.19 GB per request, 15 concurrent requests
Real figures vary widely with architecture, precision, and optimizations such as grouped-query attention or cache quantization. Treat this as an order-of-magnitude illustration, not a sizing guide for a specific model.
Worked example: context length versus concurrency
Using the script’s hypothetical 64-gigabyte budget, here is how context length trades against concurrency.
| Context length | Cache per request | Requests that fit at once |
|---|---|---|
| 8,000 tokens | 1.05 GB | 61 |
| 16,000 tokens | 2.10 GB | 30 |
| 32,000 tokens | 4.19 GB | 15 |
Each doubling of context length roughly doubles the cache per request. On the same hardware, that roughly halves the number of requests served at the same time.
This is the practical sense in which long context has a serving cost beyond tokenization counts or prompt price. It consumes the resource that determines concurrency.
KV cache vs. prompt caching
The KV cache and prompt caching are related, but they operate at different levels.
The KV cache is the internal store described above. The inference engine manages it within a single request, and it is normally invisible through a model API. Every generation uses one.
Prompt caching is a provider feature with stated rules, such as eligible prefix lengths, retention, and pricing. It lets the work done on one request’s prefix be reused by a later request. Not every API request benefits from it.
When a provider reports “cached tokens,” it is reporting reuse under its prompt-caching policy. That is related to the raw cache that made decoding fast, but it is not the same thing. A provider may keep, evict, share, or discard cached prefixes according to rules you do not control.
Through an LLM gateway or a hosted API, you observe the policy and the usage fields, not the engine’s memory. For measured behavior of the provider layer, see Arize’s prompt caching analysis.
How to reason about the KV cache as a developer
What you can do depends on whether you call a hosted API or run your own inference engine.
Hosted APIs: You cannot read the KV cache, and you do not need to. Reason about its effects instead. Watch input length, cached-token reporting where offered, time to first token, and streaming rate in your traces. Those signals separate prefill problems from decode problems.
Self-hosted inference: The levers become concrete. Your inference engine exposes maximum sequence length, cache precision, and batching limits. Each one trades against throughput.
AI observability practices make this separation visible instead of guessed. Record token counts, latency, and provider usage fields for every call.
Debugging an agent that slows down
Suppose an agent’s calls get slower as a conversation grows. Two different causes can produce the same symptom.
Resent history: Each call sends a longer conversation history, so prefill grows. Prompt caching may offset part of this where it is available. The fix is to trim or summarize the history.
Serving pressure: Streaming itself gets slower, which points to decode or serving capacity. The fix is to investigate load on the serving side.
The Phoenix documentation shows how this kind of per-call detail is captured for local and open-source stacks.
Limitations of the KV cache
- It stores computation, not understanding: The cache cannot fix a bad prompt. Evicting or truncating it carelessly mid-sequence changes what the model’s generation is conditioned on.
- Memory pressure forces trade-offs: Teams choose between shorter maximum contexts, fewer concurrent requests, or cache compression that gives up a little quality for capacity.
- Isolation is a provider question: Check your provider’s documentation for exactly what isolation exists between requests and tenants when prefix computation is reused. Do not assume it.
- A fast cache can hide waste: An agent that resends a bloated, mostly irrelevant history on every step may stay fast enough to ship. It still pays prefill and memory costs that a leaner context would avoid.
Frequently asked questions
Does the KV cache store the model’s answers?
No. It stores intermediate attention keys and values for tokens already processed, so later tokens can attend to them. It is not a database of finished responses or past conversations. Serving systems may still reuse cached prefixes across requests under a provider’s prompt-caching policy.
Why does long context slow serving down?
Each token adds key and value entries across layers, so cache memory grows with sequence length. More memory per request means fewer concurrent requests and more data moved during decoding. The slowdown is a memory and capacity effect as much as a compute effect.
Can I clear or control the KV cache through an API?
Generally not directly. Hosted APIs manage it internally. You influence it indirectly through prompt structure, context length, and any prompt-caching controls the provider exposes. Self-hosted inference engines expose explicit cache and batching settings.
How is the KV cache related to prompt caching?
Prompt caching builds on the same reuse idea, but across requests instead of within one. The KV cache always operates inside a single generation. Cross-request reuse depends on the provider’s eligibility, retention, and pricing rules.
How much memory does the KV cache use?
It depends on the model’s architecture, its precision, and the sequence length. In the illustrative example above, one token takes about 131 kilobytes and a 32,000-token request takes about 4.2 gigabytes. Real models vary widely.
Putting the KV cache to work
When you choose a context length, treat it as a capacity decision as well as a quality decision. Estimate the cache per request and multiply it by the concurrency you expect.
For every model call, record input tokens, output tokens, cached tokens where reported, time to first token, and streaming rate. Those fields let you tell prefill cost from decode cost.
In agent workflows, trace each step so you can see when history growth starts to drive latency. Arize Phoenix can capture those per-call details and show where a long context is costing you time.
