Cache hit rate is the share of requests or input tokens that a prompt cache serves from a stored prefix instead of processing again. In LLM applications, it tells you whether prompt caching is actually engaging on your traffic, which affects both cost and response time.
For example, suppose you send 200 requests in one hour, and 160 of them get at least one cache hit. That is an 80% hit rate by request. But only 1,800,000 of the 4,000,000 input tokens were cache reads, so the token-based hit rate is just 45%.
Both numbers are correct. They answer different questions, so a hit rate means little until you state its denominator.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why cache hit rate matters
Prompt caching can lower cost and shorten the wait before a response starts. Cache hit rate is the first signal of whether caching is working on your real traffic.
Cache hit rate is useful for:
- Checking that caching engages: Confirm that requests are reusing stored prefixes at all.
- Explaining cost: See how much of your input is being read from cache rather than processed fresh.
- Debugging prompt structure: Find variable content that breaks the shared prefix.
- Watching deploys: Spot when a prompt or tool change resets the cache.
Read hit rate alongside end-to-end latency and cost per completed task before you change anything. A hit rate on its own does not prove that a workload got cheaper.
How cache hit rate works
Prompt caching reuses work the model already did on a shared beginning of a prompt, called a prefix. If a new request starts with exactly the same prefix as an earlier one, the provider can read that part from cache.
A cache hit is a request where some of the input was served this way. A cache miss is a request where none of it was.
Request-based vs. token-based hit rate
You can count hits in two ways.
A request-based hit rate counts requests. Of all requests, how many got at least one cache hit?
A token-based hit rate, also called a token-weighted hit rate, counts input tokens. Of all input tokens sent, how many were read from cache rather than processed fresh?
The two numbers diverge because requests differ in size. Many short requests with small cached prefixes can produce a high request-based rate. Meanwhile, the long and expensive requests might miss every time.
The token-based rate weights each request by its size, which tracks what it costs. That makes it the better default for cost questions. Arize’s prompt caching benchmark reports caching behavior in token terms for this reason.
Hit rate depends on the serving setup
Cache eligibility and invalidation rules are specific to each provider. A hit rate is a property of one workload on one serving setup, not a portable score.
Whichever denominator you use, record it with the provider, model, and time window.
How to calculate cache hit rate
Calculate both rates over the same set of requests in a fixed time window.
Formula:
request-based hit rate = requests with >= 1 cached token / total requests
token-based hit rate = cached input tokens read / total input tokens sent
Where:
- Requests with >= 1 cached token is the number of requests where the provider reported any cache read.
- Total requests is every request sent in the window.
- Cached input tokens read is the sum of input tokens the provider reported as cache reads.
- Total input tokens sent is the sum of all input tokens in the window, cached or not.
The following Python calculates both rates for the opening example. The numbers are illustrative.
total_requests = 200
requests_with_hit = 160
input_tokens = 4_000_000
cached_tokens = 1_800_000
request_rate = requests_with_hit / total_requests
token_rate = cached_tokens / input_tokens
print(f"request-based hit rate: {request_rate:.0%}")
print(f"token-based hit rate: {token_rate:.0%}")
print(f"input tokens processed fresh: {input_tokens - cached_tokens:,}")
Output:
request-based hit rate: 80%
token-based hit rate: 45%
input tokens processed fresh: 2,200,000
Caching engages on most requests, but more than half of the input tokens are still processed fresh. If the 40 requests that missed are your longest and most expensive ones, the token-based figure is the one warning you.
Pair hit rate with time to first token
Cached prefixes skip part of the prefill step in LLM inference. That is the work the model does on the input before it generates any output.
Pair hit rate with time to first token to see whether hits are also making responses start sooner.
How to debug a low cache hit rate
Work through the usual causes in order.
- Prefix instability: Caching reuses a shared beginning. If timestamps, user IDs, or shuffled tool lists sit at the front of your prompt, every request has a different prefix and nothing matches.
- Short shared prefixes: If the common part of your requests is tiny, hits save little even when they occur. You will see this as a gap between the request-based and token-based rates.
- Traffic shape: With low request volume or long gaps between similar requests, cache entries can expire before anything reuses them.
- Prompt churn: Every edit to a system prompt or tool definition invalidates the prefix built on it. Frequent deploys can keep the cache permanently cold.
In traces, compare the cached-token count on each request with its input-token count. Then group the results by prompt version and request type.
The pattern usually points to one unstable segment of the prompt rather than a global problem. The Phoenix documentation covers capturing token usage on spans so you can run this comparison.
Limitations of cache hit rate
- Hit rate is not savings: Cached tokens may be discounted rather than free, and cache writes can cost extra. A workload engineered for hits may also send more total tokens than a leaner prompt would.
- Averages hide the tail: One aggregate number conceals the expensive misses. Report hit rate by request type.
- Provider reporting differs: What counts as a cached token, and how it appears in usage fields, varies by provider. Confirm the definitions before you compare providers.
- Optimizing for hits can distort prompts: Freezing a bloated prefix to protect the cache trades higher token cost for a better-looking hit rate.
The question that drives decisions is whether the whole task got cheaper or faster at equal quality. Hit rate helps explain that answer, but it is not the answer itself.
Cache hit rate vs. semantic caching and latency metrics
Prompt caching and semantic caching are easy to confuse.
Semantic caching reuses a whole stored answer for a new query that is similar to an old one. Prompt caching reuses computation on an identical prefix, and the model still generates a fresh answer.
Hit rate also differs from speed metrics. A hit tends to reduce time to first token. But inter-token latency, the time between tokens after generation starts, is unaffected.
Routing traffic through an LLM gateway can add its own overhead either way. Measure latency end to end rather than inferring it from hit rate.
Frequently asked questions
Which hit rate should I report, request-based or token-based?
Report the token-based rate as your primary figure, because cost scales with tokens. Include the request-based rate if you also care how often caching engages at all. Always name the denominator, provider, model, and time window.
Why is my cache hit rate low even though my prompts look similar?
Prompt caching matches an exact shared prefix, so similar prompts are not enough. A variable field near the start, reordered tools, or a newly deployed prompt version breaks the match. Inspect which segment of the prefix changes between requests.
Does a high cache hit rate guarantee lower cost?
No. Cache reads may still be billed at a reduced rate, writes may cost more, and total tokens may have grown. Confirm with cost per completed task at equal quality before you conclude that caching is paying off.
Does a cache hit make the whole response faster?
No, it mainly shortens the wait before generation starts. The cached prefix skips reprocessing, but token generation speed after that point is unchanged. Measure both time to first token and overall latency.
Putting cache hit rate to work
When you report cache hit rate, record the denominator, provider, model, time window, and request type. Report the token-based rate first, and break it down by prompt version so you can see the effect of each deploy.
Keep variable content out of the start of your prompts, and check cost per completed task to confirm caching is paying off. Tracing tools such as Arize Phoenix can show cached and total input tokens per request, which makes an unstable prefix easy to find.
