Semantic caching stores a model’s answer and returns it for a later query with a similar meaning. It cuts generation cost and latency, but it risks serving an answer that does not fit the new question, the current facts, or this user.
For example, suppose a documentation assistant has cached its answer to “How do I reset my API key?” A later query, “Where do I regenerate an API key?”, is a likely safe match. But “How do I reset my API key for the legacy console?” uses similar words and needs a different answer, so returning the cached one would be wrong.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why semantic caching matters
Every model call costs money and time. When many users ask the same questions, generating a fresh answer each time repeats work.
A semantic cache skips that work by reusing a stored answer. The catch is that “similar” is not the same as “identical,” so every hit carries some risk of being wrong.
Understanding semantic caching helps with:
- Cutting cost and latency: Skip the model call entirely for repeated questions.
- Setting the similarity threshold: Decide how close a new query must be before you reuse an answer.
- Protecting user data: Design the cache so one user’s answer is never served to another.
- Evaluating hit quality: Measure whether cached answers are correct, not just how often the cache hits.
A wrong cache hit is worse than a miss. A miss costs one model call, but a wrong hit gives the user an incorrect answer with confidence.
How semantic caching works
A semantic cache sits in front of the model and handles each query in four steps.
- Embed the query: The system converts the query into a vector that represents its meaning.
- Search for a match: It compares that vector with stored query vectors and looks for one above a chosen similarity threshold.
- Return on a hit: If a match is found, the stored answer is returned and the model is never called.
- Store on a miss: If no match is found, the model answers normally, and the new query and answer may be stored for later.
Steps 1 and 2 use the same matching idea as semantic search. For more on how query vectors are created and compared, see embeddings and vector search.
Most of the design decisions live in steps 2 and 4. You choose the embedding model, the threshold, what is allowed into the cache, and when entries expire.
Where the cache usually lives
A semantic cache often sits behind an LLM gateway rather than inside the application. Arize’s comparison of LLM gateways covers response caching as one gateway feature alongside routing and observability.
Worked example: three queries against one cached answer
Consider the documentation assistant from the opening. It has stored one exchange:
stored query: "How do I reset my API key?"
stored answer: steps for the current console
new query: "Where do I regenerate an API key?" -> likely a safe hit
new query: "How do I reset my API key for the legacy console?"
-> similar wording, different correct answer: a dangerous hit
new query (a week later, after a console redesign):
"How do I reset my API key?" -> exact wording, stale answer
All three queries could clear a loose similarity threshold. Only the first is clearly safe.
The second query is a different question that happens to use similar words. The third is the same question, but the facts behind the answer changed.
How the cache key changes the outcome
The following script simulates this cache. Suppose your embedding model gives the three new queries similarity scores of 0.91, 0.93, and 1.00 against the stored query, and your threshold is 0.90.
The script runs the lookups twice. The first time, the cache matches on similarity alone. The second time, the console version is part of the cache key.
cache = {
("current", "en"): ("How do I reset my API key?", "Steps for the current console"),
}
# Hypothetical similarity scores to the stored query "How do I reset my API key?"
requests = [
("Where do I regenerate an API key?", 0.91, ("current", "en")),
("How do I reset my API key for the legacy console?", 0.93, ("legacy", "en")),
("How do I reset my API key?", 1.00, ("current", "en")),
]
def lookup(similarity, key, threshold, use_key):
entry = cache.get(key) if use_key else next(iter(cache.values()))
if entry and similarity >= threshold:
return "HIT: " + entry[1]
return "MISS: call the model"
for use_key in (False, True):
print(f"cache key includes console version: {use_key}")
for query, sim, key in requests:
print(f" {sim:.2f} {query[:45]:45} {lookup(sim, key, 0.90, use_key)}")
Output:
cache key includes console version: False
0.91 Where do I regenerate an API key? HIT: Steps for the current console
0.93 How do I reset my API key for the legacy cons HIT: Steps for the current console
1.00 How do I reset my API key? HIT: Steps for the current console
cache key includes console version: True
0.91 Where do I regenerate an API key? HIT: Steps for the current console
0.93 How do I reset my API key for the legacy cons MISS: call the model
1.00 How do I reset my API key? HIT: Steps for the current console
Adding the console version to the key stops the dangerous legacy-console hit. Notice that the legacy query scored higher than the safe one, so no threshold alone could separate them.
The third query still hits. If the console was redesigned, that answer is stale, and only an expiry rule tied to the redesign would catch it.
This is why threshold, expiry, and cache-key design matter more than the headline hit rate. Useful key fields often include product version, user permissions, and locale.
How to evaluate a semantic cache
Track three things separately:
- Hit rate: How often the cache returns a stored answer.
- Hit accuracy: How often those stored answers are correct for the new query.
- Miss cost: What the model calls on misses cost you.
To measure hit accuracy, sample hits and grade them. Ask whether the stored answer, word for word, correctly answers this new query for this user today.
That grading is an evaluation problem like any other. The Phoenix documentation covers running evaluators over sampled production traffic.
Also watch the near-miss band just below your threshold. Those queries show what a small threshold change would start serving.
Setting expiry and exclusions
Set expiry based on how fast the underlying facts change, not on convenience.
Exclude some categories of answer from the cache entirely. Good candidates are answers that are personalized, depend on permissions, or are time-sensitive.
A cached answer that leaks another account’s details is both a cache-key failure and a privacy incident.
Limitations of semantic caching
- Stale facts: Cached answers quietly become wrong as facts change. Without expiry tied to source changes, the cache keeps serving outdated information.
- Personalization leakage: An answer generated with one user’s data can be served to another user unless user context is part of the cache key.
- Threshold sensitivity: Set the threshold too low and similar wording with different intent produces confident wrong answers. Set it too high and the cache almost never hits.
- Similarity is not equivalence: Embedding similarity measures how alike two queries are in topic, not whether the same answer is correct. Negation, dates, versions, and quantities often fool it.
- Overstated savings: Embedding, lookup, storage, and evaluation all add overhead. A wrong hit also creates downstream cost, so confirm that total task cost actually fell.
Semantic caching vs. prompt caching
These two techniques are often confused, but they solve different problems.
Prompt caching reuses computation when the beginning of the prompt is identical. The model still generates a new answer for every request. Its effectiveness is measured by cache hit rate over tokens, and a hit can never return an answer meant for a different question.
Semantic caching skips generation entirely when a different but similar query matches. It saves the full generation cost. It can also serve one user’s answer, or last month’s facts, to someone else.
Prompt caching is the safer default for repeated system prompts and documents. Semantic caching suits narrow, high-volume sets of questions that genuinely repeat and have stable answers.
You can use both. Semantic hits skip the model call, and misses still benefit from prompt caching inside the call.
Where answers must reflect current sources, compare semantic caching with a retrieval-augmented generation design. That approach retrieves fresh evidence on every request instead of replaying a stored answer.
Frequently asked questions
How is semantic caching different from prompt caching?
Prompt caching reuses computation for an identical prompt prefix and still generates a fresh answer. Semantic caching returns a stored answer for a similar new query and skips generation entirely. That saves more, but it can serve an answer that does not fit.
When is semantic caching a good fit?
Semantic caching fits high-volume, repetitive questions with stable answers that are the same for every user. Examples include product FAQs, documentation lookups, and status explanations. It is a poor fit for personalized, permission-dependent, fast-changing, or high-stakes answers.
How do I choose the similarity threshold?
Start strict, then sample matches near the threshold and grade whether each stored answer correctly answers the new query. Lower the threshold only while sampled hit accuracy stays acceptable. Recheck after you change the embedding model or the content.
Can semantic caching return another user’s data?
Yes, if user-specific context shaped the stored answer and is not part of the cache key. Either exclude personalized exchanges from the cache, or include in the key whatever context changes the correct answer.
Putting semantic caching to work
Start with a narrow set of repetitive questions whose answers are stable and the same for everyone. Exclude personalized, permission-dependent, and time-sensitive answers from day one.
Decide your cache key fields, such as product version and locale, before you tune the threshold. Then begin with a strict threshold and set expiry rules tied to content changes.
Sample and grade hits on a regular schedule, and track hit accuracy beside hit rate. Evaluation tools such as Arize Phoenix can run graders over sampled hits so you can see whether a threshold change is serving wrong answers.
