LLM Semantic Caching: Skip the Call, Not Just the Tokens

You turned on prompt caching, your input bill dropped, and you assumed you had solved repeated work. But prompt caching still runs the model on every call. If two users ask the same question in different words, you pay for the full generation twice. A semantic cache can skip that second call entirely.

What is an LLM semantic cache?#

An llm semantic cache is a vector store of past query-to-response pairs that you check before calling the model. You embed the incoming query, search for the nearest stored query, and if cosine similarity clears your threshold you return that stored response at 0 model cost and near-0 latency. Otherwise you call the model.

The word that matters is "semantic." A plain key-value cache only hits when the new request is byte-for-byte identical to an old one, so "reset my password" and "how do I reset my password" miss each other completely. A semantic cache compares the meaning of the two queries through their embeddings, so paraphrases land on the same entry. That is what lets it help real users, who almost never phrase the same intent the same way twice.

Embed, search, compare against the threshold. A hit returns a stored answer; a miss calls the model and stores the new pair.

The mechanics are a 5-step loop: embed the query, search the vector store, compare the top match against a threshold, then either return the hit or call the model and store the new pair. Open-source tools like GPTCache implement exactly this loop, and Redis ships a semantic cache built on the same idea. You are not inventing anything here, just wiring known parts together.

How is semantic caching different from prompt caching?#

Prompt caching and semantic caching solve different halves of the problem, and they compose. Prompt caching discounts the repeated input prefix but still runs the model, so you always get a fresh generation. Semantic caching can skip the model call outright when a similar query already has an answer. One trims tokens; the other removes calls.

Both major providers document prompt caching as a discount on cached input tokens, not a way to avoid generation. Anthropic's prompt caching docs and OpenAI's prompt caching guide both describe reusing a static prefix (a long system prompt, a document, a tool schema) across calls. The model still reads your changing suffix and generates a new completion every time. That is the right tool when the answer must be fresh but the context is stable.

Prompt caching discounts a repeated prefix but still runs the model; a semantic cache can skip the call. They compose.

Semantic caching sits one layer earlier. It asks a blunter question: does this query even need the model? For a support bot answering the same 30 questions all day, the answer is often no. The clean move is to stack them: put a semantic cache in front to short-circuit repeats, and keep prompt caching on for the calls that do reach the model. If you are working through cost broadly, our guide to 7 levers to reduce AI API costs treats caching as one lever among several.

How do you build a semantic cache, step by step?#

You build it as a lookup that wraps your model call: pick an embedding model, store each query with its embedding and response, and on every new query embed and search before you generate. If the nearest neighbor clears the threshold, return the stored answer. If not, call the model, then write the new pair back.

The 5 steps below are the whole thing. Step 5 is the one people skip, and it is where a naive cache goes wrong: an untuned threshold either returns confidently wrong answers or never hits, and a cache with no invalidation will happily serve a price from 3 months ago. Treat tuning and expiry as part of the build, not a later chore.

Several of these true and a cache in front of the model removes real calls; few true and it is overhead.

1. Pick an embedding model. Choose one embedding model and use it for both stored queries and incoming ones. Mixing models makes the vectors incomparable and your similarity scores meaningless.
2. Store query, response, and embedding. On a miss, write the query text, the model's response, and the query's embedding into a vector store as one record.
3. Embed and search on each new query. Embed the incoming query with the same model and run a nearest-neighbor search for the closest stored query.
4. Return on a hit above the threshold. If the top match's cosine similarity is at or above your threshold, return its stored response and stop. No model call happens.
5. Tune the threshold and set invalidation. Sweep the threshold on real traffic, and add a time-to-live or an explicit purge so stale answers expire.

When does a semantic cache actually pay off?#

A semantic cache pays off when queries repeat in meaning and the correct answer is stable over time. High-repeat, low-volatility workloads are the sweet spot: FAQs, product documentation, onboarding help, and support macros where thousands of users ask the same 20 or 30 things. There the hit rate is high and a stored answer stays correct for weeks.

It is the wrong tool for personalized or time-sensitive answers. A response that depends on the specific user, their account, the current time, or live data must not be shared across users through a cache, because a hit would hand one person another person's answer or a stale fact. The markdown table below and the checklist above draw the line: cache the stable and shared, never the personal or the fresh.

Cache only stable, shared, non-time-sensitive answers; when in doubt, call the model and store the fresh pair.

The two failure modes both come from the threshold. Set it too low, say 0.75, and loosely related queries collide, so the cache returns a wrong stored answer with full confidence. Set it too high, near 0.99, and only near-identical strings hit, so you get almost no benefit. There is no universal number; a value around 0.85 is a common starting point you then tune against your own traffic and a labeled set of query pairs.

Where should you start?#

Start by measuring how repetitive your traffic really is, then cache only the stable, shared answers. Pull a week of production queries, cluster them by meaning, and count how many collapse into a handful of intents. If a large share does, a semantic cache will remove real calls; if every query is unique, it is overhead.

From there, wrap your model call with the 5-step loop, start the threshold around 0.85, and put a time-to-live on every entry. Keep prompt caching on underneath for the calls that still reach the model. If you are still choosing a provider for those calls, our breakdown of the cheapest AI API in 2026 pairs well with the call-avoidance a cache gives you.