Semantic caching for LLMs: answering repeat questions without a model call

The least costly model call is the one you do not make. If an application has already answered a question, answering it again from a stored response costs nothing upstream and returns instantly.

There are two levels of this, and they have very different risk profiles.

Response caching: identical requests

A response cache stores a complete response keyed on the full request: the model, the messages, the parameters. When an identical request arrives, the stored response is returned without calling the provider.

Because the request is identical, the answer is exactly what the provider returned last time. That makes response caching safe to switch on broadly, with a few rules:

• Normalize before comparing. Two requests that differ only in JSON key order or whitespace are the same request.
• Store only successes. Never cache an error, a timeout or a response the provider’s safety system refused.
• Keep entries short-lived and scoped. An hour is a sensible default. Never share entries between tenants.
• Skip what is not deterministic or not cacheable. Streaming responses and most media generation are usually excluded.

A close relative is in-flight de-duplication: when identical requests arrive while the first is still waiting on the provider, make one upstream call and give every caller the same response. It catches retries and fan-out from parallel workers.

Semantic caching: requests that mean the same thing

People rarely ask a question the same way twice. “What’s our refund policy?” and “what is the refund policy” are different strings and identical questions. A response cache misses them; a semantic cache does not.

A semantic cache embeds each prompt as a vector, stores it with its response, and on a new request looks for a stored prompt whose embedding is close enough. If one is found above a similarity threshold, the stored answer is returned.

That is powerful for FAQ-style traffic: support bots, internal knowledge assistants, documentation search. It is also where caching can go wrong, because a similar prompt is not an identical one.

Where semantic caching goes wrong

A threshold set too low. “Cancel my order” and “don’t cancel my order” can be close in embedding space. A loose threshold serves the wrong answer. A strict threshold, close to identical wording, is the safe default; loosen it only with evidence from your own traffic.

Serving a weaker model’s answer. If a request asked for a frontier model, an answer written by a lightweight model is not an equivalent substitute, however similar the question. A stored answer should only serve requests for a model of equal or lower capability.

Crossing API formats. An answer stored in one API’s response format cannot simply be replayed to a client expecting another.

Long prompts that share an opening. Embedding models read a limited number of tokens. Two long prompts that share their first part and differ later can produce near-identical embeddings. A semantic cache should not guess on prompts longer than its embedding model can read in full.

Context the prompt does not show. If the right answer depends on the user, the time or the state of a system, two identical-looking prompts can still need different answers. Use a separate cache namespace per context, or exclude those requests.

Privacy. A semantic cache stores prompts and answers. It must be scoped to one organization and encrypted at rest, and there must be a way to turn it off for sensitive workloads.

How the Tessarac AI Cost Optimizer caches

The AI Cost Optimizer runs all three, with the safeguards above:

• The response cache answers identical non-streaming requests, compared after normalization. Entries last an hour by default, are encrypted and are scoped to your organization; errors and refused responses are never stored. A request header can skip the cache, skip storing, change the lifetime or put a request in its own namespace.
• In-flight de-duplication shares one upstream call between identical concurrent requests.
• The semantic cache answers near-identical prompts at a default similarity threshold of 0.97. It never serves an answer written by a less capable model than the one requested, never crosses API formats, and does not apply to prompts longer than its embedding model can read in full. It is on by default for hosted organizations; a self-hosted installation runs it once the operator enables its embedding service and vector database.

Every hit is labeled in a response header and recorded in reports, with what the request would have cost at the model you asked for. See Caching and Semantic cache.

Caching avoids repeated requests. For requests that are new but share a long, repeated context, see prompt caching.