Semantic cache
Answer a prompt that means the same as an earlier one from the earlier answer, above a similarity threshold, and never from a weaker model.
The response cache only helps when a request is byte-for-byte identical. The semantic cache also answers a prompt that means the same thing as an earlier one, for example “What’s our refund policy?” and “what is the refund policy”, from the earlier answer, without calling your provider.
Because a similar prompt is not an identical one, the semantic cache is more careful than the response cache:
- A similarity threshold. A stored answer is used only when the prompts are at least this similar, on a scale from 0 to 1. The recommended and default threshold is 0.97, a strict bar that admits only very close matches.
- Never a weaker model. A stored answer is used only if the model that wrote it is at least as capable as the model the new request asked for, by your auto mode tiers. Without tiers it is used only for the same model. An Opus answer can serve a Haiku request; a Haiku answer never serves an Opus request.
- Never a different API format. A stored answer is used only for a request expecting the same response format: an Anthropic Messages reply answers only Messages requests, and an OpenAI chat completion only Chat Completions requests.
Where it applies
| Applies | Does not apply |
|---|---|
Non-streaming requests to /v1/messages, /v1/chat/completions, and the native /anthropic/v1/messages and /openai/v1/chat/completions | Streaming requests |
| Requests translated between API shapes | |
Token counting, embeddings, Responses, Gemini generateContent, Bedrock and Azure native routes | |
| Prompts longer than the embedding model can read in full. Two long prompts that share an opening would look identical, so ACOG does not guess. | |
Requests sent with x-acog-cache-control: no-cache or no-store, and organizations with the response cache turned off |
Settings
Under Admin → Gateway settings → Semantic cache:
| Setting | Default |
|---|---|
| Semantic cache | On for hosted organizations |
| Similarity threshold | 0.97 |
Lowering the threshold increases hits, and also the chance of serving an answer to a question that only looks similar. The recommended threshold is 0.97.
What you see
A semantic hit returns x-acog-cache: hit, makes no provider call, costs nothing, and is recorded as Semantic cache in reports. On the compatible endpoints, x-acog-cache-model names the model that generated the stored answer.
A semantic hit is billed like any other saving: the saving is what the request would have cost at the model you asked for.
Privacy
Each organization’s semantic cache is separate; nothing is shared between organizations. Stored answers are encrypted at rest.
Self-hosted installations
The semantic cache needs an embedding service and a vector database (Milvus). The hosted service has both. A self-hosted installation runs it only after the operator enables them; see Configuration.