Caching

ACOG uses three kinds of caching, all on by default and none of which changes an answer.

	What is reused	Saves	Applies to
Response cache	A complete earlier response to an identical request	The whole request	Identical, non-streaming requests
In-flight de-duplication	One upstream call shared by identical requests arriving together	Every request but the first	Identical, non-streaming requests in flight at the same time
Provider prompt caching	The provider’s processing of a repeated prompt prefix	Most of the input cost of the repeated part	Every request to providers that support it

A fourth, the semantic cache, answers near-identical prompts and is described on its own page.

Response cache

When a request is identical to an earlier one (same model, same normalized body), ACOG returns the stored response without calling your provider. The reply carries x-acog-cache: hit, costs nothing, and is recorded as an Exact cache hit.

• Requests are compared after normalization: key order, whitespace in the JSON and cache_control markers do not make two requests different.
• Entries are kept for one hour by default, encrypted, and only for your organization.
• Only successful responses are stored. Errors, and responses a provider’s safety system refused, are never stored.

The response cache does not apply to:

• streaming requests;
• requests translated between API shapes (for example an OpenAI-shaped request served by a Claude model);
• requests for audio output, and media generation that is not provably deterministic;
• organizations that have turned the cache off.

Turning it off

Under Admin → Gateway settings, turn off Response cache to stop ACOG storing or serving cached responses for your organization, for example for privacy-sensitive workloads. The same page can flush the cache. Turning it off also turns off de-duplication and the semantic cache.

Controlling the response cache per request

Send x-acog-cache-control with one or more comma-separated directives:

Directive	Effect
no-cache	Do not look up a cached response, but store this one
no-store	Neither look up nor store
ttl=<seconds>	Keep this response for this long (1 to 86400)
namespace=<name>	Keep this request’s cache separate: entries in one namespace never answer another. Letters, digits, ., _ and -, up to 64 characters.

curl -s https://gateway.acog.tessarac.com/v1/messages \
  -H "Authorization: Bearer $ACOG_API_KEY" \
  -H "x-acog-cache-control: no-cache" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-haiku-4-5", "max_tokens": 64, "messages": [{"role": "user", "content": "Hello"}]}'

Unknown directives are ignored.

In-flight de-duplication

When identical requests arrive while the first is still waiting for the provider, ACOG makes one upstream call and gives every caller the same response. The extra callers’ responses carry x-acog-cache: hit and are recorded as Dedup. This commonly saves money on retries and on fan-out from several workers. It applies under the same conditions as the response cache.

Provider prompt caching

Providers bill repeated prompt prefixes at a much lower cached-input rate, but only when the request asks for caching. ACOG adds what each provider needs:

Provider	What ACOG does
Anthropic (Direct, Bedrock and Vertex)	Adds cache_control breakpoints on the end of the tool definitions, the system prompt and the conversation so far. It adds at most three, never more than Anthropic allows, and never changes breakpoints you set.
OpenAI and Azure OpenAI	Sets prompt_cache_key from your system prompt and tools, so requests sharing them land on the same cache
Gemini on Vertex	References a cached context when one exists for the request’s prefix
Custom endpoints	Left unchanged (servers declared as Qwen excepted)

This changes how the request is billed, not what it asks: the provider produces the same answer.

Cache lifetime

Anthropic and OpenAI let a request choose how long its prompt-cache entry lives. Longer lifetimes cost more to write and save more on reuse. ACOG decides the lifetime per request, from the first of:

1. the request header x-acog-cache-ttl: auto, in_memory, 5m, 30m, 1h, 24h or forever;
2. the API key’s cache lifetime setting;
3. the connection’s Cache lifetime setting (Admin → Providers);
4. the default: 5 minutes.

A lifetime the provider cannot honor is dropped (the provider’s default applies) rather than causing an error. Anthropic offers 5 minutes and 1 hour; OpenAI offers in-memory and 24 hours.

Cache refresh pings

An optional setting under Admin → Gateway settings keeps your provider’s prompt caches warm between requests by sending small refresh requests before entries expire. It is off by default because it spends money to save money, and it may be unavailable on some deployments. Refresh requests are billed by your provider and appear in reports as acog-cache-refresh.