# Caching

> The response cache, in-flight de-duplication and your provider's prompt caching: what each one saves, when it applies and how to control it.

Product: AI Cost Optimizer Gateway (ACOG). Web version: https://tessarac.com/docs/acog/optimize/caching. Last updated: 2026-10-16.

ACOG uses three kinds of caching, all on by default and none of which changes an answer.

| | What is reused | Saves | Applies to |
|---|---|---|---|
| Response cache | A complete earlier response to an identical request | The whole request | Identical, non-streaming requests |
| In-flight de-duplication | One upstream call shared by identical requests arriving together | Every request but the first | Identical, non-streaming requests in flight at the same time |
| Provider prompt caching | The provider's processing of a repeated prompt prefix | Most of the input cost of the repeated part | Every request to providers that support it |

A fourth, the [semantic cache](https://tessarac.com/docs/acog/optimize/semantic-cache.md), answers near-identical prompts and is described on its own page.

## Response cache

When a request is identical to an earlier one (same model, same normalized body), ACOG returns the stored response without calling your provider. The reply carries `x-acog-cache: hit`, costs nothing, and is recorded as an **Exact cache** hit.

- Requests are compared after normalization: key order, whitespace in the JSON and `cache_control` markers do not make two requests different.
- Entries are kept for one hour by default, encrypted, and only for your organization.
- Only successful responses are stored. Errors, and responses a provider's safety system refused, are never stored.

The response cache does not apply to:

- streaming requests;
- requests translated between API shapes (for example an OpenAI-shaped request served by a Claude model);
- requests for audio output, and media generation that is not provably deterministic;
- organizations that have turned the cache off.

### Turning it off

Under **Admin → Gateway settings**, turn off **Response cache** to stop ACOG storing or serving cached responses for your organization, for example for privacy-sensitive workloads. The same page can flush the cache. Turning it off also turns off de-duplication and the semantic cache.

### Controlling the response cache per request

Send `x-acog-cache-control` with one or more comma-separated directives:

| Directive | Effect |
|---|---|
| `no-cache` | Do not look up a cached response, but store this one |
| `no-store` | Neither look up nor store |
| `ttl=<seconds>` | Keep this response for this long (1 to 86400) |
| `namespace=<name>` | Keep this request's cache separate: entries in one namespace never answer another. Letters, digits, `.`, `_` and `-`, up to 64 characters. |

```bash
curl -s https://gateway.acog.tessarac.com/v1/messages \
  -H "Authorization: Bearer $ACOG_API_KEY" \
  -H "x-acog-cache-control: no-cache" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-haiku-4-5", "max_tokens": 64, "messages": [{"role": "user", "content": "Hello"}]}'
```

Unknown directives are ignored.

## In-flight de-duplication

When identical requests arrive while the first is still waiting for the provider, ACOG makes one upstream call and gives every caller the same response. The extra callers' responses carry `x-acog-cache: hit` and are recorded as **Dedup**. This commonly saves money on retries and on fan-out from several workers. It applies under the same conditions as the response cache.

## Provider prompt caching

Providers bill repeated prompt prefixes at a much lower cached-input rate, but only when the request asks for caching. ACOG adds what each provider needs:

| Provider | What ACOG does |
|---|---|
| Anthropic (Direct, Bedrock and Vertex) | Adds `cache_control` breakpoints on the end of the tool definitions, the system prompt and the conversation so far. It adds at most three, never more than Anthropic allows, and never changes breakpoints you set. |
| OpenAI and Azure OpenAI | Sets `prompt_cache_key` from your system prompt and tools, so requests sharing them land on the same cache |
| Gemini on Vertex | References a cached context when one exists for the request's prefix |
| Custom endpoints | Left unchanged (servers declared as Qwen excepted) |

This changes how the request is billed, not what it asks: the provider produces the same answer.

### Cache lifetime

Anthropic and OpenAI let a request choose how long its prompt-cache entry lives. Longer lifetimes cost more to write and save more on reuse. ACOG decides the lifetime per request, from the first of:

1. the request header `x-acog-cache-ttl`: `auto`, `in_memory`, `5m`, `30m`, `1h`, `24h` or `forever`;
2. the API key's cache lifetime setting;
3. the connection's **Cache lifetime** setting (**Admin → Providers**);
4. the default: 5 minutes.

A lifetime the provider cannot honor is dropped (the provider's default applies) rather than causing an error. Anthropic offers 5 minutes and 1 hour; OpenAI offers in-memory and 24 hours.

### Cache refresh pings

An optional setting under **Admin → Gateway settings** keeps your provider's prompt caches warm between requests by sending small refresh requests before entries expire. It is off by default because it spends money to save money, and it may be unavailable on some deployments. Refresh requests are billed by your provider and appear in reports as `acog-cache-refresh`.