# Streaming

> Stream responses through ACOG with server-sent events, on every route and across API shapes, and what to expect from timeouts and keep-alives.

Product: AI Cost Optimizer Gateway (ACOG). Web version: https://tessarac.com/docs/acog/api/streaming. Last updated: 2026-10-16.

Set `stream: true` (or use your SDK's streaming method) on any route, compatible or native. ACOG relays the stream as it arrives, as server-sent events (`text/event-stream`).

```python
import os
from anthropic import Anthropic

client = Anthropic(base_url="https://gateway.acog.tessarac.com", auth_token=os.environ["ACOG_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a haiku about caching."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
```

## Streams across API shapes

A streamed request is translated like any other. An OpenAI-shaped streaming request for a Claude or Gemini model receives OpenAI `chat.completion.chunk` events ending with `data: [DONE]`. An Anthropic-shaped streaming request for a GPT or Gemini model receives Anthropic events ending with `message_stop`. Bedrock's binary event stream is converted to server-sent events on the compatible routes.

Token usage is read from the stream itself, so streamed requests are measured and billed exactly like non-streamed ones.

## Keep-alives and timeouts

| | |
|---|---|
| Keep-alive | Every 15 seconds without data, ACOG sends an SSE comment line (`: keep-alive`). SDKs ignore it. It stops proxies and load balancers from closing a stream while the model is thinking. |
| Provider silence | If the provider sends nothing for 10 minutes (hosted) or 2 minutes (self-hosted, unless your operator sets `ACOG_UPSTREAM_READ_TIMEOUT_SECS`), ACOG ends the request. There is no limit on a stream's total length. |

**Prefer streaming for long generations.** A non-streaming request sends nothing until the whole answer is ready. On `gateway.acog.tessarac.com` it may wait up to 10 minutes; on the console's `acog.tessarac.com/gateway` it is closed after 60 seconds without data (`504`). A streamed request shows its first tokens at once and is kept open by its keep-alives for as long as it takes.

## What changes when you stream

- Streamed responses are never stored in the response cache or the semantic cache, and identical streamed requests are not de-duplicated, because a stream cannot be replayed.
- Batch routing never applies to a streamed request.
- A budget warning is appended as a final text event before the stream ends; see [Budgets](https://tessarac.com/docs/acog/govern/budgets.md#warnings-in-responses).
- Data loss prevention scans the request before it is sent. Streamed replies are not scanned.