Streaming

Set stream: true (or use your SDK’s streaming method) on any route, compatible or native. ACOG relays the stream as it arrives, as server-sent events (text/event-stream).

import os
from anthropic import Anthropic

client = Anthropic(base_url="https://gateway.acog.tessarac.com", auth_token=os.environ["ACOG_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a haiku about caching."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Streams across API shapes

A streamed request is translated like any other. An OpenAI-shaped streaming request for a Claude or Gemini model receives OpenAI chat.completion.chunk events ending with data: [DONE]. An Anthropic-shaped streaming request for a GPT or Gemini model receives Anthropic events ending with message_stop. Bedrock’s binary event stream is converted to server-sent events on the compatible routes.

Token usage is read from the stream itself, so streamed requests are measured and billed exactly like non-streamed ones.

Keep-alives and timeouts

Keep-alive	Every 15 seconds without data, ACOG sends an SSE comment line (: keep-alive). SDKs ignore it. It stops proxies and load balancers from closing a stream while the model is thinking.
Provider silence	If the provider sends nothing for 10 minutes (hosted) or 2 minutes (self-hosted, unless your operator sets ACOG_UPSTREAM_READ_TIMEOUT_SECS), ACOG ends the request. There is no limit on a stream’s total length.

Prefer streaming for long generations. A non-streaming request sends nothing until the whole answer is ready. On gateway.acog.tessarac.com it may wait up to 10 minutes; on the console’s acog.tessarac.com/gateway it is closed after 60 seconds without data (504). A streamed request shows its first tokens at once and is kept open by its keep-alives for as long as it takes.

What changes when you stream

• Streamed responses are never stored in the response cache or the semantic cache, and identical streamed requests are not de-duplicated, because a stream cannot be replayed.
• Batch routing never applies to a streamed request.
• A budget warning is appended as a final text event before the stream ends; see Budgets.
• Data loss prevention scans the request before it is sent. Streamed replies are not scanned.