Batch routing
ACOG can send requests nobody is waiting for to your provider's half-price batch tier automatically, and only when it is safe and actually cheaper.
Provider batch tiers cost about half as much, for the same model and the same answer, but take longer. You can opt any request in yourself (see Batches). Batch routing goes further: ACOG decides on its own to batch a request when that costs the caller nothing.
When ACOG batches a request on its own
All of these must hold:
- Batching is cheaper for this request. ACOG compares the batch price with what the request would cost synchronously, including your provider prompt-cache discounts. A request whose prompt would mostly be read from cache can be cheaper synchronously.
- Waiting costs nothing. Either the provider’s batch tier is currently answering about as fast as its synchronous API (measured), or ACOG has good evidence nobody is waiting for this answer, on a batch tier measured as healthy.
- The answer can arrive on the same connection. The batch job must be expected to finish within the hold window (50 seconds on the hosted service), because a caller who did not opt in must receive an ordinary response, never a job handle.
- It is not a streaming request.
When anything is uncertain (no recent measurement, an unclear intent) the request is sent synchronously. A wrong guess here would cost someone minutes; a missed saving costs half of one request.
If the caller sends a follow-up while a request is held in the batch tier, ACOG pulls the held request back and answers it synchronously, and does not batch that conversation again for a day.
Which providers
Batch routing applies to Direct Anthropic and Direct OpenAI connections. Bedrock and Vertex AI batch jobs take hours and return job handles, so ACOG never chooses them on its own; opt in explicitly to use them.
Turning it off
Under Admin → Gateway settings, turn off Batch auto-routing. Requests that opt in explicitly are unaffected.
In reports
Batched requests are recorded with the disposition Batch and priced at the batch rate. ACOG’s own measurement requests, which keep the batch tier’s latency known, appear as acog-batch-probe and are billed by your provider.