LLM batch APIs: the discount for work nobody is waiting on

Not every model call has a person waiting on the other end. Nightly document classification, evaluation runs, embedding a backlog, summarizing yesterday’s tickets, generating product descriptions: the result is needed today, not in the next two seconds.

Every major provider prices that kind of work below real-time requests. Anthropic’s Message Batches API, OpenAI’s Batch API, the Gemini API’s batch mode and Vertex AI batch inference all run the same model and return the same kind of answer, at a discount to the synchronous price, in exchange for a slower turnaround. Each provider states its discount on its own pricing page.

What you trade away

Latency. A batch is a job, not a call. You submit many requests, poll for completion and fetch the results. Providers state their turnaround as a window:

• Anthropic: most batches finish in less than an hour; a batch that has not finished within 24 hours expires. (Batch processing)
• OpenAI: each batch completes within 24 hours, often sooner. (Batch API)
• Gemini API: the target is 24 hours, usually much quicker. (Batch mode)
• Vertex AI: most jobs complete within 24 hours of starting; incomplete work is cancelled and only completed requests are billed. (Batch inference)

Interactivity. No streaming, no follow-up turn mid-job, and a different code path for submitting and collecting results.

Some provider features. Check the documentation for anything you rely on. For example, Anthropic serves prompt-cache hits inside a batch on a best-effort basis, and its fast mode is not available with batches.

Which workloads fit

Good fits share one property: nobody is blocked while the request runs.

• Offline classification, tagging and extraction over a corpus
• Evaluation and regression suites for prompts and models
• Content generation that is reviewed later anyway
• Back-filling summaries, translations or embeddings
• Scheduled reports and digests

Poor fits are anything a user, an agent loop or another service is waiting on: chat, coding assistants, tool-calling agents, and any request inside a request.

Two ways to get the batch price

Opt in explicitly. Rewrite the job to use the provider’s batch API. This is the right choice for large, clearly offline workloads, and it is predictable.

Let the gateway decide. Many workloads are batch-shaped but were written synchronously: a script that loops over a thousand records and waits for each one. Rewriting all of them is rarely anyone’s priority.

The Tessarac AI Cost Optimizer supports both. You can submit batches explicitly through its Batches API. It can also decide on its own to send an ordinary synchronous request to the batch tier, under strict conditions: the batch price must actually be lower for that request (including any prompt-cache discount it would get synchronously), the batch tier must currently be answering about as fast as the real-time API or there must be good evidence nobody is waiting, the answer must be expected back within the hold window on the same connection, and the request must not be streaming. When anything is uncertain, the request runs synchronously, and a follow-up message in the same conversation pulls a held request straight back. This automatic routing applies to direct Anthropic and OpenAI connections, and can be switched off. See Batch routing.

Related tiers

Some providers also sell an in-between tier: real-time requests at a lower price with best-effort availability. OpenAI’s flex processing is priced at batch rates for slower, occasionally unavailable responses, and the Gemini API has a similar Flex tier. Both are in beta or preview at the time of writing. At the other end, priority or fast tiers charge more for lower latency, which is worth knowing when you read an unexpectedly high invoice line.

For the rest of the techniques that lower a model bill, see how to reduce LLM API costs.