How to reduce LLM API costs: a practical checklist

AI model bills grow in a way most infrastructure bills do not. A feature that cost little in testing becomes a large line item once real users, longer conversations and agent loops arrive. Coding assistants and tool-using agents are the steepest case: every step resends a growing context, so one long task can send the same system prompt and files many times over.

Part of almost every bill pays for work that has already been done, or for capability the request did not need. This guide is the checklist we work through, in order, from the changes that never alter an answer to the ones that need care.

1. Measure before you cut

You cannot reduce what you cannot attribute. Before changing anything, make sure every request records:

• who made it (the application, team or person, through its own key, not a shared provider key);
• which model answered, and the tokens it used, including the cached-input tokens every major provider reports in the response (Anthropic and OpenAI also report cache writes);
• what it cost, priced at the time.

This alone usually surfaces the biggest wins: one feature on a frontier model it does not need, one agent loop with no limit, one script retrying on every timeout. See controlling AI spend with budgets and attribution.

2. Stop paying twice for identical requests

Response caching. Identical requests (same model, same messages, same parameters) can be answered from a stored response, with no provider call. Normalize requests before comparing them, store only successful responses, and scope entries to one tenant.

De-duplicate requests in flight. When identical requests arrive while the first is still waiting on the provider, make one upstream call and share the result. Retries and parallel workers cause more of these than most teams expect.

Semantic caching. Users ask the same question in different words. A semantic cache answers near-identical prompts from an earlier answer. It is powerful for support and knowledge assistants, and needs strict safeguards. See semantic caching for LLMs.

3. Make repeated context cheap with prompt caching

Every major provider bills a repeated prompt prefix at a reduced cached-input rate, but only when the prefix is byte-for-byte stable and, on some providers, only when the request asks for caching. Put tool definitions, system prompts and reference documents first; put the user’s input and anything variable last; never put timestamps or IDs in the stable part. For agentic and coding workloads this is often the single largest lever. See prompt caching on Anthropic, OpenAI and Gemini.

4. Remove tokens that carry no meaning

Input tokens are billed whether or not they matter. Two reductions are safe because they change nothing the model needs:

• collapse redundant whitespace and remove invisible characters (zero-width spaces and similar) from message text, leaving code blocks alone;
• drop retrieval chunks that repeat a chunk already in the same request.

Beyond that, trim with care: shorter system prompts and fewer retrieved passages lower cost, but they can change answers, so test them like any other prompt change.

5. Send work nobody is waiting on to the batch tier

Anthropic, OpenAI and Google all run batch requests on the same models at a discount to the real-time price, with results within a window of up to a day. Nightly jobs, evaluations, backfills and bulk classification belong there. See LLM batch APIs.

6. Use the least costly model that can do the job

Within one provider, model tiers can differ in price per token by an order of magnitude or more. Most applications send everything to one capable model. Route easy requests (lookups, short answers, reformatting, simple classification) to a lighter model, and keep the strong model for the work that needs it.

Routing is the one lever on this list that changes which model answers, so it needs safeguards: grade difficulty rather than prompt length, keep a conversation on one model, and send anything uncertain to the strongest tier. See LLM model routing.

A related lever never changes the model: the same model is often sold on several platforms (Claude, for example, on Anthropic, AWS Bedrock and Google Vertex AI), and your committed-use discounts and credits differ between them. Sending each request to the platform that is least costly for you lowers the bill while the same model answers.

7. Put limits in place

Budgets per organization, team and person, with warnings before anything is blocked; rate limits for runaway loops; model access rules so the most expensive models are available only to the teams that need them. Limits do not reduce the cost of a request, but they stop one mistake becoming a month’s budget.

8. Prove the saving

For every request, record what it actually cost and what it would have cost sent directly to the model that was asked for, at the provider’s list price. The difference is the saving, and it is checkable: a request answered from cache, for example, simply does not appear on the provider’s invoice.

Doing all of this in one place

Each item on this list can be built into each application. In practice, few teams keep eight cost techniques correct across every service, provider and prompt change. That is what an LLM gateway is for.

The Tessarac AI Cost Optimizer is a bring-your-own-key gateway that applies these techniques wherever a request qualifies: response caching, in-flight de-duplication, the semantic cache, provider prompt caching, prompt cleanup, automatic batch routing, opt-in model routing with model: "auto", and cost arbitrage across platforms (off until you set it up). A request that names a model is answered by that model, with two documented exceptions you can switch off: the semantic cache may reuse an equally or more capable model’s answer, and a request a provider refuses with an empty answer is retried once on a fallback model (switched off per request). The gateway records the actual and direct cost of every request, and on the hosted service bills only a share of the saving it measures. See how ACOG saves money for every mechanism and its default, or the pricing page.