What is an LLM gateway, and when do you need one?

An LLM gateway is a service that sits between your applications and the AI model providers they call: Anthropic, OpenAI, Google, AWS Bedrock, Azure OpenAI or a model you host yourself. Instead of every application holding its own provider keys and calling each provider directly, every application sends its requests to one endpoint, and the gateway forwards them.

That one change of address is the whole integration. Because the major SDKs let you set a base URL, putting a gateway in front of an existing application is usually a configuration change, not a rewrite.

Why teams put a gateway in front of their models

Calling a provider directly works well for one application and one team. It stops working well when several things become true at once:

• Several applications, several teams, several providers. Keys end up copied into many services. Nobody can say which team spent what, or revoke one application’s access without touching the others.
• The bill grows faster than usage. Model spend is driven by tokens, and tokens grow with longer prompts, agent loops, retries and retrieval context. Without a central place to see requests, there is no central place to reduce them.
• Sensitive data reaches prompts. Customer records, credentials and source code get pasted into prompts by people and pipelines alike. A policy that lives in every application is a policy that is missing from some of them.
• Providers fail. A rate limit or an outage at one provider becomes an outage of your product unless something can retry or fail over.

A gateway is the one place where all of that can be handled, once, for every application.

What an LLM gateway does

Different gateways emphasize different jobs. The useful ones fall into four groups.

Access and authentication

The gateway issues its own keys to your applications and people, and holds the provider credentials itself. Applications never see a provider key. Revoking an application’s access is one action, and a leaked application key exposes a scoped gateway key rather than your provider account.

Routing and reliability

The gateway decides which provider connection serves each request. That can be as simple as “this model name goes to this account”, or as involved as failing over between platforms that host the same model, or choosing the model per request by how hard the request is. See LLM model routing.

Cost reduction

Because every request passes through it, a gateway can avoid paying for the same work twice. The main techniques are:

• answering identical repeat requests from a response cache, with no provider call at all;
• answering near-identical prompts from a semantic cache (semantic caching);
• switching on each provider’s own prompt caching, so repeated context is billed at the cheaper cached-input rate (prompt caching);
• sending work nobody is waiting for to the provider’s batch tier (batch APIs);
• sending each request to the least expensive model that can handle it.

Governance and visibility

Budgets per team or person, rate limits, model allow-lists, data loss prevention on prompts, prompt-injection screening, and a record of every request with its tokens and cost. This is what lets finance and security teams answer questions about AI usage without asking every engineering team.

What to look for

Bring your own keys. A gateway that resells model capacity puts a margin on every token and puts your provider relationship in someone else’s hands. A bring-your-own-key gateway sends each request with your own provider account, so your contracts, discounts and data agreements stay yours.

Both major API shapes. Most applications speak either the OpenAI Chat Completions API or the Anthropic Messages API. A gateway that speaks both, and can translate between them, lets either SDK reach any model.

Optimizations that do not change the answer. Caching an identical request, de-duplicating concurrent identical requests and enabling provider prompt caching all return exactly what the provider would have returned. Anything that can change the answer, such as choosing a different model, should be something you opt into explicitly.

Measured, checkable savings. A gateway that claims to save money should record, per request, what the request cost and what it would have cost sent directly, so the claim can be reconciled against your provider invoice.

Where it runs. A hosted gateway is the quickest start. A self-hosted one keeps request traffic inside your own environment, which some data-residency and regulated workloads require.

Streaming, tools and images. Production traffic streams responses, calls tools and sends images. A gateway has to pass all of these through correctly, not just plain text completions.

When you do not need one

A single application, a single provider and a small, predictable bill rarely justify another hop in the request path. The case for a gateway gets stronger with every additional team, application and provider, and with every month the AI line on the invoice grows faster than the business does.

How the Tessarac AI Cost Optimizer fits

The Tessarac AI Cost Optimizer is a bring-your-own-key LLM gateway. It speaks the OpenAI and Anthropic APIs, connects to Anthropic, OpenAI, Google Vertex AI, AWS Bedrock, Azure OpenAI and OpenAI-compatible servers, and applies the techniques above wherever a request qualifies. Only requests that ask for model: "auto" let it choose the model. A request that names a model is answered by that model, with two documented exceptions you can switch off: the semantic cache may reuse a stored answer written by an equally or more capable model, and when a provider refuses a request with an empty answer, the gateway retries it once on a fallback model and says so (that retry is switched off per request). It runs hosted or in your own Kubernetes cluster, and on the hosted service you pay a share of the saving it measures (self-hosted installations take an annual licence). The quickstart takes about ten minutes.