LLM model routing: sending each request to the least costly model that can answer it

Providers sell their models in tiers. A lightweight model, a balanced everyday model, an advanced reasoning model and a frontier model can sit an order of magnitude or more apart in price per token, from the same provider. Yet most applications pick one model, usually a capable one because it was the safe choice during development, and send it everything.

Everything includes a lot of easy work: greetings, lookups, reformatting, short classifications, simple extraction. A frontier model answers those correctly. So would a model that costs a small part of the price.

Model routing means choosing the model per request instead of per application. Done well, it is one of the largest levers on an AI bill. Done badly, it quietly degrades answers, which costs far more than it saves.

Three ways to route

Static routing by task

The application decides which model each feature uses: the classifier calls a lightweight model, the planner calls a frontier one. It is simple, predictable and easy to test, and it is the right first step. Its limit is that one feature sees both easy and hard requests, and a static choice has to be sized for the hard ones.

Rules and heuristics

Route on things you can measure cheaply: prompt length, whether tools or images are involved, which customer tier is calling, keywords. Rules catch obvious cases well. They are brittle at the boundary, and they need maintenance every time the models change.

Difficulty grading

Grade each request’s difficulty, then send it to the least costly model whose capability clears that grade. The grader can be a classifier you train, or a small, inexpensive model asked to judge the request. This adapts to the request itself rather than to the feature it came from.

What goes wrong

Grading the wrong thing. A short prompt is not necessarily an easy one. “Prove this lemma” is short.

Routing on price instead of capability. A model’s tier should reflect what it can do. Ranking by price alone puts an expensive but weak model above a strong but inexpensive one.

Switching models mid-conversation. An agent that starts a task on one model and continues it on another loses consistency, and on providers with prompt caching it throws the cache away. A conversation should stay on one model.

Failing open to the cheap side. When the grader errors, times out or is unsure, the safe default is the stronger model. A wrongly cheap answer that a user acts on costs more than a missed saving.

Ignoring capabilities. A request with images or tool calls must only go to models that support them.

Not measuring. Without recording which model answered and what the request would have cost on your default model, nobody can tell whether routing is saving money or degrading quality.

Routing without changing models you chose

There is a second kind of routing that never changes the model: the same model is often sold on several platforms. Claude, for example, is available from Anthropic directly, on AWS Bedrock and on Google Vertex AI. List prices differ by platform, and your committed-use discounts and credits differ even more. Sending each request for that model to whichever platform is least costly for you lowers the bill while the same model answers.

How the Tessarac AI Cost Optimizer routes

The AI Cost Optimizer does both, and keeps them separate.

Auto mode applies only to requests that ask for it with "model": "auto". A request that names a model is answered by that model, with two documented exceptions you can switch off: the semantic cache may reuse a stored answer written by an equally or more capable model, and when a provider refuses a request with an empty answer, the gateway retries it once on a fallback model and says so (that retry is switched off per request). Each model in your pool is placed on one of four tiers (lightweight, balanced, advanced, frontier) by what it can do, never by its price, and you can move any model. Each auto request is first given a free structural check, and if that does not settle it, a small judge model (the least costly one in your own pool, on your own connection) grades it. On any doubt or error the request goes to the strongest tier. Conversations stay on one model, recognized by session headers, and requests needing images or tools only go to models that support them. You can also make your default model a ceiling, so auto mode never routes above its price. See Auto mode.

Cost arbitrage sends a model available on several platforms to the one that is least costly for you, counting the reserved rates and credits you enter. It is off by default, and never changes which model runs. See Cost arbitrage.

Every request records the model that answered and what it would have cost sent directly (for an auto request, priced at your default model), so the saving and the routing decision can be checked request by request.

Routing pairs well with the techniques that avoid paying for repeated work: prompt caching, semantic caching and batch APIs.