Every gateway routes. Only one spends the minimum.
LiteLLM, a DIY proxy, and calling providers directly all move tokens. None of them look at how hard a prompt is, route it to the cheapest model that can answer it, verify that answer, or bill you on what they save. Here is exactly where the Tessarac AI Cost Optimizer is different — and where it isn't.
Every cost lever, side by side.
| Cost lever | Tessarac | LiteLLM | DIY gateway | Direct to provider |
|---|---|---|---|---|
| One endpoint, every provider (BYOK) | ✓ | ✓ | ~ | — |
| Load-balancing / failover | ✓ | ✓ | ~ | — |
| Difficulty-based model routing | ✓ | — | — | — |
| Answer check with escalation | ✓ | — | — | — |
| Full-response cache ($0, no upstream call) | ✓ | ~ | ~ | — |
| Same-model cheapest-deployment arbitrage | ✓ | ✓ | — | — |
| Prompt / RAG token trimming | ✓ | — | — | — |
| In-flight request coalescing | ✓ | — | — | — |
| Batch-tier orchestration | ✓ | ✓ | ~ | — |
| Per-request naive-vs-actual cost ledger | ✓ | — | — | — |
| Billed only on proven savings | ✓ | — | — | — |
✓ full · ~ partial / manual · — not offered.
The benchmark
How we test it against LiteLLM.
Both gateways get identical prompts. LiteLLM is pinned to the model a customer would otherwise choose;
the AI Cost Optimizer receives model: auto and
routes each prompt by difficulty. Both sides call real Vertex-backed Anthropic models, and the benchmark
harness computes the cost of every call itself, from one shared price book — never from a figure either
proxy reports about itself.
No gateway tax
A control run switches every saving lever off — caching off, and unique prompts that never repeat — and puts the two gateways within noise of each other: the optimizer adds no measurable cost of its own.
Where the two differ, and where they don't.
Five per-request mechanisms, each an advantage LiteLLM's own documentation confirms it lacks — and how provider prompt caching is counted.
Difficulty-based routing
A cheap-model judge grades each prompt and sends it to the cheapest model that can answer it — easy to haiku, medium to sonnet-5, hard to opus-4-8 or sonnet-5.
Answer check with escalation
When an auto-routed answer comes back empty, refused, or not the JSON the request asked for, the gateway retries it on the next candidate model, so a cheap pick that falls short is not the answer you get while another candidate is left.
Prompt simplification
Meaning-preserving whitespace and invisible-character trimming shrinks the token bill without changing the wording you sent.
RAG chunk de-duplication
Exact-duplicate retrieved chunks are dropped before they reach the model, so you never pay to send the same context twice.
In-flight request coalescing
Simultaneous identical calls collapse into a single upstream request, and the one answer fans back out to every caller.
Provider prompt caching
When your client marks its own prompts for caching (cache_control),
the optimizer keeps those markers (unless you choose a cache lifetime, which then applies to every marker), and that caching is yours, not ours. When it
doesn't, the optimizer adds them — and only the caching it caused counts toward what it bills.
Two ways to run it
Gateway-only, or the full savings tier.
You choose how much of the optimizer to switch on — per workload. Both modes are the same hosted endpoint, your keys, your models, your output.
Option 1
Gateway-only (passthrough)
Point at the AI Cost Optimizer as a hosted pass-through proxy with the cost-saving tier switched off — one endpoint, every provider, load-balancing and failover — and it simply proxies your calls. It behaves much like a LiteLLM-class gateway, except there is nothing to install, run, patch, or operate: you point at an endpoint instead of standing up and maintaining your own. You don't pay for savings features you're not using.
The control run above found that passthrough adds no measurable cost of its own.
Option 2
Full savings tier
Turn on auto-mode difficulty routing, response caching on by default, prompt simplification, RAG chunk de-duplication, and in-flight request coalescing. Same endpoint, same output — now every request pays the minimum it needs. This is the auto-mode configuration the benchmark above puts against LiteLLM.
Passthrough is the managed, LiteLLM-class baseline — the hosted alternative to running a gateway yourself. The savings tier is what it upgrades to. Start in passthrough and turn the savings levers on when you're ready.
Where each approach leaves money on the table.
vs. LiteLLM
LiteLLM is a capable router — it load-balances and fails over across the deployments you list, and it ships its own response cache. What it doesn't do, by its own docs, is grade a prompt's difficulty, route to the cheapest model that can answer it, or check that answer and retry another model when it falls short. Those are the levers that move the bill, and they are exactly what the Tessarac savings tier adds — with response caching on by default rather than operator-configured.
vs. a DIY gateway
You can build a proxy, a cache, and a router yourself — teams do. Then you own the eviction policy, the semantic-cache safeguards, the judge, the batch plumbing, and the price book, forever. Tessarac is that system, maintained, with a per-request ledger proving what it saved.
vs. calling providers directly
Straight-to-provider is the simplest path and the most expensive one: every call pays full price, every repeat regenerates, and every hard-coded model choice overpays on the easy prompts. One base-URL change routes it all through the optimizer instead.
Point one base URL at Tessarac.
Keep your keys, your models, and your output. Change the endpoint and pay for only the intelligence each request actually needs.