Every gateway routes. Only one spends the minimum.
LiteLLM, a DIY proxy, and calling providers directly all move tokens. None of them look at how hard a prompt is, route it to the cheapest model that can answer it, verify that answer, or bill you on what they save. Here is exactly where the Tessarac AI Cost Optimizer is different — and where it isn't.
Benchmarked
The same models, up to 80% cheaper than LiteLLM.
We sent identical prompts through the AI Cost Optimizer and through a LiteLLM gateway running the same premium models, priced both from one shared price book, and let auto-mode routing and caching do their work.
80.6%
lower total cost in auto mode across the benchmark suite
<250ms
to serve a cached answer instead of regenerating it
$0
upstream cost on every request served from cache
Measured · 2026-08-26
Auto mode, model by model.
LiteLLM was pinned to the model a customer would otherwise choose; the AI Cost Optimizer receivedmodel: auto and routed each prompt by difficulty. Identical prompts on both sides, real Vertex-backed Anthropic models, and cost computed independently by the benchmark harness from one shared price book — never read from a proxy's own figure.
| Requested model | LiteLLM | ACOG (auto) | Saved | ACOG routed to |
|---|---|---|---|---|
| claude-haiku-4-5 | $0.013624 | $0.014906 | −9.4% | haiku · opus-4-8 · sonnet-5 |
| claude-sonnet-5 | $0.037084 | $0.008619 | 76.8% | haiku · sonnet-5 |
| claude-sonnet-4-6 | $0.048639 | $0.008607 | 82.3% | haiku · sonnet-5 |
| claude-opus-4-8 | $0.090100 | $0.008584 | 90.5% | haiku · sonnet-5 |
| claude-opus-5 | $0.096635 | $0.014906 | 84.6% | haiku · opus-4-8 · sonnet-5 |
| Total | $0.286082 | $0.055622 | 80.6% |
Honest caveat. Against pinning thecheapest model (haiku), auto runs about 9% higher — it upgrades the hard prompts to a stronger model to protect quality. Against pinning any premium model, the realistic case, auto saves 77–90%.
No gateway tax
A control run with caching off and unique, never-repeated prompts put both gateways within noise of each other — $0.315737 LiteLLM vs$0.316679 ACOG. With every saving lever switched off, the optimizer adds no cost of its own — so the savings above come from real mechanisms, not measurement bias.
−0.3%
gateway overhead vs LiteLLM with all savings off — i.e. parity
Identical prompts both sides · cost computed by the harness, never from a proxy's self-report · real Vertex-backed Anthropic models (5 models × 3 prompts × 4 samples) · N=120, R=4 · auto run20260826T162744Z-61d0c8 · overhead control20260826T155014Z-723f12 · one shared price book (SHA-256 c976ec1b21517aae…) · zero failures.
What produces the 80.6%.
The saving is not a discount — it is a set of per-request mechanisms, each one an advantage LiteLLM's own documentation confirms it lacks.
Difficulty-based routing
A cheap-model judge grades each prompt and sends it to the cheapest model that can answer it — easy to haiku, medium to sonnet-5, hard to opus-4-8 or sonnet-5.
Answer-adequacy verification
The judge re-reads the response and escalates to a stronger model only when the cheap one falls short, so quality holds while cost drops.
Prompt simplification
Meaning-preserving whitespace and invisible-character trimming shrinks the token bill without changing the wording you sent.
RAG chunk de-duplication
Exact-duplicate retrieved chunks are dropped before they reach the model, so you never pay to send the same context twice.
In-flight request coalescing
Simultaneous identical calls collapse into a single upstream request, and the one answer fans back out to every caller.
Native prompt caching — parity
Anthropic's own prompt caching (cache_control) is passed through identically by both gateways — measured parity, not a differentiator, and not the source of these savings.
Every cost lever, side by side.
| Cost lever | Tessarac | LiteLLM | DIY gateway | Direct to provider |
|---|---|---|---|---|
| One endpoint, every provider (BYOK) | ✓ | ✓ | ~ | — |
| Load-balancing / failover | ✓ | ✓ | ~ | — |
| Difficulty-based model routing | ✓ | — | — | — |
| Answer-adequacy verification (judge) | ✓ | — | — | — |
| Full-response cache ($0, no upstream call) | ✓ | ~ | ~ | — |
| Same-model cheapest-deployment arbitrage | ✓ | ✓ | — | — |
| Prompt / RAG token trimming | ✓ | — | — | — |
| In-flight request coalescing | ✓ | — | — | — |
| Batch-tier orchestration | ✓ | ✓ | ~ | — |
| Per-request naive-vs-actual cost ledger | ✓ | — | — | — |
| Billed only on proven savings | ✓ | — | — | — |
✓ full · ~ partial / manual · — not offered. Internal benchmark, Aug 2026; savings vary by workload and traffic mix.
Two ways to run it
Gateway-only, or the full savings tier.
You choose how much of the optimizer to switch on — per workload. Both modes are the same hosted endpoint, your keys, your models, your output.
Option 1
Gateway-only (passthrough)
Point at the AI Cost Optimizer as a hosted pass-through proxy with the cost-saving tier switched off — one endpoint, every provider, load-balancing and failover — and it simply proxies your calls. It behaves much like a LiteLLM-class gateway, except there is nothing to install, run, patch, or operate: you point at an endpoint instead of standing up and maintaining your own. You don't pay for savings features you're not using.
The overhead control above confirms passthrough adds no measurable cost of its own (−0.3% versus a comparable gateway).
Option 2
Full savings tier
Turn on auto-mode difficulty routing, response caching on by default, prompt simplification, RAG chunk de-duplication, and in-flight request coalescing. Same endpoint, same output — now every request pays the minimum it needs. This is the measured80.6% auto-mode result above.
Passthrough is the managed, LiteLLM-class baseline — the hosted alternative to running a gateway yourself. The savings tier is what it upgrades to. Start in passthrough and turn the savings levers on when you're ready.
Where each approach leaves money on the table.
vs. LiteLLM
LiteLLM is a capable router — it load-balances and fails over across the deployments you list, and it ships its own response cache. What it doesn't do, by its own docs, is grade a prompt's difficulty, route to the cheapest model that can answer it, or verify that answer with a judge. Those are the levers behind the measured 80.6%, and they are exactly what the Tessarac savings tier adds — with response caching on by default rather than operator-configured.
vs. a DIY gateway
You can build a proxy, a cache, and a router yourself — teams do. Then you own the eviction policy, the semantic-equivalence gate, the judge, the batch plumbing, and the price book, forever. Tessarac is that system, maintained, with a per-request ledger proving what it saved.
vs. calling providers directly
Straight-to-provider is the simplest path and the most expensive one: every call pays full price, every repeat regenerates, and every hard-coded model choice overpays on the easy prompts. One base-URL change routes it all through the optimizer instead.
Point one base URL at Tessarac.
Keep your keys, your models, and your output. Change the endpoint and pay for only the intelligence each request actually needs.