LLM Gateways Sound Smart. Most Routing Is Just a Fallback Chain.

Your app calls Claude directly, and it works fine until the afternoon the API starts returning 529s and every request just stops answering. Switching to a backup provider would have meant an hour of rewriting client code, so you didn't have one ready. LLM model routing and gateways solve exactly this: a layer between your app and the model providers that decides where each request goes, so a bad afternoon for one provider doesn't become a bad afternoon for you.

What is an LLM gateway, and why not just call the provider's SDK directly?#

An LLM gateway is a proxy that sits between your application and one or more model providers, so your code calls one stable endpoint instead of a specific vendor's SDK. It normalizes the request shape, then decides which provider actually handles it, whether that decision is a fixed default or a live fallback.

Hardcode one provider's SDK and you inherit three problems at once: an outage on their side is an outage on yours, a price change lands on your bill with no warning, and moving to a different model later means rewriting every call site. A gateway does not remove these risks. It moves them into one place you can see and control, instead of leaving them scattered through your codebase.

If you have already made a client swappable through the OpenAI-compatible request shape, a gateway is what you get when you keep several of those swaps live behind one endpoint at the same time, with a policy deciding which one answers each call, instead of you changing a base URL by hand.

Every number here is read from the vendor's own docs, not a summary of a summary.

The market around this idea is bigger than the marketing suggests. OpenRouter and Vercel's AI Gateway both publish 0% markup on the underlying model price; LiteLLM is a free, self-hosted proxy that normalizes over 100 providers behind one endpoint. Source None of that is quality routing yet. It is plumbing, and the plumbing is most of what a gateway actually is.

What does model routing actually route on: quality, cost, or uptime?#

Routing sounds like one feature, but it is really 3 separate jobs: keep you up when a provider fails, send each request to the cheapest model that can handle it, or pick a model by measured output quality. Gateways default to the first two. The third stays rare, because it needs evaluation data most teams do not have.

Take LiteLLM's own router as the clearest evidence. Its documented strategies are simple-shuffle (the default, weighted by requests per minute), latency-based, usage-based, least-busy, and cost-based, plus a slot for a custom strategy you write yourself. Source Every one of those is a rule over an operational metric: how fast a deployment answered, how busy it is, what it costs. None of them read the answer and judge whether it was actually good.

Most of what ships as smart routing is the left column; the right column is a narrower, separate bet.

Braintrust names this gap directly, in a framework it calls the 3 jobs of model routing: resilience (stay up during an outage), cost (match a request to the cheapest model that clears the bar), and quality (pick a model using evaluation scores from your own real traffic). By that framing, OpenRouter, Vercel's AI Gateway, and Portkey cover resilience and cost through fallbacks and rules, without quality scoring wired into the routing decision itself. Source Worth saying plainly: that is Braintrust's own framing, and Braintrust sells the quality-routing piece, so weigh the comparison with that in mind.

A narrower set of products chase quality routing specifically. Not Diamond trains a router on evaluation data to predict which model answers a given prompt best, and says that cuts inference cost 20 to 40% for coding agents. Martian claims a similar approach, cutting cost 20 to 97% while often matching or beating GPT-4 on its own benchmarks. Both figures are vendor numbers from vendor pages, not independently verified, and both are a distinct product built for this one job, not a feature bolted onto a general gateway.

How does automatic fallback keep you up when a provider goes down?#

Fallback means the gateway detects a failed or slow call to your primary model, then automatically retries the identical request against a different provider without your code changing. You get a response instead of an error, usually within the same request, and the switch happens because of a rule you set once instead of a decision made under pressure.

This is not hypothetical. Anthropic's own status page logged 4 model-affecting incidents in 3 days at the end of July 2026 alone: elevated errors across all Claude models on July 29, degraded performance on Opus 4.8 plus a second elevated-error incident on July 30, and degraded performance on Sonnet 5 on July 31. Source Each one resolved within hours, but a request that landed during the gap either failed outright or waited, unless something else was already lined up to answer it. None of this is specific to Anthropic; every provider runs a public status page for the same reason. Claude's is simply the one with a dated, public log handy for this example.

The mechanism is standard across the market, even if the trigger differs. OpenRouter fails over to the next provider automatically on an error, and its :floor and :nitro routing variants let you bias that choice toward cost or speed. Source Vercel's AI Gateway bundles automatic retries, load balancing, and managed fallback into every plan at no extra charge. Source Cloudflare's AI Gateway does the same on its own edge network, alongside caching and rate limiting.

If you already build in n8n, this will look familiar: our guide to multi-model fallback in n8n wires the same ladder, primary model, cheaper backup, local model, human, using node-level error outputs. The difference is scope. That ladder protects one workflow. A gateway protects every app and workflow that calls it, from one place, without you wiring the same error branch twice.

How does cost-based routing send easy queries to a cheap model?#

Cost-based routing applies a rule before the call: classify the request, then send it to the cheapest model expected to handle it, reserving the expensive model for what actually needs it. It is the same idea as tiering your own code by hand, model choice as an if-statement, just enforced centrally instead of copy-pasted into every workflow you own.

Our cost hub, 7 levers that cut an AI API bill, names this exact mechanism as lever 4 and left it without a deep-dive; this is that deep-dive. The mechanism is not exotic: send the boring, routine share of traffic (a classification, a short summary, a form reply) to a budget model, and reserve flagship rates for the request that actually needs the reasoning.

The gateway enforces the same if-statement you would otherwise copy into every workflow.

The spread that makes this worth doing is not small. Claude's Haiku 4.5 lists at $1 in and $5 out per million tokens against Opus 4.8's $5 and $25, a 5x gap on both sides. Source Route even half your volume down a tier and the bill moves by multiples, not by a few percentage points, as long as the cheap model actually clears the bar on that half. That last condition is the whole game: a wrong answer that gets redone on the expensive model anyway is not a saving.

Which gateway fits: OpenRouter, Vercel AI Gateway, LiteLLM, Portkey, or Cloudflare?#

5 products currently anchor this market, and they split on one axis more than any other: managed and hosted (OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway) versus self-hosted and config-driven (LiteLLM, and Portkey's now open-sourced core). Pick the managed side for speed to first request, the self-hosted side for keeping routing logic and keys inside your own infrastructure.

None of these 5 do real quality routing out of the box, which is the honest thread running through this whole piece. If you specifically need model choice driven by evaluation scores rather than cost or uptime rules, that is a narrower category, Not Diamond, Martian, or Braintrust's own gateway, layered on top of one of the 5 above, not a replacement for it.

What does a gateway cost you in latency and vendor risk?#

A gateway adds a real network hop, plus a new dependency you did not have before. The honest tradeoff: you trade a little latency and one more vendor relationship for centralized fallback, cost control, and a single integration surface. For most builders that trade is worth it, but it is a trade, not a free upgrade.

Every pro here is a real mechanism; every con is a real cost. Neither cancels the other out.

The size of that hop depends heavily on how the gateway is built. Compiled proxies add close to nothing: one benchmark measured roughly 11 microseconds of overhead for a Go-based gateway under load. Python-based proxies like LiteLLM add more, commonly single-digit milliseconds and reportedly higher under heavier logging load. Set against typical LLM inference of 500ms to several seconds, that hop is usually a small fraction of the total wait, not the bottleneck.

The dependency risk is easy to underweight, because the failure mode is quiet. A self-hosted gateway with no redundancy of its own becomes a new single point of failure sitting in front of every provider you use, and a managed gateway's own outage takes down every app wired through it at once, providers included. Bringing your own key does not remove this: Vercel's AI Gateway explicitly falls back to its own system credentials when your key fails, which keeps you running but means your request still passed through Vercel's infrastructure either way. Source

Do you actually need a routing gateway, or is one provider's SDK enough?#

Not always. If you call one model from one provider and can live with an occasional outage, a gateway adds cost and complexity for a problem you do not have yet. The case for one gets real once you are multi-provider, an outage would actually hurt, or you want cost routing across more than a single project.

The decision comes down to 3 questions asked in order, and most builders can answer all of them honestly in under a minute.

Three questions, in order: multi-provider today, outage risk, and cost routing across more than one app.

Answer no early and you are done: stay on the SDK, or hand-wire the one fallback path you actually need. Answer yes all the way down and a gateway is doing real work for you, not just adding a hop for its own sake.

What should you set up this week?#

Start small: put your current provider behind an OpenAI-compatible client if it is not already, then add one fallback provider through a managed gateway like OpenRouter or Vercel AI Gateway, and force a failure to prove it actually switches. Cost-based routing and a second gateway opinion can wait until that first fallback is proven.

None of this requires believing the marketing about smart routing. Prove the fallback fires, know what you are actually being charged, and add quality routing only once you have evaluation data good enough to trust it, the same discipline that makes any of the 3 jobs, resilience, cost, or quality, actually work.