Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is LLM routing? Model routers and cascades, explained

LLM routing sends each query to the cheapest model likely to answer it well; cascades try a small model first and escalate. How both work, with a Python sim.

What is LLM routing? Model routers and cascades, explained

LLM routing is the practice of sending each request to a different model depending on what the request needs, usually so that easy queries go to a small, cheap model and only hard ones reach an expensive frontier model. A router decides before any model is called, based on the query; a cascade calls the cheap model first and escalates to a stronger one when the answer looks unreliable. Both trade a little quality risk for large cost and latency savings, and both depend on how well you can predict which queries are hard.

Why route between LLMs at all?

Because model prices span orders of magnitude and most traffic is easy. The FrugalGPT paper (Chen, Zaharia and Zou, 9 May 2023) opened with the observation that LLM API fees varied by up to two orders of magnitude, which is still true. On Anthropic’s current price list, Claude Haiku 5.5 costs $0.10 per million input tokens for prompts up to 100,000 tokens and Claude Opus 5.5 costs $4, a factor of 40, with output at $0.50 against $20.

If a small model can answer most of your queries correctly, paying frontier prices for all of them is waste. If it can’t answer the hard ones, using it for everything is a quality bug. Routing is the attempt to get both right.

How does an LLM router work?

A router is a classifier that runs before generation. It looks at the query (and sometimes the conversation, user tier or task type) and picks a model. Implementations range from simple to learned:

  • Rules. Route by task type, input length, presence of code or images, or customer plan. Cheap, transparent and often surprisingly effective.
  • Learned routers. Train a small model to predict whether the weak model’s answer will be good enough. RouteLLM (Ong et al., 26 June 2024) trains routers on human preference data to choose between a strong and a weak model, and reports cutting costs “by over 2 times in certain cases” without reducing response quality. The routers also kept working when the strong and weak models were swapped for others at test time.
  • Difficulty prediction with a quality knob. Hybrid LLM (Ding et al., 22 April 2024) routes between a small and a large model based on predicted query difficulty and a target quality level that can be tuned at test time. It reports “up to 40% fewer calls to the large model, with no drop in response quality”.

The appeal of a router is that each query pays for only one model call. The weakness is that it has to judge difficulty from the question alone, before seeing any answer.

How does an LLM cascade work?

A cascade tries models in order from cheapest to most expensive and stops at the first answer that passes a check. FrugalGPT lists the LLM cascade as one of three cost strategies alongside prompt adaptation and LLM approximation, and its implementation “learns which combinations of LLMs to use for different queries”. The paper reports matching GPT-4’s performance with up to 98 percent cost reduction, or improving accuracy over GPT-4 by 4 percent at the same cost, on its benchmarks.

The check is the whole game. Options include a scoring model trained to predict correctness, the model’s own confidence or log-probabilities, a self-consistency vote, or a hard verifier: the JSON parses, the unit test passes, the SQL compiles and returns rows. Hard verifiers are the best when your task has them, which is one reason cascades work so well for code and structured output.

The cost of a cascade is that escalated queries pay twice: the cheap attempt and the expensive one. They also wait twice, which shows up in p95 latency.

What does a cascade look like in code?

This simulation runs 10,000 queries with hidden difficulty through a two-model cascade at today’s Anthropic list prices. The small model solves the easiest 65 percent, the large one solves 95 percent, and the small model’s confidence tracks difficulty with noise. Everything about the traffic is simulated; the prices are real.

# cascade.py: a two-model cascade on simulated traffic. Python 3, stdlib only.
# Prices: Anthropic list prices, USD per 1M tokens, read 2026-10-31.
# The traffic, the models' skill and the confidence signal are simulated.
import random

random.seed(7)
SMALL = ("claude-haiku-5-5", 0.10, 0.50)
LARGE = ("claude-opus-5-5", 4.00, 20.00)
IN_TOK, OUT_TOK = 1_500, 400

def call_cost(model):
    _, p_in, p_out = model
    return IN_TOK * p_in / 1e6 + OUT_TOK * p_out / 1e6

queries = [random.random() for _ in range(10_000)]          # hidden difficulty, 0 easy to 1 hard
small_ok = [d < 0.65 for d in queries]                       # small model solves the easy 65%
large_ok = [d < 0.95 for d in queries]                       # large model solves 95%
# The small model's confidence tracks difficulty, with noise. Real systems use a
# verifier, log-probs or a trained router here, and none of them is perfect.
conf = [min(1, max(0, 1 - d + random.gauss(0, 0.15))) for d in queries]

def run(threshold):
    cost = correct = escalated = 0
    for i in range(len(queries)):
        cost += call_cost(SMALL)
        if conf[i] >= threshold:
            correct += small_ok[i]
        else:
            escalated += 1
            cost += call_cost(LARGE)
            correct += large_ok[i]
    return cost, correct / len(queries), escalated / len(queries)

n = len(queries)
print(f"{'policy':<22} {'accuracy':>8} {'escalated':>9} {'cost per 10k':>12}")
print(f"{'small only':<22} {sum(small_ok) / n:>8.1%} {0:>9.0%} {call_cost(SMALL) * n:>12.2f}")
print(f"{'large only':<22} {sum(large_ok) / n:>8.1%} {1:>9.0%} {call_cost(LARGE) * n:>12.2f}")
for t in (0.3, 0.5, 0.7):
    cost, acc, esc = run(t)
    print(f"{f'cascade, conf < {t}':<22} {acc:>8.1%} {esc:>9.0%} {cost:>12.2f}")

Output (Python 3.13.11):

policy                 accuracy escalated cost per 10k
small only                65.0%        0%         3.50
large only                95.0%      100%       140.00
cascade, conf < 0.3       86.4%       30%        45.42
cascade, conf < 0.5       93.6%       50%        73.39
cascade, conf < 0.7       94.9%       69%       100.74

The threshold is the quality knob. Escalating when confidence is below 0.5 keeps 93.6 percent accuracy against 95.0 for the large model alone at about half the cost ($73.39 against $140.00 per 10,000 queries). Pushing the threshold to 0.7 gets within 0.1 points of the large model but saves only 28 percent, because 69 percent of traffic now pays for both calls.

Two lessons transfer to real systems. First, the curve flattens: the last point of accuracy is the most expensive one. Second, everything depends on the confidence signal. With a perfect signal you would escalate exactly the 35 percent the small model gets wrong; with noise, you escalate more and still let some wrong answers through. Improving the signal moves the whole curve, which is why RouteLLM and Hybrid LLM spend their effort there.

Where does LLM routing break?

The router becomes a silent quality regression. If it sends a hard query to the small model, the user gets a fluent wrong answer and nobody sees an error. Sample routed traffic and grade it, per route, continuously.

Distribution drift. A router trained on last quarter’s queries can misroute this quarter’s. Retrain or recalibrate on fresh logs.

Latency on escalation. Cascades double the wait for hard queries. If p95 latency matters more than cost, a pre-call router is safer than a cascade.

Prompt and format differences. Two models rarely respond identically to the same prompt. Keep per-model prompts and test each route’s outputs against the same eval set.

A simpler option was available. Anthropic’s guide on optimizing for cost and intelligence advises tuning one model before adding a second: sweep effort on your current model and only keep a pairing that “beats the single model’s whole curve of score against spend.” In its own measurements, an advisor pairing on GPQA Diamond added cost while staying within run-to-run noise of the executor alone, because the executor never consulted the advisor on any of the 198 questions.

What I would do

From a data-platform seat, the line I would circle is observability. A router is a traffic-splitting system, and traffic splits without per-route metrics are how silent regressions ship. I’d log the route, model, tokens, latency and a sampled quality grade for every request before adding any routing logic.

Then I’d start with rules, not a learned router: route by task type where tasks are clearly different (classification and extraction to the small model, multi-step reasoning to the large one). For tasks with a hard verifier, such as JSON validity, a passing test or SQL that compiles, I’d use a cascade, because the check is reliable and cheap. Only when traffic is large and mixed would I train a router, and I’d evaluate it against the simplest baseline: one model at a lower effort setting.

Routing also fits inside agents. A small model can grade and rewrite queries in an agentic RAG loop while a larger one writes the final answer, and the same cost-per-solved-task math from how to choose an LLM applies to each step. If you also want to save the cost of repeated prompts, prompt caching stacks with routing.

FAQ

What is the difference between LLM routing and a cascade?

A router picks one model before generation, based on the query. A cascade calls a cheap model first, checks the answer, and escalates to a stronger model if the check fails, so hard queries pay for two calls.

How much can LLM routing save?

It depends on your traffic mix and your confidence signal. Published results include RouteLLM cutting costs by over 2 times in some cases without quality loss, Hybrid LLM making up to 40 percent fewer calls to the large model, and FrugalGPT reporting up to 98 percent cost reduction on its benchmarks.

What is RouteLLM?

A 2024 paper by Ong et al. that trains router models on human preference data to choose between a strong and a weak LLM per query, trading cost against response quality.

Is model routing the same as mixture of experts?

No. Mixture of experts routes tokens between expert sub-networks inside a single model. LLM routing chooses between separate models, usually from outside, at the request level.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.