Last month’s LLM bill lands, and most of the line items turn out to be for things like “what’s the return policy” and “reformat this into a table,” questions a fraction of the cost would have answered just as well. The waste isn’t a bad call anyone made; it’s what happens by default when every request, easy or hard, gets sent through the same frontier model because nobody’s had time to sort them by hand. Sets up the waste without naming the term yet.
Key Takeaways
- Small model routing sends each request to the cheapest model capable of answering it well, instead of routing everything through the same frontier model by default.
- Most production traffic turns out to be routine (classification, extraction, formatting, short lookups), so a small model handles a lot of it without any real drop in quality.
- The routing decision itself runs on a fast, lightweight classifier that looks at the prompt before any model call happens, adding only a small amount of latency.
- Teams typically build the routing logic one of three ways: hardcoded rules, semantic matching, or a trained classifier, each with a different tradeoff between simplicity and accuracy.
- The real risk here has nothing to do with cost: it’s a hard prompt getting misjudged as easy and quietly landing on a model that can’t handle it, a design problem with known fixes rather than a reason to skip routing.
What is small model routing?
Small model routing is a technique that evaluates each incoming prompt and automatically sends it to either a smaller, cheaper language model or a larger, more capable one, based on how complex the task actually is rather than sending every request to the same model by default.
The idea matters because real usage tends to be far more lopsided than teams assume. As teams put LLMs behind more of their product, the traffic they handle skews more toward easy, repetitive tasks over time, not less, which is exactly what makes routing everything through one model increasingly wasteful. A support widget answering “what’s your return policy” a thousand times a day doesn’t need the same model that reasons through a multi-step refund dispute once, but running both through the same frontier model treats them as equally expensive problems when only one of them is.
That default is understandable given how most teams start. A single API call is simple to build and simple to monitor, and when volume is still small, the excess cost barely shows up as a line item worth chasing. The problem is that decision doesn’t get revisited as volume grows. What made sense at a few hundred requests a day compounds into real money at a few hundred thousand, and by then the traffic mix has usually shifted even further toward the routine end of the spectrum than it was at launch.
How small model routing actually works
A lightweight triage layer sits in front of the model stack and looks at the prompt before any model call happens. It scores the prompt for complexity, using signals like length, task type, domain, or how much reasoning the phrasing implies. Based on that score, the request either gets executed on a small model (classification, extraction, formatting, short lookups, basic summaries) or escalated to a frontier model (multi-step reasoning, ambiguous instructions, code generation, anything where being wrong is expensive).
Azure AI Foundry’s Model Router already does a version of it in production, where a single deployment analyzes each prompt in real time for complexity and task type, then routes it across models from several vendors while holding output quality inside a set band.
Amazon Bedrock’s Intelligent Prompt Routing does something narrower but related, choosing between two models in the same family based on which one is predicted to answer a given prompt well enough. Because this evaluation is a lightweight classification step rather than a full model call, it adds only a small amount of latency relative to generation time itself, not enough to offset the savings from routing to a cheaper model.
Three ways teams implement small model routing
Most routers are built one of three ways, and what separates them is how much they trust a fixed rule versus something learned from data.
Rule-based routing
The simplest version hardcodes logic on prompt length, keywords, or user tier: anything under 50 tokens goes to the small model, anything containing “explain” or “write code” escalates. It’s cheap to build and easy to reason about, since anyone can read the rules and predict what they’ll do. It’s also brittle the moment real traffic stops matching the assumptions baked into those rules, which happens quite often.
Semantic routing
Semantic routing embeds the prompt into a vector and matches it against known task clusters or domain-specific models, catching nuance that a keyword rule would miss entirely. Two prompts that are the same length and contain none of the trigger words a rule-based system watches for can still land in very different complexity buckets, and semantic routing is built to notice that. It costs a bit more compute per decision than a fixed rule, since it has to run an embedding step first.
Classifier-based routing
The most accurate approach trains a small model, often a compact encoder model far smaller than the LLMs it’s routing between, to predict directly whether a cheaper model is likely to produce an acceptable answer. This scales better than hand-written rules and adapts as traffic shifts, but it needs labeled data and ongoing upkeep as that traffic changes, which is real work, not a one-time setup cost.
All three get the routing decision right most of the time. “Most of the time” is exactly where the quality question in this piece’s title comes from.
Where small model routing can quietly hurt quality, and how to prevent it
The failure mode here is a router misjudging a hard prompt as easy and never giving the capable model a chance to catch it, not the small model itself being incapable. A small model asked something within its range does that job fine. The actual risk sits one step earlier, in the decision about which model gets asked at all.
Three things prevent this in practice.
- Build an escalation path for low-confidence classifications instead of a hard binary choice, so a request can go to the small model and still get flagged for handoff if the router isn’t confident.
- Test the router and the small model against a representative sample of real production prompts before shipping, not synthetic benchmarks written to make the system look good.
- Monitor outcomes, not just which model handled a request, because a router can look like it’s working while quietly degrading the exact cases it never surfaces to anyone reviewing it.
A router with no escalation path isn’t cost savings. It’s a quality regression waiting to be noticed by a customer instead of a dashboard.
Rolling out small model routing without breaking anything
Start with the highest-volume and lowest-stakes traffic first, like FAQs, formatting, and short extraction tasks, rather than routing everything on day one. Keep the frontier model as the default for any task type the router hasn’t been validated against yet, as an unvalidated router making decisions on unfamiliar traffic is how a rollout turns into an incident. Track cost per successful outcome, not token price alone, since a cheap answer that has to be redone by the expensive model twice isn’t actually cheap. Rather than by the price per token, measuring this way actually tells you whether the rollout is working.
Keeping model routing decisions governed once they’re live: Lyzr OpenController
Choosing which model answers a given prompt is a routing question. Making sure that choice stays inside policy once there are multiple models, multiple providers, and multiple teams building on it is a different problem: a governance one. Lyzr OpenController keeps whatever routing logic a team builds or buys accountable to policy once it’s running.
Its four connected capabilities apply to that problem directly. Find automatically discovers the agents, models, tools, data, and workflows actually running across the estate, so a routing layer’s model choices don’t stay invisible to whoever owns policy. Ship evaluates, validates, and governs agents and workflows before they reach production, which covers a newly introduced model too, since it’s part of whatever workflow is calling it. Run monitors agents, applications, APIs, and infrastructure in real time from one control plane. Lastly, Improve turns real usage, performance, cost, and security signals into the next round of policy.
This means someone can answer, with a straight face, which models are actually deciding what your users see. If that’s a question your team can’t currently answer, booking a demo is a reasonable next step.
FAQ
Small model routing is a technique that automatically sends each prompt to either a smaller, cheaper language model or a larger, more capable one, based on the task’s actual complexity, instead of sending every request to the same model by default.
It depends on how routine your traffic mix is. Vendors report large reductions when most requests are simple, but the reliable answer is to measure your own traffic rather than assume a published figure applies to it.
Not when the router includes an escalation path. The real risk is a hard prompt getting classified as easy and never reaching a model capable of handling it, not the small model being incapable on its own turf.
Routing decides which model handles a request upfront, based on a complexity score. Cascading tries the cheap model first and only escalates to a more capable one if that attempt fails or scores low on confidence.
Generally not as reliably as frontier models, which is exactly why routing exists: it sends multi-step reasoning, ambiguous instructions, and high-stakes tasks to a more capable model rather than stretching a small one beyond what it’s good at.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
