Auto model routing picks the model for each request instead of making a developer choose one model for everything. This walkthrough covers how routing works across Lyzr repositories, how the wider industry approaches it, what the available test data shows, and what could improve next.
The Gist
- The problem: Most traffic is easy. Sending every request to a frontier model overpays, while sending everything to a small model fails on harder cases.
- What Lyzr ships: A four-tier router — SIMPLE, MEDIUM, COMPLEX, and REASONING. A classifier scores the request, and a tier map converts the tier into a provider, model, and credential.
- A design rule worth copying: Routing is advisory. If routing fails, the request falls back to the configured model instead of failing.
- What the internal test data shows: A learned router matched premium-model accuracy (94.4% vs. 94.9%) at 77.5% lower cost. Hand-written rules scored 87.7% vs. 95.4%.
- What to improve next: Evaluate in shadow mode before enabling routing, escalate when a response fails checks, make routing session-aware, and bring the different routing engines behind one tier contract.
The Problem Auto Model Routing Solves
Model prices can differ by one to two orders of magnitude, and latency varies too. In the benchmark run in rekori, a small model cost about $0.00012 per MMLU request, while the premium model cost about $0.0042, roughly 35 times more. Yet the two scored within a few points of each other on GSM8K math word problems.
That gap creates an opportunity. Three pressures push teams toward routing.
1. Cost
A fixed premium model pays frontier prices for greetings, lookups, and reformatting tasks that may not need that level of capability.
2. Latency
Smaller models generally answer faster. Deep-reasoning models can take long enough to time out: in the internal test, the Sonnet-high configuration timed out on 17 of 300 MMLU requests.
3. Choice Fatigue and Vendor Risk
Non-expert users should not have to choose between dozens of models, and no single vendor wins on every task. In the internal test, OpenAI’s premium model led on MMLU, 97.7% to 75.0%, while the tested models converged on GSM8K.
For a broader introduction to the approach, see Lyzr’s guide to Auto Model Routing in Lyzr and its article on small model routing.
How Auto Model Routing Works Across Lyzr Repositories
The code review covered the NeuralgoLyzr organization. The similarly spelled neurogolyzr organization did not resolve on GitHub. Routing appears at four layers, and each layer makes a different decision.
lyzr-agent: Routing Inside the Agent Runtime
Location: lyzr_agent/llm/auto_router.py
This is per-agent and per-organization complexity routing inside the agent runtime. It is the main implementation and the one configured through the Studio UI.
1.rekori: Gateway-Level Routing
rekori uses a Bifrost-based gateway. Its routing options include a lexical Complexity Router, embedding-based Semantic Routes, and a TypeSafe Jev classifier. These feed a CEL rules engine, and the repository also contains a benchmark for mixed-provider routing.
2. cowork-monorepo: An Auto Pseudo-Model
Location: services/backend/internal/catalog
A department can force chats onto Auto. The catalog then returns only the models that the routing layer is allowed to read.
4. lyzr-ai-sa: Intent Routing
Location: openspec/add-request-routing
This path classifies a solutions-architect message into intents such as lookup or full_solutioning. It uses a rule-based baseline and a promotion gate. This is intent routing, not model routing.
The Request Path in lyzr-agent
The runtime follows this decision sequence:
- Check whether auto routing is enabled. If enable_auto_routing is false, use the configured model.
- Resolve the applicable configuration. An explicit agent request flag takes precedence, followed by an enabled agent configuration, then the organization’s default tier map.
- Validate the routing setup. If the configuration is invalid, the required message or assets are missing, or the request is otherwise ineligible, use the fallback path.
- Classify the request. Depending on configuration, the router uses a heuristic classifier, an LLM classifier, or Jev.
- Resolve the tier. The tier map returns a provider, model, and credential.
- Make the model call. The normal LyzrLLM call path handles credential swapping, billing, and organization-level fallbacks.
- Record the decision. The trace includes the llm_auto_routed_tier span attribute so the root span reflects the tier that actually answered, including fallback behavior.
When routing fails or times out, the configured model remains the fallback. The design is intended to keep a routing problem from becoming a failed user request.
Design Choices in the Code Worth Knowing
| Design choice | What the code does |
| Classifier only | The LiteLLM ComplexityRouter classifies the request. The selected target then goes through LyzrLLM’s existing call path, preserving billing, streaming adapters, and fallbacks. |
| Fail open | Invalid configuration, a missing message, a classifier error, or a missing router returns the configured primary target. Routing failure does not fail the request. |
| Three classifiers | heuristic is the default in-process option; llm uses a small model to score the request; jev uses a typed classifier. Jev has a default confidence threshold of 0.55, below which the request goes to REASONING. |
| Configuration precedence | The agent request flag takes precedence, followed by an enabled agent configuration and then the organization default. A saved but disabled agent configuration does not block the organization default. |
| Hard exclusions | Messages carrying assets skip routing. vertex_ai cannot be a tier target because the credential-swap path has no Vertex branch. |
| Cheap hot path | One router instance per distinct configuration is held in a 128-entry LRU cache and shared across requests without per-request state. |
| Observability | The served target and tier are written to the trace using llm_auto_routed_tier, including fallbacks. |
| Safer configuration UX | The Studio UI pre-fills tiers from platform defaults, warns about an inverted cost ramp, flags four identical tiers, and blocks Save for unsupported providers. |
How the Heuristic Classifier Scores a Request
The gateway’s analyzer is the best-documented implementation. It scores the last user message from 0 to 1 using weighted signals.
| Signal | Weight or behavior |
| Code presence | 30% |
| Reasoning markers | 25% |
| Technical terms | 25% |
| Token count | 10% |
| Greetings | Small dampening effect |
| Default boundaries | 0.15, 0.35, and 0.60 |
| Reasoning keywords | Two or more can force the REASONING tier |
The heuristic runs in under a millisecond and does not make a network call. It is fast and inexpensive, but keyword signals can mistake a difficult question for a simple one.
The Jev Classifier Makes a Different Trade-Off
Jev makes a network call, taking about 180–300 ms and costing about $0.00008 per request in the described setup. It can assess information that a lexical scorer may miss, such as whether a request contains regulated data.
The gateway keeps Jev disabled by default and includes several safeguards:
- Per-key opt-in for egress.
- A hard timeout.
- A circuit breaker that pauses after five failures for 30 seconds.
- “No signal on failure” behavior, so a failed classification does not become a hard request failure.
Jev offers a richer signal, but the additional network call adds latency and cost. The right choice depends on the traffic and the context the router needs to understand.
Pros and Cons of Auto Model Routing
Pros
- Lower cost on mixed traffic: The internal same-item test showed 77.5% savings for learned routing.
- Lower median latency: Most routine requests can go to smaller models.
- Vendor resilience: Mixed-provider routing can avoid a vendor’s weak spots or outages.
- Simpler configuration for builders: Administrators configure four tiers rather than requiring every agent builder to select a model for each task.
- A policy hook: The routing layer can also support data-sensitivity, region, and compliance rules.
- Gradual adoption: Routing is opt-in at the agent or organization level, with existing behavior retained when it is off.
Cons
- Silent misroutes: A hard request sent to a weak model may produce a plausible-looking answer, making quality loss difficult to notice.
- Crude heuristics: In the reported end-to-end run, rules scored 87.7% compared with 95.4% for the premium baseline. Exam-style questions can look lexically simple despite being difficult.
- Less deterministic behavior: Similar prompts can be sent to different models, complicating debugging and evaluation.
- Another dependency to operate: A classifier needs monitoring and tuning. LLM- and Jev-based classifiers add latency and cost to requests.
- Cache and consistency loss: Switching models mid-conversation can discard provider prompt caches and change tone or tool behavior.
- Data egress: An external classifier may send prompts to a third party, which needs to be considered in the deployment and data-handling design.
For related infrastructure context, see What Is an LLM Gateway? and Lyzr’s guide to framework-agnostic AI platforms.
How the Industry Approaches Model Routing
Production routing systems tend to share the same structure: a relatively cheap decision step in front of a pool of models, with a dial that balances cost and quality. The main difference is how the decision is made.
| System | Approach | Published claim in the source |
| RouteLLM (LMSYS / Berkeley, open source) | Trains on Chatbot Arena preference data. Routes between stronger and weaker models using similarity-weighted ranking, matrix factorization, a BERT classifier, or a causal-LLM classifier. | Over 85% cost reduction on MT Bench, 45% on MMLU, and 35% on GSM8K at 95% of GPT-4 performance. |
| Azure AI Foundry Model Router | Uses a trained ML model to predict the best model for each prompt. Offers Cost, Balanced, and Quality modes and allows the candidate set to be restricted. | Up to 60% savings against a high-end model at similar accuracy. |
| Amazon Bedrock Intelligent Prompt Routing | Predicts model performance per request and chooses the cheapest model likely to answer well. The router is configured with two models from one family. | Up to 30% cost reduction without accuracy loss. |
| OpenRouter Auto Router | openrouter/auto selects from a curated set of models and reports which model it chose. The source says it was originally powered by NotDiamond, with a usage-ranking router in beta. | No percentage stated in the source. |
| Gateways such as Bifrost and LiteLLM | Operator-controlled rules, complexity routing, or semantic routing. The source describes rekori and lyzr-agent as building on this pattern. | Depends on the operator’s rules. |
These are vendor- or paper-reported results on their own benchmarks, so treat them as upper bounds rather than guarantees. The more useful numbers are the ones measured on your own traffic.
Four Routing Patterns
1. Rules and Heuristics
Use keywords, prompt length, and metadata to classify a request. This approach is fast and explainable, but generally weaker on accuracy. It is the default approach described in the Lyzr implementation.
2. Semantic Matching
Embed the prompt and match it to labeled examples. This can capture topic and intent better than simple keyword rules, but performance depends on whether the embedding model supports the language and domain.
3. Learned Routers
Train a model on graded outcomes so it can predict which candidate model is likely to perform best. This can improve routing quality, but requires labeled data and retraining as traffic changes.
4. Cascades
Try a cheaper model first, then escalate if a check fails. A cascade avoids relying entirely on an upfront prediction, but escalation adds latency and means some requests pay for more than one model call. See Lyzr’s related guide to model cascading and distillation.
What Our Own Test Data Shows
The rekori test campaign, dated August 7, 2026, ran six model configurations on MMLU and GSM8K. The team then fitted a cluster-then-rank learned router on 810 graded outcomes and checked it live on 450 fresh questions.
Results on the Same 450 Items
| Configuration | Accuracy | Cost per request |
| Learned mixed router, α = 0.5 | 94.4% | $0.00202 |
| Premium always (gpt-5.5, high effort) | 94.9% | $0.00894 |
Only two of the 450 questions separated the two configurations, which the source describes as statistical noise. The reported cost reduction is 77.5%.
The source also specifies what the evaluation did not cover. Creative writing, code generation, and agent workflows do not have an answer key in this test, so the results do not establish savings for those workloads. All reported results are in English.
What This Means
The result is promising for the evaluated benchmark tasks, but it should not be generalized automatically to every production workload. Teams should test on representative requests and measure quality alongside cost before enabling a router broadly.
What I Would Change
These are recommendations based on the code and test reports described in the source, not announced or planned work.
1. Run Every Router in Shadow Mode Before It Acts
The AISA design already calls for this pattern: classify and record the decision without acting on it, then enable routing only after it beats the baseline on a labeled set. The source says lyzr-agent has no equivalent today.
Add a shadow flag that logs the tier the router would choose. Periodically replay a sample against the premium model to estimate the quality loss before routing changes live traffic.
2. Replace the Keyword Scorer With the Learned Router Where Traffic Allows
The reported results show learned routing beating hand-written rules by about seven percentage points. The cluster-fitting tooling already exists in rekori. Keep the heuristic as a fallback, which fits the existing fail-open design.
Add Escalation, Not Just Prediction
The current decision is one-shot. Escalate one tier when the response fails an inexpensive check, such as:
- A refusal where a useful answer is expected.
- Malformed structured output.
- A failed tool call.
- Low self-reported confidence, where that signal is meaningful.
This makes a misroute less likely to become a silent quality regression, while bounding the additional cost.
3. Make Routing Session-Aware
The agent path classifies only the latest user message, while the gateway blends up to ten turns. Short follow-ups such as “do it” or “continue” are easy to misroute without conversation context.
Pin a model for the session unless the tier needs to move up. This can also help keep prompt caches warm and behavior more consistent.
4. Filter by Capability Before Scoring Complexity
Hard constraints should exclude unsuitable targets before complexity scoring. Relevant capabilities include context length, tool-calling support, structured output, and vision. The current implementation skips routing for messages with assets; routing those requests to a vision-capable tier could keep them in scope.
5. Use Health and Latency as Routing Signals
The Sonnet-high timeouts on MMLU show that provider behavior can vary by task. Feed recent timeout and error rates into tier selection so a degraded target can be skipped before the fallback chain has to catch it.
6. Unify the Engines Behind One Tier Contract
The agent runtime, gateway, and Jev paths classify requests separately, and the source says they already disagree on low confidence: the agent jumps to REASONING, while the gateway moves up one tier.
Choose one behavior, share one tier schema, and record one cost attribution model.
7. Tighten Classifier Time Budgets
The agent’s Jev and LLM classifiers default to a 3,000 ms timeout, while the gateway budgets 300 ms. A pre-call step that can add three seconds before the first token needs a budget closer to the gateway’s, with the heuristic available as an immediate fallback.
Report Savings and Quality Together
The trace already carries llm_auto_routed_tier. Add:
- Tier distribution.
- Escalation rate.
- Spend compared with an always-premium counterfactual.
- Evaluation score by tier.
A savings number without a quality number is how silent misroutes stay hidden.
Final Takeaway: Route by Need, Then Measure the Trade-Off
Auto model routing can reduce cost and latency by matching each request to an appropriately capable model.
The internal results in this source show a learned router approaching premium-model accuracy at substantially lower cost on the evaluated questions. They do not establish the same savings for every workload.
The implementation’s fail-open behavior is a useful safety property, but it does not eliminate the risk of a request being sent to a model that is too weak. Before broad rollout, evaluate routing in shadow mode, measure quality and cost together, add escalation for detectable failures, and account for session context and provider health.
For more on putting routing into practice, read Auto Model Routing in Lyzr, Small Model Routing, and Model Cascading and Distillation for Production AI.
For the infrastructure layer around multi-provider calls, see Lyzr’s LLM gateway architecture guide.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
