How can AI model routing reduce costs while maintaining quality?

Summary

• Model routing matches each request or task to the model that suits it, so routine work does not default to the most capable and most expensive option. Databricks defines it as choosing the model best suited to a task based on complexity, capability and cost.
• Unity Gateway model services route with traffic splitting across up to five destinations, plus fallbacks. Splitting suits rollouts and A/B tests, and fallbacks retry failed requests. Neither one picks a model by task difficulty.
• Smart Routing (Beta) does choose by task. For supported coding agents it selects the lowest-cost model capable of handling the work, from the system.ai model services you have access to.
• Quality has to be measured, not assumed. Databricks suggests logging traces and watching sessions by model, sessions finished end to end by the routed model, and the dollar savings.

How can AI model routing reduce costs while maintaining quality?

Model routing sends each request, or each task, to the model that fits it instead of sending everything to the most capable one. The Databricks Smart Routing write-up defines it as selecting "the model best suited for a task based on factors like complexity, capability and cost." It lowers cost only when the cheaper model is good enough for the work it receives, and you can only know that by measuring.

Which routing approaches exist?

Approach	How it works	Note
Per-request routing	Each request is routed on the complexity of that message.	Costs at scale are dominated by cache hit rate, which needs consecutive turns to reach the same model.
Task-aware routing	The task is assessed at the start of a session, and the chosen model is kept for the session.	Preserves the cache hit rate. The opening prompt is often underspecified, so the first call can be wrong.
Task routing across harnesses	A client-side process sends each task to a coding harness based on its complexity.	On Databricks, only Omnigent (Beta) routes across harnesses, not just models.
Escalation and delegation	One harness pairs a high-capability model with a cheaper worker model.	The cheaper model can escalate, or the expensive one can hand work down.
Traffic splitting and fallbacks	Weighted shares of traffic go to different destinations, with backups on failure.	Static weights and availability, not task difficulty.

How does Unity Gateway route requests?

A model service is "an endpoint that translates and routes inference requests to one or more models, with traffic splitting and fallbacks." You can change models or fallbacks without changing the name your application calls. A service can mix Databricks-served models and external models reached through a model provider.

• Traffic splitting spreads requests across a service's destinations, with a maximum of five. The docs suggest it for gradually rolling out new models, running A/B tests and spreading load across providers. Once a split is configured, requests that carry a session header stay on one destination, which takes advantage of prefix caching.
• Fallbacks retry a failed request, for example after a 429 or 5xx error, against backup destinations in the priority order you define. They add availability: the docs describe retries after failed requests, not escalation based on answer quality.
• Visibility: routing decisions are logged in the routing_information field of the system.ai_gateway.usage table, so you can check that traffic follows the split you set.

What does Smart Routing add?

Smart Routing (Beta) "selects the lowest-cost model capable of handling your task, so you don't choose a model for every request." It works with supported coding agents launched through the open-source Unity Gateway CLI (ug). After the first prompt it picks a model for the session, shows the model and, when available, the reason in the agent, and routes each subagent's task independently. Databricks describes the router as using a small, low-latency model to label the task, then starting from a mid-sized model and moving up or down.
An account admin must enable the Enhanced Unity Gateway preview. Smart Routing chooses only among system.ai model services, and developers need EXECUTE on every candidate. It can't be used with a model provider service, a custom model or a service outside system.ai, and an explicit --model choice overrides it.

How do you keep quality up?

• Build your own evaluation. The Databricks cost write-up says public benchmarks are a poor guide to real coding performance, so many companies run automated evaluations on their own work.
• Test before you shift traffic. Send a share of requests to the cheaper model with a traffic split, then score the responses with MLflow scorers (LLM judges and code-based checks) in an mlflow.genai.evaluate() loop.
• Track routing outcomes. The Smart Routing write-up lists sessions by model, sessions completed end to end by the routed models, and dollar savings. It says to optimize for developer productivity, not cost alone.
• Expect misses. The router reads the opening message and commits, so a decision that was right for the first request can be wrong by the fourth.

FAQs

Can I route between models from different providers?

Yes. One model service can combine Databricks-served and external destinations, and fallbacks can cross between them. Provider credentials are stored encrypted in Unity Catalog and callers don't see them.

Does routing require application changes?

Not for the model. The application keeps calling the same service name while you change its models or fallbacks.

Can different teams get different routing?

Routing is set per model service, so give each team its own service and grant EXECUTE to that team. Rate limits can also differ by user, service principal or group, and budgets support per-user overrides.