Skip to main content

What is an AI router and why do enterprises need one?

Summary

  • An AI router is the layer between applications and models that decides which model, or which provider endpoint, handles each request.
  • Routers decide with rules (weighted splits, ordered fallbacks, session pinning) or with a classifier that judges the task and picks the lowest-cost model that can handle it.
  • Enterprises need one because they run many models from many providers, models change often, and no single application team should own provider credentials, failover logic and spend controls.
  • The router is also the natural place to enforce governance, because permissions, rate limits, budgets and logging all apply where traffic converges.
  • On Databricks, Unity Gateway model services route across Databricks-hosted and external models behind one name, and Smart Routing (Beta) picks models per task for supported coding agents.

What is an AI router and why do enterprises need one?

An AI router is a control layer that receives model requests and decides where each one goes: which model, which provider, and what happens if that destination fails. Enterprises need one because they no longer run a single model. They mix hosted and external models, change them as better ones ship, and pay very different prices per token for each. Without a router, every application hard-codes those choices, and changing a model means changing code across many teams.

What is an AI router?

The term covers two related jobs. Traffic routing distributes requests by fixed rules: split traffic by weight, retry against a backup on errors, keep a conversation on one model. Intelligent routing reads the task and chooses the model, usually so routine work doesn't run on the most expensive one. Most enterprise routers live inside an AI gateway, because routing and governance need the same position in the request path.

Common routing approaches

Approach How it decides Typical use
Weighted split At random per request, weighted by configured percentages Gradual rollout of a new model, A/B tests, spreading load
Ordered fallback Retries a failed request against backups in a set order Availability during provider errors or rate limiting
Session affinity Keeps requests with the same session header on one destination Consistent multi-turn behavior and prompt-cache reuse
Task-aware routing A classifier labels the task and picks a capable, lower-cost model Cost control for agent workloads
Policy-aware routing Routes only to models the caller is permitted and budgeted to use Governance across teams

Why enterprises need one

  • Model changes without code changes. A stable service name lets the platform team change the model behind it while applications keep calling the same name.
  • Availability. Provider errors and rate limits happen. An ordered fallback keeps requests flowing.
  • Cost. Routine tasks rarely need a frontier model. Routing by task, plus per-user budgets, keeps spend in line with value.
  • Credential custody. Provider keys sit in one governed place, not in each application's configuration.
  • One audit point. Every request passes through a place that can log it, rate-limit it and apply guardrails.

How Databricks approaches AI routing

On Databricks, routing is part of Unity Gateway, the generally available governance layer for enterprise AI built on Unity Catalog.
Model services. A model service is "an endpoint that translates and routes inference requests to one or more models". It can route to Databricks-served models or to external models through a model provider, and "You can mix these destinations in one service." Callers use its fully qualified name, catalog.schema.name, for real-time requests or for batch inference through ai_query, and you can "Change the underlying model without changing the name the application calls."
Splits, fallbacks and affinity. With traffic splitting and fallbacks, you give each destination a traffic percentage, and splitting works across "a maximum of 5 destinations". Fallbacks retry a failed request, for example after a 429 or 5xx error, against backups "in the exact order specified". Requests that carry a session header stay pinned to one destination.
Credentials. Model provider services store external provider credentials as Unity Catalog securables, giving administrators "a single, governed chokepoint for provider credentials".
Task-aware routing (Beta). Smart Routing "selects the lowest-cost model capable of handling your task" for supported coding agents launched with the ug CLI. It needs EXECUTE on every candidate model, and it keeps the main session on its default model while routing subagents independently.
Governance around the router. Rate limits apply per user, group or service, budgets can alert or block, and every request is recorded in system.ai_gateway.usage (usage tracking).

FAQs

Is an AI router the same as an AI gateway?

Routing is one function of a gateway. A gateway also handles access control, credentials, guardrails, rate limits and logging, which is why routers usually sit inside one.

Does routing affect answer quality?

It can. Before you split traffic across models, test each one on your own workload so every destination is fit for the task.

Can a router work across model providers?

Yes. On Databricks, one model service can mix Databricks-served models and external models connected through model provider services.

Is Smart Routing generally available?

No. Smart Routing is in Beta. Unity Gateway itself is generally available.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.