AI gateway with intelligent model routing

Summary

• An AI gateway with intelligent model routing sits between your applications and your models, directing each request to the right endpoint by cost, latency, or task, and failing over automatically when one is throttled or down.
• The capabilities that matter most are traffic routing and splitting, automatic fallback, per-user and per-endpoint rate limits, spend caps, usage tracking, request and response guardrails, and provider-agnostic support.
• Routing lets a small, fast model handle routine calls while a larger model serves the hard ones, and it keeps production apps available when a provider degrades, without changing application code.
• Databricks Unity Gateway is a governed control plane for model and agent traffic: it routes and load-balances across endpoints, enforces rate limits and spend caps, logs every request to Unity Catalog, and applies guardrails including PII detection.
• Because it runs through Unity Catalog, models, agents, and MCP services are governed as securable objects with the same permission model you use for tables and volumes.

AI gateway with intelligent model routing
An AI gateway with intelligent model routing sits between your applications and the models they call, then decides which endpoint should serve each request. Rather than pinning every app to one provider, you route on cost, latency, availability, or task type, fall back automatically when an endpoint is throttled or erroring, and enforce one set of governance rules across all traffic. Done well, it keeps production AI available under load, holds spend in check, and gives a platform team a single point of control.
Routing is the piece buyers underestimate. A gateway that only proxies calls still leaves each app tied to one model. A gateway with real routing sends routine requests to a small, fast model, reserves a larger model for the hard ones, and shifts traffic the moment a provider degrades, without touching application code.

What is an AI gateway?

An AI gateway is a control plane for model traffic. Applications, agents, and notebooks send inference requests to the gateway instead of calling a provider directly, and it handles authentication, routing, rate limiting, logging, and safety checks in one place. Developers call a stable endpoint; the platform team decides which models serve it and how requests are capped. Intelligent routing adds the dynamic layer: using rules or request metadata, the gateway splits traffic across endpoints, load-balances across replicas, and falls back to a secondary model when the primary is unavailable, transparently to the caller.

Key capabilities to evaluate

Capability	Why it matters
Traffic routing and splitting	Send requests to different models by rule or metadata, and split traffic for staged rollouts.
Automatic fallback	When the primary is throttled or errors, route to a backup with no code change so the app stays up.
Rate limiting	Cap tokens or queries per user, endpoint, or key so one workload can't starve the rest.
Spend caps and attribution	Halt requests when a budget is exceeded, and attribute spend by user, team, and use case.
Usage tracking	Record requests, tokens, and latency to see who uses which model and why.
Guardrails and PII handling	Screen prompts and outputs, restrict topics, and redact or block sensitive data.
Payload logging	Keep a full, queryable record of requests and responses for debugging and audit.
Provider-agnostic support	Route to models you host, open-source models, and external providers through one interface.

How Databricks approaches AI gateways with routing

Databricks Unity Gateway is the governed control plane for model and agent traffic on the platform, and it runs through Unity Catalog. Applications call a model service, and the gateway routes each request according to policy.

• Routing and fallback. A model service can route to one or many models with traffic splitting and fallback. If the primary endpoint is rate-limited or errors, traffic shifts to a backup automatically, with no application change.
• Rate limits and spend caps. Cap tokens or queries per user, endpoint, or key, and set hard spend limits that stop requests when a budget is hit. Spend is tracked across Databricks-hosted models, external models, and coding or custom agents.
• Usage tracking. Requests, token usage, and latency land in Unity Catalog system tables, with cost attributed by user, team, tool, and use case.
• Guardrails. Built-in guardrails screen requests and responses, detect PII such as names, addresses, and card numbers, restrict to approved topics, and redact or block on a hit. They fail closed, so a misconfiguration blocks the request rather than letting it through.
• Payload logging. Full requests and responses are logged to Unity Catalog Delta tables, known as inference tables, for monitoring and audit.
• Governed as data. Models, agents, and MCP services are Unity Catalog securable objects, governed with the same permissions you use for tables and volumes.
• External providers. Register providers such as OpenAI and Anthropic centrally in Unity Catalog, so developers reach approved models without handling raw provider keys.

For details, see the Databricks documentation on Unity Gateway serving endpoints and AI governance on Databricks.

Use cases

Cost-aware inference. Route routine classification or formatting to a small, inexpensive model and send complex reasoning to a larger one, so you pay for capability only when a task needs it.
High-availability apps. Define fallback chains so a throttled or failed provider doesn't take a feature down; traffic reroutes to a healthy endpoint.
Multi-team platforms. Give teams self-service access to approved models while the platform team sets budgets, rate limits, and logging centrally.
Agent and MCP traffic. Govern the calls that agents and MCP services make the same way you govern direct model calls, with the same limits, logging, and guardrails.

FAQs

How is intelligent routing different from load balancing?

Load balancing spreads traffic evenly across identical endpoints. Routing decides which model should handle a request based on cost, task, or availability, and can fall back to a different model when the first choice is unavailable.

Can the gateway route to models outside Databricks?

Yes. Unity Gateway is provider-agnostic. You can route to models hosted on Databricks, open-source models you run yourself, and external providers such as OpenAI and Anthropic through one governed interface.

What happens when the primary model is down or throttled?

The gateway routes to a configured fallback endpoint automatically, with no application change, so requests keep succeeding while the primary recovers.

How does the gateway help control cost?

Rate limits cap usage per user or endpoint, hard spend caps stop requests when a budget is hit, and usage tracking attributes spend by team and use case.