Which AI gateways support model routing and cost controls?

Summary

• Model routing directs requests to different foundation models based on cost, capability, or availability, reducing spending and improving resilience.
• Cost controls include per-team budgets, rate limits, and request-level tracking so teams can forecast spending and prevent runaway costs.
• Leading gateways record routing decisions and per-request costs as queryable data so you can optimize rules and attribute costs for chargeback.

Which AI gateways support model routing and cost controls?

Model routing and cost controls are how teams reduce AI spending while maintaining reliability. A gateway with routing can direct requests to different models based on rules: send routine queries to cheaper models, escalate complex queries to capable models, and automatically fall back to alternative providers if a primary fails. Cost controls prevent overspending through budgets, rate limits, and transparent cost tracking.
Without these controls, every request hits the same expensive endpoint, and provider outages break your application.

What is model routing and cost control?

Model routing picks which model serves each request via traffic splitting (A/B testing), ordered fallbacks (retry on failure), or intelligent routing (pick by request properties). This reduces spending and improves availability.
Cost controls limit spending: per-team budgets, rate limits (requests per minute), request-level cost tracking, and spend alerts.

Key capabilities to evaluate

Capability	Why it matters
Traffic splitting	Test new models or providers; roll out gradually
Ordered fallback destinations	Ensure availability; retry cheaper models on failure
Dynamic routing by request properties	Route simple queries to cheap models, complex to capable ones
Per-user or per-team budgets	Prevent uncontrolled spending; enable chargeback
Rate limiting	Prevent runaway agents or retry loops
Request-level cost recording	Understand which models drive costs; optimize rules
Policy-based routing	Define rules without application code changes
Fallback circuit breakers	Skip failing providers; recover gracefully

How Databricks approaches model routing and cost controls

Unity Gateway provides integrated routing and cost controls. Configure a model service with a primary destination and up to five ordered fallback destinations. When a request arrives, Unity Gateway tries the primary first. If it fails (rate limit, timeout, error), it automatically retries against fallbacks in order.
Examples:

• Multi-provider resilience: Primary is a Databricks-hosted model. Fallbacks include OpenAI and Anthropic. If Databricks' model is unavailable, the gateway automatically tries OpenAI, then Anthropic. End users see one endpoint and experience no interruption.
• Cost optimization: Primary is a lightweight model. Fallback is a capable model. Most requests stay cheap; escalations only happen when needed.

Unity Gateway features:

• Traffic splitting: route portions of requests to each model for A/B testing and canary rollouts.
• Rate limits: enforce per-user, per-team, or per-endpoint request and token limits.
• Spend budgets: set account, workspace, or per-user budgets in Unity Gateway. Exceeding a budget can pause requests or trigger alerts.
• Per-request cost tracking: Unity Gateway records the model, provider, tokens, and cost for every request in system.ai_gateway.usage tables in Unity Catalog, queryable via SQL.
• Tags and cost attribution: tag requests for cost grouping by team, project, or use-case.

Because cost data is stored in Unity Catalog, you can build dashboards, set alerts, and analyze retrospectively: which models are used, which users drive costs, and whether routing rules work as intended.

Use cases

Multi-model resilience: A company's AI assistant routes through Databricks, OpenAI, then Anthropic. When OpenAI has an outage, requests automatically route to Anthropic with no application code changes.
Cost optimization: A data team queries Lakebase with an LLM assistant. The gateway routes simple queries (lookups) to cheap models and complex queries (analysis) to capable models. Cost tracking reveals per-query costs, enabling teams to optimize routing.
Per-team budgets and chargeback: A platform team manages a shared AI gateway. Teams are assigned budgets via tags. When a team exceeds its budget, the gateway downgrades requests to cheaper models or alerts. Teams see detailed usage reports for cost allocation.

FAQs

How does fallback differ from load balancing?

Load balancing distributes requests across healthy endpoints to spread load. Fallback is a sequential retry: try primary, and only if it fails, try the next option. Fallback is for reliability and cost optimization.

Can I set different budgets for different teams?

Yes. Unity Gateway allows per-user, per-workspace, and tag-scoped budgets. Tag requests with team or project metadata, and configure budgets that apply to those tags. This enables fine-grained cost control and chargeback.

What happens if all fallbacks fail?

The gateway returns an error to the caller. This is by design: it's better to fail clearly than hide a problem or retry indefinitely. You can configure timeouts and circuit breakers to fail fast.

Can I route based on request content?

Unity Gateway supports policy-based routing (by user, endpoint, region, time). For content-aware routing (classify prompt first, then route), implement that at the application level or in a custom MCP client.