How do AI gateways handle model provider outages and failures?

Summary

• When a model provider is rate-limited, erroring, or down, an AI gateway keeps applications running by rerouting traffic to a healthy model automatically, with no application change.
• Automatic fallback is the core mechanism: you list backup models on an endpoint, and if the primary fails, the gateway tries the next one in order until a request succeeds.
• Traffic splitting and load balancing spread requests across endpoints and regions, so no single provider is a bottleneck or a single point of failure.
• Rate limits and spend controls stop a spike or a retry storm during an incident from exhausting a provider's quota or running up a bill.
• Databricks Unity Gateway, generally available, provides fallback, traffic splitting, rate limits, and full request logging to Unity Catalog; service policies and spend controls are in Beta today, so confirm their current state for your workspace.

How do AI gateways handle model provider outages and failures?

An AI gateway keeps applications running when a model provider degrades by standing between your apps and the models and rerouting traffic automatically. When a provider is rate-limited, returning errors, or down, the gateway falls back to a backup model, spreads load across endpoints and regions, and enforces limits that keep one failure from cascading, all without the application changing its code. The provider becomes a swappable dependency behind a stable endpoint rather than a single point of failure.

What a provider failure looks like

Model providers fail in ordinary ways: a rate limit rejects a burst of requests, an endpoint returns server errors, a region slows down, or a model gets deprecated. Hardcode one provider into an application and any of these can take a feature down. A gateway absorbs the failure by treating the provider as one option among several and deciding, per request, where traffic should go.

Key capabilities to evaluate

Capability	Why it matters
Automatic fallback	When the primary model is rate-limited or errors, route to a backup with no code change so the request still succeeds.
Traffic splitting and load balancing	Spread requests across endpoints and regions so no single provider is a bottleneck or a point of failure.
Rate limits	Cap requests or tokens per user, group, or endpoint so one workload cannot exhaust a provider's quota for everyone.
Spend controls	Stop requests when a budget is hit, so a retry storm during an incident does not become a runaway bill.
Ordered retries	Try backup models in a defined order until one responds, then stop.
Observability	Record which endpoint served each request, including fallbacks, so you can see and diagnose an incident.
Provider-agnostic switching	Change or add providers by configuration, without redeploying applications.

How Databricks approaches this with Unity Gateway

Databricks Unity Gateway is a governed control plane for model and agent traffic, and it is generally available. Applications call a serving endpoint, and the gateway decides how each request is handled.

• Automatic fallback. Configure fallback models on an endpoint. If the primary is rate-limited or returns an error, the request routes to the next model in the list, in order, until one succeeds, and the attempt that served the request is recorded. The application does not change. If a model's quota is exhausted, traffic falls back to another model; if a provider has an outage, requests route to an alternative.
• Traffic splitting and load balancing. An endpoint can split traffic across multiple models to balance load, and requests can be distributed across providers and regions for production uptime. Traffic splitting works for external model endpoints, Foundation Model API provisioned throughput endpoints, and custom model endpoints.
• Rate limits. Set limits at the endpoint, user, or group level so a spike or a stuck loop cannot exhaust capacity, which protects availability before an incident starts.
• No code change. Endpoints are OpenAI-compatible, so you write the application once and switch or add models by updating endpoint configuration rather than redeploying.
• Observability. Requests, tokens, and latency are tracked in system tables, and full request and response payloads can be logged to inference tables as governed Delta tables, so you can reconstruct what happened during a provider incident. See AI governance with Unity Gateway and Unity Gateway for serving endpoints.
• Guardrails and policies. Guardrails screen requests and responses and fail closed, so a misconfiguration or evaluator timeout blocks a request rather than letting an unchecked one through. Service policies and spend controls add content rules and hard budget caps; these are in Beta today, so confirm their current state for your workspace before you rely on them.

Use cases

Keep a production feature up during an outage. Define a fallback chain so a throttled or failed provider reroutes to a healthy one and the user never sees an error.
Absorb a demand spike. Split traffic and cap per-user rates so a burst does not exhaust one provider's quota and take the whole application down.
Swap a model safely. Point the endpoint at a new model in configuration when a provider deprecates one, with no application redeploy.

FAQs

What does an AI gateway do when the primary model is down?

It routes the request to a configured fallback model, tried in order until one succeeds, with no application change. If every fallback fails, the request fails and the attempts are logged for review.

Is failover automatic or manual?

Automatic. Once you configure fallback models on the endpoint, the gateway handles rate-limit and error responses by moving to the next model itself, so no one has to intervene during an incident.

Does routing through a gateway add latency?

A gateway adds a routing and policy step in the request path, so it introduces some overhead in exchange for failover, limits, and logging. On Databricks that step runs inline as the request is routed.

How does the gateway keep an incident from becoming a cost overrun?

Rate limits cap usage per user, group, or endpoint, and spend controls, currently in Beta, can stop requests when a budget is hit, so a retry storm during an outage does not turn into a runaway bill.