How hard is it to build an AI gateway?

Summary

• A forwarding proxy is the easy part of an AI gateway. The effort sits in the layers around it: access control, rate limits, spend control, failover, credential handling and logging.
• Each layer forces tradeoffs. The Unity Gateway documentation spells out several: rate limits favor low latency over exact counts, budgets enforce against estimates, and request logging has size and error-response gaps.
• Provider credentials are a central problem, because without a registry every team embeds its own keys in application code or notebooks.
• Build your own when you need routing or policy logic that existing gateways lack. Use a governed control plane when your engineers are better used on agents and applications.
• Unity Gateway is generally available (August 2026) and built on Unity Catalog. Some capabilities, such as service policies, remain in Beta.

How hard is it to build an AI gateway?

A basic gateway that forwards requests to one model provider is a small project. A production gateway is much harder, because it also has to control who can call what, limit consumption, keep working when a provider fails, protect provider credentials, and record what happened. Each of those jobs has edge cases, and the Unity Gateway documentation shows what some of them look like.

What does an AI gateway have to do?

An AI gateway sits between applications or coding agents and the models they call. Databricks describes Unity Gateway as "a single control plane for model inference, tools, and skills across your AI applications and coding agents." If you build a gateway yourself, you own every design decision that comes with that scope.

Key capabilities to evaluate

Layer	Decision you would own	What the Unity Gateway docs show
Provider APIs	Which request formats to accept and translate	Model services support OpenAI-compatible Chat Completions and provider-native APIs such as Anthropic Messages for compatible models
Credentials	Where provider keys live and who can use them	Model provider services store encrypted credentials, and callers do not see them
Access control	Who may call which model or tool	EXECUTE, CREATE SERVICE, READ METADATA and MANAGE privileges on Unity Catalog securables
Rate limits	Exact counting or low latency	Concurrent requests are not checked ahead of time, so short bursts above a limit can occur
Spend control	Estimates or billed cost	Enforcement is approximate, and the docs say not to rely on it as an absolute spend cap
Failover	Which errors trigger a retry, and in what order	Failed requests, such as a 429 or 5xx error, retry against backup destinations in priority order
Logging	What to record and what to skip	Inference tables skip payloads above 10 MiB, and logs may be missing for some error responses

Where do teams get stuck?

Rate limits trade exactness for latency. The Unity Gateway rate limiter is, in the docs' words, "designed for low latency." It records usage after a response is sent and enforces limits independently across service instances, so you may see bursts followed by brief pauses. Precedence adds design work too: service, default per-user, and custom user, service principal and group limits, where a user-specific limit wins over a group limit.
Spend control rests on estimates. Budgets track a near real-time estimate. Requests already in progress when a threshold is reached are not interrupted, and spend in alert emails, the budget page and system.billing.usage can differ at the same moment because each source updates at a different rate. External provider spend is estimated from published prices, so it can differ from the invoice.
Credentials need a home. The docs put it plainly: without a central registry, every team that uses an external model embeds its own provider credentials in application code or notebooks. A model provider service moves them into Unity Catalog, where a key can be backed by a customer-owned Unity Catalog secret so that rotation needs no change to the service.

Should you build or buy?

Build when your requirements are unusual: routing logic, policy rules or network constraints that no existing gateway supports, or a gateway that is itself part of the product you ship. Use a governed control plane when the requirements are the common ones (access, budgets, failover, logging) and your engineers are better used on agents and applications. Plan for upkeep too: the release notes list new models and features added throughout August, September and early October 2026.

How does Databricks approach this?

Unity Gateway is the Databricks governance solution for enterprise AI, built on Unity Catalog. It became generally available on August 4, 2026, and some capabilities, including service policies and agent services, remain in Beta.

• Routing: a model service routes requests to one or more models with traffic splitting and fallbacks, and you can change models or fallbacks without changing the name your application calls.
• Models: Databricks hosts frontier proprietary and open-weight models, and you can bring existing capacity from providers such as OpenAI and Anthropic.
• Access: Unity Catalog governs models, MCP servers and functions with the same privileges you use for data, so there is no separate access model for AI.
• Monitoring: a built-in usage dashboard shows who used what, inference tables log request and response payloads, and audit logs show who accessed which AI service and when.

FAQs

Can I build a minimal gateway myself?

Yes. A proxy that forwards requests to one provider and logs them is a contained project. The effort grows with per-user limits, budgets, failover, guardrails and audit logging, and again whenever providers add models or APIs.

What happens when a provider fails?

With Unity Gateway fallbacks, a failed request is retried against backup destinations in the order you define. Traffic splitting covers up to 5 destinations, and fallbacks are independent of it: after the primary destination is chosen, splitting is not re-applied during retries.

Is Unity Gateway available everywhere?

It requires a workspace in a supported region, and it is not supported on AWS GovCloud.