Why do I need an AI gateway?

Summary

• You need an AI gateway once more than one team, agent, or application calls AI services. It gives you one place to decide who can call which models and tools, record usage, apply guardrails, and keep an audit trail.
• Without a gateway, each application keeps its own provider credential and its own limits. In Unity Gateway, a model provider service holds the credential: Databricks encrypts and stores it, doesn't return it on read, and callers don't see it.
• Spend control combines usage tracking, budgets that alert or block, and rate limits by service, user, or group. No rate limits are configured by default.
• You can change the models behind a service, or add fallbacks, without changing the name your application calls.
• Guardrails (service policies, Beta) check requests and responses, and the sensitive data policy can block or redact. MCP servers use the same permission model as models.

Why do I need an AI gateway?

You need an AI gateway when AI use spreads beyond a single application. At that point, questions such as who can call which model, what each team is spending, which requests carried sensitive data, and what an agent did with a tool need one answer, not one per application. An AI gateway provides that answer in one place. The Unity Gateway docs put the goal this way: "Control which AI services teams can use, route and manage AI traffic, set guardrails with service policies, and monitor usage from one control plane."

What problems does an AI gateway solve?

Problem	Without a gateway	What Unity Gateway documents
Scattered credentials	Each application stores its own provider key	A model provider service holds the credential, and callers don't see it
Unclear access	Each application decides who can call what	Unity Catalog privileges such as EXECUTE on model services, model providers, and MCPs
Cost blind spots	Spend sits on separate provider bills	system.ai_gateway.usage records tokens by requester, and budgets can alert or block
Uneven safety checks	Each team decides whether to filter content	Service policies (Beta) attach to model services, model provider services, and MCP services
Provider churn	Switching models means changing code	Change models or fallbacks without changing the service name
Unaudited tools	Agents call tools with no central record	MCP servers are governed securables with tool allowlists, rate limits, and usage tracking

How does Unity Gateway control spend and traffic?

Budgets "help you monitor and control monthly Unity Gateway spend using filters applied to billing records." When a threshold is reached, admins can send an alert or block usage. Block thresholds are enforced approximately, so some spend over the threshold can occur before enforcement takes effect.
Rate limits cap requests per minute and, for model services, tokens per minute. They can apply to the whole service, to every user by default, or to individual users, service principals, and groups. When a limit is exceeded, the service returns HTTP 429. The cost analysis dashboard breaks spend down by model service, target model, and requesting user or service principal. Smart Routing (Beta) selects the lowest-cost model capable of handling a task for supported coding agents.

How do guardrails and audit work?

The docs draw the line this way: "Unity Catalog grants determine whether a principal can call a service. Service policies govern how that interaction proceeds, based on the content of the request and response and on who is making the call."
Built-in policies cover unsafe content, jailbreak attempts, hallucinated content, and sensitive data. Policies run on the request before Databricks invokes the service and on the response afterward, and any error during evaluation results in DENY. For audit, system.ai_gateway.usage captures request and response details, inference tables can log raw JSON payloads, and you can audit access through system.access.audit.

Do you always need an AI gateway?

No. If one small team calls one model for work that involves no sensitive data, direct access is simpler. A gateway earns its place as more teams, providers, agents, and sensitive data enter the picture, because each addition adds credentials, limits, and logs to manage.

FAQs

What if we already have accounts with model providers?

You can bring them in. Unity Gateway connects external providers through a model provider service in Unity Catalog, and Databricks encrypts and stores the credential. External model spend shown in Unity Gateway is informational and might not match your final provider invoice.

Does an AI gateway cover coding agents?

Unity Gateway can govern an external coding agent's model access. You grant EXECUTE to control who can use a model service, set a rate limit, create a budget, and confirm the requests in usage tracking.

Do batch SQL calls with ai_query pass through every control?

No. Only usage tracking applies to ai_query batch calls. Rate limits, service policies, inference tables, and fallbacks don't.

Is Unity Gateway generally available?

Yes. The August 4, 2026 release note says Unity Gateway "is now generally available," and notes that some capabilities, including service policies and agent services, remain in Beta.