Where does an AI gateway sit in the enterprise AI architecture?

Summary

• An AI gateway sits in the runtime path as a central control plane between the applications and agents that request AI and the models, tools, and MCP servers that fulfill those requests. On Databricks this control plane is Unity AI Gateway, built on Unity Catalog.
• It is the front door for AI interactions: every model call, tool invocation, and agent invocation flows through the gateway, is evaluated against the policies defined in Unity Catalog before execution, and is logged after.
• Governance is layered across three planes — an AI assets layer where Unity Catalog manages models, functions, connections, and services as securable objects; an AI traffic layer where the gateway routes requests and applies rate limits, fallbacks, and traffic splitting; and a behavior layer where service policies evaluate request and response content.
• Because it is part of Unity Catalog, the gateway governs your AI estate with the same permission model you already use for data, so there is no separate access model to maintain for AI.
• It provides one governed entry point to Databricks-hosted and external models, with guardrails, usage tracking, payload logging, and cost controls with budgets and caps.

Where does an AI gateway sit in the enterprise AI architecture?

An AI gateway sits in the runtime path between the applications and agents that request AI and the models, tools, and MCP servers that fulfill those requests. Instead of every application connecting directly to every model and tool, traffic flows through one governed control plane where access is authorized, policies are enforced, and usage is recorded. On Databricks, that control plane is Unity AI Gateway, built on Unity Catalog: it routes traffic to the model and MCP services your organization uses, enforces access and cost controls, and records usage from one place. See The AI gateway: a governance layer for agentic AI.

Why Databricks Unity AI Gateway

• The front door for AI interactions. Unity AI Gateway sits in front of your AI estate so that every model call, every tool invocation, and every agent invocation flows through the gateway, is evaluated against the policies defined in Unity Catalog before execution, and is logged after. This gives you one consistent place to authorize, observe, and control AI traffic.
• Three governance layers. Governance is layered into distinct planes. Unity Catalog manages the AI assets layer, where models, functions, connections, and services are registered as securable objects. The gateway manages the AI traffic layer, routing requests and applying rate limits, fallbacks, and traffic splitting. Service policies manage the behavior layer, evaluating each request and response and allowing, denying, or requiring approval based on who is calling and what the request and response contain. See how AI governance works on Databricks.
• Built on Unity Catalog. Because the gateway is part of Unity Catalog, you register an AI asset once as a securable object and then grant and revoke access using the same privileges you already use for tables and volumes, so you do not maintain a separate access model for AI.
• Unified access to models. The gateway is a single entry point to models hosted on Databricks and to external models from any provider, so applications and agents reach the model landscape through one governed path. See Build intelligent agents on every leading model.
• Guardrails, rate limiting, and traffic management. AI guardrails filter unsafe content and detect sensitive data such as PII on requests and responses, rate limits manage capacity and cost, and traffic splitting and fallbacks keep production endpoints available.
• Observability and cost control. Usage tracking records requests, token usage, and latency through system tables, payload logging captures full requests and responses in Unity Catalog inference tables, and budgets set thresholds and hard caps with alerts or blocking when spend exceeds them.

Getting started

• Read AI governance with Unity AI Gateway for an overview of the control plane.
• Review how AI governance works on Databricks to see the asset, traffic, and behavior layers.
• Set rate limits, traffic splitting and fallbacks, and budgets on your model and MCP services.
• Read The AI gateway: a governance layer for agentic AI for how the gateway fits an agentic architecture.

FAQs

Where does an AI gateway fit in an AI architecture?

It sits in the runtime path as a central control plane between applications and agents and the models, tools, and MCP servers they call, so that traffic is authorized, policy-checked, and logged in one governed place instead of through many direct connections.

What does an AI gateway govern?

On Databricks, Unity AI Gateway governs three layers: the AI assets registered in Unity Catalog, the AI traffic routed to those assets, and the behavior of each request and response as evaluated by service policies.

Do I need a separate access model for AI?

No. Because Unity AI Gateway is part of Unity Catalog, AI assets are governed with the same privileges you use for tables and volumes, so your existing data governance extends to models, tools, and agents.