What is an AI gateway and why do I need one?

Summary

• An AI gateway is a governed control plane that sits between your users, applications, and agents and the LLMs or model endpoints they call, enforcing identity, policy, and limits on every request and response.
• You need one because running many models and AI services otherwise leaves access rules, monitoring, and billing fragmented across providers; a gateway gives platform teams one place to control access, spend, and safety without per-provider integration.
• Databricks delivers this with Unity AI Gateway, the enterprise control plane built on Unity Catalog that routes every model and MCP request and governs it with the same permissions you already use for tables and volumes.
• Core capabilities: unified access to native and external models, permissions and rate limits, guardrails and PII detection, usage and cost tracking, request/response payload logging, and fallbacks.
• Unity AI Gateway brought these controls to model serving endpoints (generally available June 2025); Unity AI Gateway extends them across models, agents, MCP servers, and tools from one control plane.

What is an AI gateway and why do I need one?

An AI gateway is a centralized control plane that sits between the callers of AI (your users, applications, and agents) and the language models or model endpoints they use. It governs LLM traffic at the point of consumption, inspecting each request before the model processes it and each response after, with full awareness of who is calling and what your organizational policies allow. You need one because, as teams adopt more model families and AI services, each provider brings its own access rules, monitoring, and billing, which leaves governance fragmented and adds operational overhead. An AI gateway removes that friction by acting as a single control layer: platform teams get one view of usage, spend, and performance while the gateway enforces consistent permissions, guardrails, and rate limits, so multi-model AI becomes enterprise-ready without a separate integration for every provider.

Why Databricks Unity AI Gateway

Unity AI Gateway is the Databricks governance solution for enterprise AI. Built on Unity Catalog, it extends governance beyond your data and AI assets to the runtime interactions between models, agents, MCP servers, and tools, so you govern AI with the same model you already use for data. AI governance on Databricks spans three dimensions: asset governance through Unity Catalog, traffic governance through the gateway, and behavior governance through service policies.

• Unified model access. Databricks serves large language models natively through Foundation Model APIs, giving you pay-per-token access with no model infrastructure to run. You can also connect external model providers with your own key and route to their APIs through the same access control and traffic management, so every model reaches your teams through one governed interface.
• Governance and permissions. Register each AI asset, such as a model, MCP server, function, or connection, as a Unity Catalog securable object, then grant and revoke access with the same privileges you use for tables and volumes. You keep one access model for data and AI instead of a separate one for each provider.
• Guardrails and PII detection. AI Guardrails (in Public Preview) filter requests and responses for safety and sensitive data. Safety filtering blocks unsafe and harmful content, and PII detection flags sensitive information including credit card numbers, email addresses, phone numbers, bank account numbers, and social security numbers. Service policies can allow, deny, or require approval for individual requests and responses based on who is calling and what the request and response contain.
• Rate limits. Control who has access and how much: enforce consumption limits, such as requests and tokens per minute, on model and MCP services per user, group, or endpoint. rate limits protect production applications from being starved of capacity and help manage cost.
• Usage and cost tracking. Monitor requests, token usage, and latency across the gateway using system tables, and attribute cost to services, target models, principals, and tags. Set budgets with per-user thresholds and hard caps so spend stays predictable and chargebacks are straightforward.
• Payload logging. Log requests and responses to Unity Catalog-managed Delta tables (inference tables) for monitoring, debugging, and audit. Capturing what data was sent to and returned from model APIs lets you investigate data-leakage concerns and meet audit requirements.
• Fallbacks and traffic routing. Configure fallbacks so an external model request routes to a backup model on error, minimizing production outages, and use traffic splitting to load-balance requests across multiple model destinations for higher availability.

Getting started

• Read AI governance with Unity AI Gateway to see how asset, traffic, and behavior governance fit together.
• Review How AI governance works on Databricks to plan permissions, rate limits, and service policies for your teams.
• Enable governance on your endpoints with AI Gateway for serving endpoints, then turn on usage tracking and payload logging.
• Apply rate limits and configure traffic splitting and fallbacks to keep applications resilient.

FAQs

What does an AI gateway do?

An AI gateway governs the traffic between callers and models: it authenticates and authorizes each request, enforces permissions and rate limits, applies guardrails to requests and responses, logs payloads, tracks usage and cost, and can route to fallback models, all from one control plane.

Why do I need an AI gateway?

Without one, every model provider and AI service has its own access rules, monitoring, and billing, so governance and cost control become fragmented. A gateway centralizes access, safety, and spend controls so platform teams can make multi-model AI enterprise-ready without integrating each provider separately.

What is the difference between Unity AI Gateway and Unity AI Gateway?

Unity AI Gateway brings governance, monitoring, and production readiness to model serving endpoints and became generally available in June 2025. Unity AI Gateway is the enterprise control plane built on Unity Catalog that extends this governance across models, agents, MCP servers, and tools with a richer UI and expanded observability.

How does the AI gateway detect PII?

AI Guardrails (in Public Preview) can scan requests and responses and flag sensitive information such as credit card numbers, email addresses, phone numbers, bank account numbers, and social security numbers, and block unsafe or harmful content through safety filtering.

Where is AI gateway usage data stored?

All AI Gateway data is logged into Delta tables in Unity Catalog: usage and cost are tracked through system tables, and request and response payloads are logged to inference tables for monitoring, debugging, and audit.