How do I implement an AI gateway in an enterprise?

Summary

• Unity AI Gateway (generally available) is a unified governance layer that sits on your model serving endpoints and routes every model and MCP request through centralized control.
• It builds on Unity Catalog, where models, MCP servers, functions, and connections are governed as securable objects with the same privileges and access policies used for tables and volumes.
• Implementation covers permissions (grant EXECUTE to specific groups), rate limits (queries- and tokens-per-minute per principal), guardrails (safety, PII detection, allowed topics, blocked keywords), and service policies that allow, deny, or require approval per request.
• Usage and cost are tracked with inference tables for payload logging and system tables for operational usage, plus budgets and hard spend caps.
• Best practice: test guardrails and policies in log-only mode before switching to enforcement, and register external model providers and MCP servers so all AI traffic is governed centrally.

How do I implement an AI gateway in an enterprise?

An enterprise AI gateway gives you one place to govern how models, agents, and tools are accessed — controlling who can call them, how much they can consume, what content is allowed, and how usage is logged. On Databricks, Unity AI Gateway (formerly Unity AI Gateway) provides this as a control plane on your model serving endpoints. It routes every model service and MCP request through centralized governance and extends the same foundation you already use for data — Unity Catalog — to models, agents, MCP servers, and runtime AI interactions. Implementing it is a matter of layering identity, access, traffic controls, content guardrails, and observability onto your serving endpoints.

Why Databricks Unity AI Gateway for enterprise AI governance

• Governance built on Unity Catalog. Models, MCP servers, functions, and connections are managed as securable objects governed with the same privileges and access-control policies you use for tables and volumes, so AI assets inherit your existing governance model. See How AI governance works on Databricks.
• Access control and permissions. Grant EXECUTE on endpoints to specific groups rather than the whole workspace, following least-privilege principles, and use attribute-based access policies to scope what agents and users can reach.
• Rate limiting. Configure limits per principal using both queries-per-minute and tokens-per-minute caps, with tighter limits for service principals than for interactive users, and an endpoint ceiling above individual overrides.
• Guardrails on endpoints. Enable guardrails when creating a serving endpoint, including safety filtering for harmful content, PII detection, up to 50 allowed topics, and up to 50 blocked keywords. See Safeguard AI workloads with Unity AI Gateway guardrails.
• Service policies for behavior governance. Policies can allow, deny, or require approval for individual requests and responses based on the caller and the content. Built-in policies block PII, unsafe content, and prompt-injection or jailbreak attempts, and you can add prompt-based (LLM-as-judge) policies or SQL rules scoped by caller and tool.
• Usage tracking and cost attribution. Enable inference tables to log and audit request and response payloads, and use system tables to monitor operational usage and cost per endpoint, joined to billing usage for analysis.
• Budgets and spend controls. Set budgets with per-user thresholds and hard spend caps that halt requests when a budget is exceeded, using rate limits and budgets together as layered cost control. See Introducing AI spend controls in Unity AI Gateway.
• Provider and MCP management. Register external model providers with centrally managed credentials and route requests with traffic splitting and fallback, and register MCP servers so you can govern how agents discover and use tools.

Getting started

1. Provision users from your identity provider through single sign-on and group sync, then map those groups to endpoint access.
2. Create or open a model serving endpoint and enable AI Gateway on it.
3. Grant EXECUTE to the specific groups that need access, and set per-principal rate limits in queries- and tokens-per-minute.
4. Turn on guardrails (safety, PII detection, allowed topics, blocked keywords) and add service policies, testing them in log-only mode before switching to enforcement.
5. Configure inference tables for payload logging and system tables for usage, then set budgets and hard spend caps.
6. Review what's new in Unity AI Gateway for service policies, guardrails, observability, and cost controls.

FAQs

What is Unity AI Gateway?

Unity AI Gateway is a generally available governance layer that sits on Databricks model serving endpoints and routes every model and MCP request through centralized control, built on Unity Catalog so AI assets are governed like your data.

How do I control AI spend?

Set per-principal rate limits using queries- and tokens-per-minute caps, and configure budgets with per-user thresholds and hard spend caps that automatically halt requests when a budget is exceeded. Usage and cost are tracked in system tables.

How do I add guardrails and content controls?

Enable guardrails on a serving endpoint for safety filtering, PII detection, allowed topics, and blocked keywords, and add service policies that allow, deny, or require approval per request. Test in log-only mode before enforcing.

How is AI usage audited?

Enable inference tables to log request and response payloads to Unity Catalog and use system tables to monitor operational usage and cost per endpoint, giving you an auditable record of AI interactions.