How does an AI gateway work?

Summary

• An AI gateway is a centralized governance and control plane that sits between users and AI models, extending Unity Catalog's security model to AI assets and runtime decisions.
• The gateway handles routing, rate limiting, model fallback, traffic splitting, and observability, keeping all request logs and token usage inside governed Unity Catalog Delta tables.
• Access to model services is controlled through Unity Catalog permissions; credentials for external providers are registered centrally, eliminating credential sprawl across teams.
• Unity Gateway adds minimal latency and supports OpenAI-compatible APIs and native Databricks APIs, so clients can connect with a simple base URL change.
• All AI traffic is encrypted, audited, and subject to networking controls including IP allowlists, Private Link ingress, and deny-by-default serverless egress.

How does an AI gateway work?
An AI gateway is a centralized governance and control plane that extends Unity Catalog governance to AI assets and runtime interactions. Instead of scattering model access, credentials, and policies across teams and applications, an AI gateway is the single entry point for all AI services: models, MCP servers, agents, and skills. It sits in the request path between users and models, evaluates requests against policies defined in Unity Catalog, executes the request, and logs everything back to Unity Catalog for audit, cost attribution, and compliance.
Databricks Unity Gateway is built on Unity Catalog, meaning AI assets (models, functions, connections, services) are registered as UC securables and governed with the same permissions, auditing, and policy controls as data assets. This unified model eliminates the security and audit gaps that emerge when AI and data are governed separately.

What is Unity Gateway?

Unity Gateway is a three-layer governance model for enterprise AI. The AI assets layer manages models, functions, connections, and services as securable objects with standard privilege-based access control. The AI traffic layer is the central control plane that routes requests, applies rate limits, fallbacks, and traffic splitting between providers and endpoints. The behavior layer uses service policies (attribute-based access control) to evaluate each request and govern request/response content: allow, deny, or require approval.
When a request reaches the gateway, it is evaluated against policies before execution and logged after. The gateway is API-compatible: it supports OpenAI-compatible endpoints and also native APIs like the Databricks Messages API and Gemini API.

Key capabilities to evaluate

Capability	Why it matters
Unified governance	All AI assets, data, and access controls live in one place; audit trails are comprehensive and compliant.
Routing and resilience	Rate limits cap tokens/queries per minute per user, endpoint, or key; fallback routes traffic to a backup model if the primary degrades; load balancing spreads demand across multiple providers or regions.
Multi-provider support	Bring your own capacity or use Databricks-managed capacity; mix Databricks-hosted models with external model providers and existing subscriptions through a single gateway.
Observability and cost tracking	All requests, token usage, and latency are tracked in UC system tables and logged to Delta tables for monitoring, debugging, and cost attribution.
Access control	Permissions are enforced via UC: creating a service requires USE CATALOG, USE SCHEMA, CREATE SERVICE, and EXECUTE on each model; using a service requires EXECUTE on the service plus USE CATALOG and USE SCHEMA.
Network security	Private Link ingress keeps traffic off the public internet; IP access lists limit connections to approved networks; serverless egress uses deny-by-default posture; all traffic is encrypted in transit (TLS 1.2+).

How Databricks approaches AI gateway governance

Unity Gateway is the governance layer for agentic AI workloads. Credentials for external model providers are registered centrally in Unity Catalog rather than scattered across development environments and applications, eliminating credential sprawl and reducing breach surface area. Requests flow through the gateway, which evaluates them against UC policies before execution. Request/response payloads are logged to UC-managed Delta tables, enabling audit, debugging, and compliance reviews.
Clients integrate with a simple base URL change:

from openai import OpenAI

client = OpenAI(
  api_key="<databricks_pat_token>",
  base_url="https://<workspace-url>/ai-gateway/v1"
)

response = client.chat.completions.create(
  model="system.ai.databricks-claude-opus-4-6",
  messages=[{"role": "user", "content": "Hello"}]
)

Or via curl:

curl https://<workspace-url>/ai-gateway/v1/chat/completions \
  -H "Authorization: Bearer <databricks_pat_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "system.ai.databricks-claude-opus-4-6",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Use cases

Agent orchestration: Route multi-model calls based on request type, enforce rate limits, and audit all interactions through Unity Catalog.
GenAI application development: Centralize credential management and rate limiting so teams only need a workspace URL and a single PAT token, eliminating scattered API keys.
Cross-cloud model federation: Bring your own capacity on AWS, Azure, and GCP; configure fallback routing between clouds; track spend from a single UC dashboard.

FAQs

What happens if my primary model provider goes down?

You can configure fallback routing: if the primary model (e.g., Claude) becomes unavailable, the gateway automatically routes traffic to a backup model (e.g., GPT-4) with no code changes. The fallback is transparent to clients.

How do I control who can use which models?

Access is controlled via Unity Catalog permissions. To use a model service, users need EXECUTE on the service plus USE CATALOG and USE SCHEMA on the containing schema. You can grant permissions to users, groups, or service principals just as you would any UC object.

Can I use Unity Gateway with models running outside Databricks?

Yes. Credentials for external model providers are registered in UC and managed by the gateway. You can mix Databricks-hosted models, external model APIs, and your own infrastructure in a single gateway.

What data does the gateway log?

The gateway logs requests, responses (full payload), token usage, latency, caller identity, model service used, and timestamp. All logs are stored in UC-managed Delta tables for review, debugging, and audit.