What does an AI gateway do?

Summary

• An AI gateway is a centralized control plane that governs access, routes traffic, and monitors costs and security across all AI models, agents, and tools in an organization.
• It sits between users and AI services to enforce consistent governance policies, guardrails, and rate limits without requiring changes to existing application code.
• AI gateways enable enterprises to manage compliance, prevent unauthorized API usage, and gain visibility into which teams are consuming which models.
• They abstract away the complexity of managing credentials, integrations, and policies across multiple AI providers, from proprietary models to open-source tools.

What does an AI gateway do?
An AI gateway solves a fundamental problem in enterprise AI adoption: how to govern and control access to AI services without slowing down innovation. It intercepts requests, authenticates users, checks permissions, applies rate limits, and routes traffic, all without requiring code changes.

What is an AI gateway?

An AI gateway is centralized infrastructure that enforces three layers of control: who can access what (access control), how traffic is routed (traffic management), and what happens to requests and responses (behavior policies). Think of it as a firewall for AI: every request is checked against governance policies defined in a central catalog.

Key capabilities to evaluate

Capability	Why it matters
Centralized credential management	Eliminates credential sprawl by storing API keys once and reusing them across thousands of assets, reducing the risk of leaked credentials and simplifying rotation.
Multi-model routing	Routes requests to different models (proprietary or open), splits traffic between endpoints, and implements fallbacks if one service is unavailable.
Cost attribution and controls	Tracks spending per team, project, or user; enforces hard budgets; and prevents runaway costs from uncontrolled API usage.
Rate limiting and quotas	Prevents denial-of-service scenarios and ensures fair resource allocation by limiting requests per user, team, or application.
Request/response inspection	Enables guardrails against prompt injection, PII leakage, and harmful outputs through policy-driven content inspection.
Unified logging and audit	Captures every request, response, token count, and latency in a queryable audit log for compliance and troubleshooting.
Developer experience	Supports standard client libraries and APIs (like OpenAI-compatible endpoints) so teams don't need to rewrite applications.

How Databricks approaches AI governance

Databricks offers Unity Gateway, which went generally available on August 4, 2026, as the governance solution for enterprise AI. Unity Gateway extends the permissions model from Unity Catalog (used for tables, volumes, and functions) to the entire AI stack: models, model services, agents, MCP servers, and connections.
Unity Gateway operates on three governance layers:
Asset Governance: Models, model services, agent services, and MCP servers are registered in Unity Catalog as securable objects. You grant access (SELECT, EXECUTE) using the same role-based access control (RBAC) system used for tables and volumes. A team cannot call a model endpoint without explicit permission in the catalog.
Traffic Governance: Every request to a model, agent, or MCP tool is routed through Unity Gateway's control plane. The gateway authenticates the caller, checks their Unity Catalog permissions, applies rate limits and budget controls, and then routes the request to the appropriate endpoint (Databricks-hosted model, external provider like OpenAI or Anthropic, or a custom model service). All requests and token usage are logged to system tables and queryable Delta tables in Unity Catalog.
Behavior Governance: Service policies (currently in Beta) allow, deny, or require approval for individual requests and responses based on guardrails like PII detection and prompt injection mitigation.
Integration Pattern:
Unity Gateway endpoints are OpenAI-compatible, so existing client code works unchanged. You change only the base URL to point to your Databricks workspace:

from openai import OpenAI

client = OpenAI(
  api_key="<databricks_pat>",
  base_url="https://<workspace>/ai-gateway/v1"
)

response = client.chat.completions.create(
  model="system.ai.databricks-claude-opus-4-6",
  messages=[{"role": "user", "content": "Your prompt"}]
)

This simplicity is intentional. Teams can adopt governance without rewriting applications or learning new SDKs.
What gets governed:
Models, model services, MCP servers, AI agents, and custom integrations. All interactions are logged to Unity Catalog system tables, where you can query token usage, caller identity, and request outcomes for cost attribution and compliance.

Use cases

Cost control: A gateway enforces per-team budgets and rate limits to prevent runaway API spending, with alerts when teams approach their cap.
Compliance and audit: A gateway provides a centralized, queryable audit trail of who accessed which models, when, and for what purpose. This satisfies regulatory review requirements.
Model access control: A regulated model should only be accessible to specific teams. A gateway enforces this at the request level via Unity Catalog permissions.

FAQs

Do I need to change my code to use an AI gateway?

No, if your application uses an OpenAI-compatible client library. You change only the base URL and authentication token; the request/response contract stays the same.

Can a gateway work with multiple AI providers?

Yes. A gateway routes requests across Databricks-hosted models, OpenAI, Anthropic, Amazon Bedrock, and others, letting teams use the best model for each task without lock-in.

How do I track costs by team or project?

The gateway logs all requests to Unity Catalog system tables, including token counts and caller identity. Query these tables with SQL to attribute costs and set per-team budgets that the gateway enforces in real time.