What is the best AI gateway?
Summary
- No AI gateway is best in the abstract. The best one covers the AI traffic you run (models, external providers, agents, and tools), enforces the access and guardrail rules you need, and gives you usage and audit data you can query.
- Judge any gateway on coverage, access control, routing and limits, guardrails, observability, spend control, and maturity, including which capabilities are generally available and which are in preview.
- Unity Gateway is the Databricks option. It is built on Unity Catalog, so privileges on models, model services, and MCP servers work like privileges on data. It is generally available, while service policies and Smart Routing are in Beta.
- Read the fine print on what applies where. Only usage tracking applies to
ai_querybatch calls, and governance configured on a legacy endpoint doesn't carry over to Unity Gateway. - Pilot on a non-critical workload and measure latency from the gateway's own usage data before you commit.
What is the best AI gateway?
The best AI gateway is the one that covers all the AI traffic you run and enforces the rules you need on it. A gateway that fronts models but not the tools your agents call leaves one gap, and one that logs calls but can't restrict them leaves another. Unity Gateway is built on Unity Catalog and extends governance "beyond your data and AI assets to the runtime interactions between models, agents, MCP servers, and tools," which suits teams that already govern their data and AI assets there.
How do you evaluate an AI gateway?
Apply these criteria to any candidate. The last column shows what the Unity Gateway docs state.
| Criterion | Question to ask | Unity Gateway documents |
|---|---|---|
| Coverage | Which models, providers, and tools can it front? | Databricks-served models, external models through model providers, and MCP servers |
| Access control | Can you grant access per service, group, or principal? | EXECUTE, CREATE SERVICE, READ METADATA, and MANAGE privileges on Unity Catalog securables |
| Routing and limits | Does it split traffic, fall back, and cap usage? | Traffic splitting across up to five destinations, fallbacks in priority order, and requests-per-minute and tokens-per-minute limits |
| Guardrails | What checks run, and what happens when one errors? | Service policies (Beta): built-in unsafe content, jailbreak, hallucination, and sensitive data policies, plus custom and external policies; an evaluation error results in DENY |
| Observability | Can you query who called what and inspect payloads? | system.ai_gateway.usage, inference tables, a built-in usage dashboard, and system.access.audit |
| Spend control | Can you alert on or stop spend? | Budgets with alert and block actions; block thresholds are enforced approximately |
| Maturity | Which parts are generally available? | Unity Gateway is generally available; service policies and Smart Routing are Beta |
How does Unity Gateway approach the job?
The docs organize Unity Gateway around four jobs: control access to AI services, route and manage AI traffic, set guardrails and access policies, and monitor usage, cost, and risk. Model services support unified APIs such as OpenAI-compatible Chat Completions, plus provider-native APIs for compatible models. Rate limits can apply to a whole service, to every user by default, or to individual users, service principals, and groups, and none are configured until you set them. Among the built-in guardrails, sensitive data detection is deterministic (pattern-based, with no evaluator model) and can redact matched values as well as block. Smart Routing (Beta) selects the lowest-cost model capable of handling a task for supported coding agents, and an account admin enables it from the account console Previews page.
How do you narrow the choice?
- List the AI traffic you have today: models, external providers, agents, and tools.
- Mark your non-negotiables: access rules, guardrails, audit, and spend limits.
- Check the lifecycle stage of each capability, not just the product.
- Pilot on a non-critical workload, and use the gateway's own usage data to measure latency.
FAQs
Which controls apply to ai_query batch calls?
Only usage tracking. Rate limits, service policies, inference tables, and fallbacks don't apply to ai_query calls, and ai_query supports Databricks-provided model services only.
Do settings from an older endpoint-level gateway carry over?
No. Governance configured on a legacy endpoint doesn't carry over, and the existing configuration stays on that endpoint. Recreate permissions, rate limits, budgets, service policies, usage tracking, inference tables, and traffic routing in Unity Gateway.
What happens when a guardrail blocks a request?
Databricks returns a successful HTTP 200 response instead of an error status. The response names the blocking service policy, and a top-level databricks_service_policy object carries the structured detail, including the block reason.
Can Unity Gateway front models that Databricks doesn't host?
Yes. You bring capacity from external providers to Databricks through a model provider service in Unity Catalog. Databricks encrypts and stores the provider credential, and callers don't see it.
Who can query the usage table?
By default, both the account admin and metastore admin roles are required to view or query system.ai_gateway.usage.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.