What is the best open-source AI gateway?

Summary

• There is no single best open-source AI gateway. Start by separating tools that run models from tools that route and govern traffic to them, because the two are often confused.
• vLLM, Ollama and Ray Serve serve or run models. MLflow's gateway and Agent Router (formerly Envoy's AI gateway project) are Apache 2.0 projects that route requests across model providers.
• Compare candidates on provider coverage, rate limits, failover, credential handling, access control, logging, and who operates and upgrades the service.
• Open source gives you the code and control, and you run it yourself. A governed platform service such as Unity Gateway ties access, budgets and audit to Unity Catalog privileges.
• The Unity Gateway CLI (ug), which connects coding agents, is described in the Databricks docs as an open-source launcher.

What is the best open-source AI gateway?

There is no single best open-source AI gateway, because the term covers different layers and the right choice depends on what you need to control. Some open-source tools run models, some route requests across model providers, and governance features such as budgets and audit trails vary by project. Decide which layer you are choosing first, then compare candidates on the criteria below.

What counts as an AI gateway?

An AI gateway sits between applications or coding agents and the models they call. It exposes one API, routes requests, and applies limits and logging. Model-serving tools sit behind that layer: they load a model and answer requests. The table shows open-source projects from both layers, described in their own words. It is neither exhaustive nor a ranking.

Project	Layer	How its own docs describe it	License
vLLM	Model serving	"a fast and easy-to-use library for LLM inference and serving," with an OpenAI-compatible API server	Apache 2.0
Ollama	Local model runner	Run a model on your computer; supports a subset of the OpenAI API	MIT
Ray Serve	Model serving	"a scalable model serving library for building online inference APIs"	Apache 2.0 (Ray)
MLflow's gateway	Gateway	"Unified API gateway for all LLM providers," with rate limits, fallbacks, guardrails and traffic splitting	Apache 2.0 (MLflow)
Agent Router	Gateway	"The open source control plane for AI and agent traffic, powered by Envoy"; formerly Envoy's AI gateway project	Apache 2.0

Key capabilities to evaluate

Compare any candidate on these points. The last column shows how the Unity Gateway documentation describes each one.

Capability	What to check	Unity Gateway documentation
Provider coverage	Which providers and API formats it routes to	OpenAI-compatible and provider-native APIs; model provider services for OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Microsoft Foundry and custom endpoints
Rate limits	Whether limits apply per service, user and group	Requests-per-minute and tokens-per-minute limits at service, user and group level
Failover and splitting	Fallback order and weighted routing	Fallbacks in priority order; traffic splitting across up to 5 destinations
Credentials	Where provider keys live and who can read them	Encrypted in a model provider service and not returned on read
Access control	Whether it follows your identity provider and data permissions	Unity Catalog privileges such as EXECUTE and MANAGE
Logging and audit	What is recorded, where, and for how long	Inference tables for payloads, and audit logs for access
Operations	Who runs, scales, patches and upgrades it	Configured as Unity Catalog objects through the UI, REST API, SDKs, CLI or Terraform

When should you choose open source, and when a governed platform?

Choose open source when you need to read and change the code, must self-host, and have people to run, scale, patch and upgrade the service. Choose a governed platform service when you want access, budgets and audit tied to the permissions you already manage in your data platform. Whichever you pick, check what the project documents rather than assuming a feature exists.

How does Databricks approach this?

Unity Gateway is the Databricks governance solution for enterprise AI, built on Unity Catalog. It routes requests to Databricks-hosted models, including open-weight models, or to external providers that you register once as model provider services. Access to models, providers and MCP servers uses the same Unity Catalog privileges as tables, so there is no separate access model for AI. Unity Gateway became generally available on August 4, 2026.
If you run your own serving layer, the documentation's example of a custom provider is an OpenAI-compatible endpoint registered with a base URL and a credential. vLLM and Ollama both document OpenAI-compatible APIs, so test the specific server and API paths you plan to use.

FAQs

Is vLLM an AI gateway?

vLLM describes itself as a library for LLM inference and serving, with an OpenAI-compatible API server. Gateways sit in front of servers like it: the Agent Router site lists self-hosted vLLM among the destinations it can route OpenAI-compatible requests to.

What hardware do vLLM and Ollama need?

vLLM lists support for NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs, plus hardware plugins for other accelerators. Ollama's documentation describes local models that run on your computer and cloud models that run without a local GPU. Check each project's documentation for model sizes and memory.

Which providers can Unity Gateway connect to?

A model provider service supports providers such as OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock and Microsoft Foundry, plus custom providers defined by an endpoint base URL and a bearer token or API key header. Credentials are encrypted and not returned on read.

Does the Unity Gateway CLI count as open source?

The Databricks tutorial describes it as "An open-source launcher that handles OAuth and writes each agent's configuration file." You install it from github.com/databricks/unity-gateway, so check the repository's license file for the exact terms.