How can I switch LLM providers without rebuilding governance?

Summary

• Put governance in the gateway, not the app. With Databricks Unity Gateway, you register each model provider once as a Unity Catalog object, so changing the model behind an application doesn't mean re-implementing access, cost controls, or logging.
• One interface across providers. Unity Gateway exposes an OpenAI-compatible API and a single endpoint across external model providers, so pointing at a different model is a configuration change, not an application rewrite. See supported APIs and integrations.
• Controls travel with the gateway: per-principal rate limits, budgets, usage and spend tracking, and request and response logging apply the same way no matter which model serves the request.
• Access is governed by Unity Catalog: providers are securables you grant with EXECUTE, and API keys are stored centrally and encrypted, never handed to client applications.
• Unity Gateway is generally available; some capabilities, including service policies, remain in Beta, so confirm current feature status before you depend on them.

How can I switch LLM providers without rebuilding governance?

Keep governance in the gateway instead of in each application. Databricks Unity Gateway registers every model, external provider, and tool as a Unity Catalog object and routes calls through one governed interface, so the provider a request lands on is a configuration detail. When you change the model behind an app, the access rules, rate limits, budgets, and logging stay in place because they were never wired into the application to begin with. That is the difference between changing a setting and rebuilding your controls.

What lets you change providers without rebuilding governance?

The goal is to decouple the model choice from the governance around it. These are the capabilities that make that possible.

Capability	Why it matters
Governance decoupled from app code	You can point at a different provider without touching the application or re-implementing policy
Unified, OpenAI-compatible interface	One API across providers means changing the target model is a config change, not a rewrite
Providers as governed objects	Access, credentials, and permissions are managed centrally, not per application
Consistent controls across providers	Rate limits, budgets, and logging apply the same way regardless of which model runs
Centralized, encrypted credentials	Provider API keys live in one governed place, not scattered across app configs
Usage and spend visibility	Cost tracking and audit stay comparable when the backend model changes

How Databricks approaches this with Unity Gateway

Unity Gateway is built on Unity Catalog, so the same governance you use for tables extends to models, providers, and tools.
Providers are securables. You register an external model provider once as a Unity Catalog object that holds its connection details and API key. The key is stored through a Unity Catalog connection and encrypted, and it isn't exposed to the applications that call the model. Access uses standard grants: EXECUTE to query a provider, MANAGE to administer it. See connect and govern any external model provider.
One interface across providers. Unity Gateway exposes an OpenAI-compatible API and a single endpoint across external model providers (for example, OpenAI, Anthropic, and Google), and it also supports provider-native APIs where you need provider-specific features. Because the client points at the gateway, changing which model serves a request is a configuration change, not an application rewrite. See supported APIs and integrations.
Controls stay put when the model changes. Because every call routes through the gateway, the same controls apply no matter which provider is behind it:

• Rate limits per user, team, or agent, plus budgets and spend caps.
• Usage and spend tracking on by default, with estimated external-model spend aggregated into a governed system table (system.ai_gateway.external_model_spend) and a usage dashboard.
• Request and response logging to governed Delta tables for monitoring and audit. See how AI governance works.
• Traffic splitting and automatic fallback across models, so you can route or fail over without app changes.

Newer controls: guardrails and service policies. Unity Gateway adds LLM-based guardrails on inputs and outputs, and service policies that allow, deny, or require approval for an action based on the user, agent, model, or content, including PII detection and redaction. Unity Gateway is generally available; some capabilities, including service policies, remain in Beta, so confirm the current status of a specific feature before relying on it. See what's new in Unity Gateway and safeguarding AI workloads with guardrails.

FAQs

Why does switching providers usually mean rebuilding governance?

Because teams often wire access, rate limits, and logging into each application or into a provider's own SDK. When governance lives in the gateway, the provider becomes a setting, and the controls don't move with it.

Does my application code change when I change the model?

Not for the governance layer. With an OpenAI-compatible endpoint, applications call the gateway and the target model is selected by configuration, so access, budgets, and logging carry over.

Where do provider API keys live?

In Unity Catalog. Each provider is registered with its credentials stored through a governed, encrypted connection, and keys aren't exposed to the calling application.

Can I track cost across different providers?

Yes. Usage and spend tracking is on by default, and estimated external-model spend is written to a governed system table and surfaced in a usage dashboard, so cost and audit stay consistent as the backend model changes.

Is Unity Gateway generally available?

Yes. Unity Gateway is generally available. Some capabilities, including service policies, remain in Beta, so confirm the current status of a specific feature before you depend on it.