How can an AI gateway manage experimental and production AI models?
Summary
- An AI gateway manages both by putting each behind its own governed service, with separate access, rate limits, budgets and logs, so an experiment can't spill into production traffic or spend.
- Experimental services stay open to the team running the test, under tight limits and an early-alerting budget; production services get scoped access, workload-sized limits, fallbacks and guardrails.
- Promotion becomes a traffic shift: add the candidate model behind the production service with a small share of traffic, compare results, then make it primary and keep the old model as a fallback.
- Logging per service and lineage make every change auditable and show who depends on a production model before you change or retire it.
- On Databricks, Unity Gateway model services are Unity Catalog securables with rate limits, traffic splitting, fallbacks, tag-scoped budgets, inference tables and service policies (Beta).
How can an AI gateway manage experimental and production AI models?
An AI gateway manages experimental and production models by putting each behind its own governed service, with its own access list, limits, budget and logs, and by promoting a model through a traffic shift, not a code change. Experimental services stay open to the people testing them, under tight limits and a small budget. Production services get scoped access, fallbacks, guardrails and full request logging. Applications keep calling one stable name while the model behind it changes.
The separation matters because experiments and production want opposite things. Experiments need breadth and speed: new models, new prompts, new providers, a small audience. Production needs predictability: stable names, known capacity, guardrails and an audit trail. Run both through the same endpoint and the same limits, and one undermines the other.
What to configure differently for each environment
The table below is a suggested pattern built from the controls Unity Gateway offers, not a Databricks-prescribed setup.
| Control | Experimental services | Production services |
|---|---|---|
| Location and access | A sandbox catalog or schema; access for the team running the experiment | A production schema; access for the application's service principals and approved users |
| Rate limits | Low service and per-user limits | Limits sized to the workload, with custom limits for specific users or groups |
| Spend | A tagged budget that alerts early and can block | A tagged budget with alerts to the owning team |
| Routing | One destination, or an A/B split across candidate models | A primary model plus ordered fallbacks |
| Guardrails | Baseline content policies | Policies for PII, prompt injection and unsafe content |
| Logging | Inference tables for evaluation | Inference tables and usage tracking for debugging and audit |
| Change management | Change freely | Check downstream consumers before any change |
How Databricks approaches experimental and production models
In Unity Gateway, a model service is an endpoint that routes inference requests to one or more models. It's also a Unity Catalog securable: it lives in a catalog and schema and inherits schema settings such as workspace bindings. That makes environment separation a matter of where you create the service and whom you grant.
- Separate services per environment. The model services docs describe a test service and a production service with different service rate limits, and separate services per team with their own grants. Grant
EXECUTEonly to the people and service principals that should call each one. - Stable production names. Give an application a model-agnostic service and change the underlying model without changing the name the application calls.
- Promote with traffic. Use traffic splitting to send part of the traffic to a new model before a broader rollout. Session affinity keeps a session on one destination. Fallbacks retry failed requests against backup destinations in the order you define. The
routing_informationfield insystem.ai_gateway.usageshows where requests actually went. - Limits that fit the stage. Rate limits apply by requests per minute (QPM) or tokens per minute (TPM) at the service level, as a per-user default, or as custom limits for users, service principals and groups. Requests over a limit get an HTTP 429 response.
- A budget per workload. Tag a service, for example
project=support-assistant, and scope a monthly budget to that tag with alert or block actions. - Guardrails (Beta). Service policies attach to a model service to govern each request and response, for example to block a response that contains unsafe content or PII. Built-in guardrails cover PII, prompt injection and unsafe content. Service policies are in Beta and are enabled separately from Unity Gateway itself, which is generally available.
- Logs and lineage. Send each service's requests and responses to its own inference table so its traffic can be inspected separately. Lineage tracks downstream consumers of a model service, so you can scope the impact of deprecating or changing it and notify its owners.
FAQs
Can experimental and production traffic share a model?
Yes. Rate limits, grants, routing and logging are part of each model service's configuration, so separate services can apply different controls even when they route to the same underlying model.
How do I promote an experimental model to production?
Add the candidate as a destination on the production service with a small traffic share, compare results in its inference table and the usage table, raise the share, then make it primary and keep the previous model as a fallback. Note that traffic splitting can't be configured on fallback destinations.
Do rate limits stop every burst?
Not exactly. Unity Gateway's rate limiter is designed for low latency, so short bursts slightly above a configured limit can occur, and the average rate converges to the limit over a longer window.
What if an experiment needs a model from another provider?
Register the provider once as a model provider service and route an experimental service to it. Callers use their Databricks identity and never see the stored credentials.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.