OpenAI-compatible API endpoints for open source models

Summary

• Databricks Model Serving exposes a unified OpenAI-compatible API across multiple foundation model providers, allowing you to use the same client code regardless of the model source.
• The MLflow Deployments SDK enables external model endpoints that accept standard OpenAI client calls (completions, chat, embeddings), with custom provider support for open source models behind proprietary proxies.
• Query endpoints using the OpenAI Python client by configuring the workspace base_url and authentication token, eliminating the need to learn provider-specific SDKs.
• Unity Gateway adds governance to model endpoints with usage tracking, payload logging, request guardrails, and rate limits for enterprise control.

OpenAI-compatible API endpoints for open source models
Databricks Model Serving provides a unified OpenAI-compatible API across external foundation models, letting your applications query open source models (or any OpenAI-compatible endpoint) using the standard OpenAI Python client. This eliminates the friction of learning provider-specific SDKs and makes it easy to switch between models.

What is an OpenAI-compatible API?

An OpenAI-compatible API mirrors the OpenAI API surface so that applications written against it can query different model providers without code changes. The client sends messages to a /chat/completions or /completions endpoint and receives structured responses. This compatibility layer allows development teams to experiment with multiple models or switch providers without refactoring applications.

Benefits of OpenAI-compatible endpoints

Benefit	What it means
Single SDK	Use the OpenAI Python client for all model providers, no SDK switching
Code portability	Change the model parameter or base_url without rewriting application code
Provider independence	Test multiple providers or migrate to open source without application refactoring
Reduced learning curve	Teams familiar with OpenAI API can query any external model immediately
Governance ready	Add security policies and usage tracking on top without changing the API

How Databricks enables OpenAI-compatible endpoints

Model Serving and External Models
Databricks Model Serving hosts external model endpoints that are natively OpenAI-compatible. All external models (whether from OpenAI, Anthropic, Cohere, cloud-hosted AI services, or custom providers) are queried using the same OpenAI API. You configure the endpoint in the workspace and query it with the OpenAI client library.
MLflow Deployments SDK
Create external model endpoints using the MLflow Deployments SDK. The SDK defines which provider and model to use, and Databricks exposes it as an OpenAI-compatible endpoint:

from openai import OpenAI
import os

client = OpenAI(
  api_key=os.environ.get("DATABRICKS_TOKEN"),
  base_url="https://<workspace-name>.cloud.databricks.com/serving-endpoints"
)

response = client.chat.completions.create(
  model="your-endpoint-name",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain data lineage."}
  ],
  max_tokens=256,
  temperature=0.7
)

You use the standard OpenAI client without learning a new SDK. The base_url points to your Databricks workspace, and authentication is handled via your Databricks token.
Custom Provider Support
For open source models or proprietary models behind custom proxies that expose an OpenAI-compatible interface, use the custom provider option. This allows you to wrap any OpenAI-compatible service and route it through Databricks Model Serving.

Enterprise governance with Unity Gateway

After creating external model endpoints, configure Unity Gateway to add governance layers:

• Usage tracking monitors API calls per model, endpoint, and user.
• Payload logging records request/response pairs for audit and troubleshooting.
• Request guardrails enforce content filters or rate-limit sensitive use cases.
• Rate limits prevent abuse or unexpected cost spikes.

Unity Gateway ensures your external model endpoints comply with security and compliance policies.

Automated endpoint creation with Genie Code

Genie Code in Agent mode can automatically generate notebooks that create external model endpoints using the MLflow Deployments SDK. This accelerates onboarding and reduces manual configuration errors.

Use cases

Model Experimentation – Data scientists test multiple open source models (LLaMA, Mistral, Qwen) or proprietary providers (OpenAI, Anthropic) by switching model endpoints without refactoring application code.
Cost Optimization – Fine-tune or replace expensive models with lower-cost open source alternatives by updating the endpoint configuration, not the calling code.
Vendor Lock-In Mitigation – Teams avoid depending on a single model provider by building against the OpenAI-compatible interface and routing to different providers based on availability or cost.
Internal Proxy Compliance – Organizations with strict network policies or proprietary model proxies can route through custom provider endpoints while maintaining OpenAI API compatibility for downstream applications.

FAQs

Can I use the same code to query different model endpoints?

Yes. The OpenAI client needs only the base_url and model parameter to change. The rest of your application code remains identical, enabling rapid model switching and A/B testing.

Do I need to use the Databricks SDK for external models?

No. You use the standard OpenAI Python client (or any OpenAI-compatible HTTP client). Model Serving handles translation to the underlying provider API.

What happens if an external model provider goes down?

You can quickly update the endpoint to route to a fallback model or provider. Downstream applications continue to work without code changes because they're querying the Databricks endpoint, not the provider directly.

How does Databricks handle authentication with external providers?

Databricks stores provider credentials (API keys) securely and injects them server-side when calling external models. Your application only needs a Databricks token; provider credentials are never exposed to the client.