How do I optimize AI model selection for cost and quality?

Summary

• Route through one governed gateway. The Unity Gateway (Unity Gateway) gives you a unified, OpenAI-compatible API to route across Databricks-hosted foundation models, external models, and custom models without code changes.
• Match the serving mode to the workload. Use pay-per-token endpoints for experimentation and prototyping, and provisioned throughput for production workloads that need consistent performance and throughput SLAs.
• Control cost with governance. Set rate limits per endpoint, user, or group, configure budgets and spend alerts, and split traffic with automatic fallbacks.
• Track usage and spend. Usage tracking and payload logging to system tables give per-endpoint and per-user cost attribution.
• Measure quality with MLflow. MLflow experiment tracking and evaluation help you measure output quality when choosing a model.

How do I optimize AI model selection for cost and quality?

Optimizing model selection means routing each workload to the model that meets its quality bar at the lowest cost, then tracking usage and quality so you can adjust over time. On Databricks, the Unity Gateway (also called the Unity Gateway) is the governance and routing layer that makes this practical, giving you one API across many models plus the controls and telemetry to manage cost and quality together.

Why use the Unity Gateway to optimize model selection

• Unified routing across model types. The gateway exposes a unified, OpenAI-compatible API so you can route across Databricks-hosted foundation models, external models, and custom models registered in Unity Catalog, and switch between providers without changing application code.
• Serving modes matched to the workload. Foundation Model APIs offer pay-per-token endpoints, which are well suited to experimentation, demos, and proofs of concept, and provisioned throughput endpoints, which provide consistent performance and throughput SLAs for production workloads.
• Cost controls built in. The AI Gateway enforces rate limits at the endpoint, user, or group level, supports budgets with spend alerts, and allows traffic splitting with automatic fallbacks so you can manage capacity and availability.
• Usage tracking and cost attribution. Usage tracking and payload logging write to system tables, giving per-endpoint and per-user cost attribution and estimated spend for external models, so you can see where budget goes and adjust routing.
• Quality measurement with MLflow. MLflow experiment tracking and evaluation help you measure the output quality of candidate models against your own tasks so model choice is evidence-based, not guesswork.
• Automatic model selection (in preview). Smart Routing, currently in preview, can automatically route each request to a model based on task complexity -- sending simpler tasks to lower-cost models and complex reasoning to higher-capability models, with fallback if a model is unavailable.

Getting started

• Set up the AI Gateway to route and govern AI traffic across your models.
• Choose a serving mode: start with pay-per-token via Foundation Model APIs, then move production workloads to provisioned throughput.
• Query foundation, external, and custom models through one interface with Model Serving.
• Read The AI Gateway: a governance layer for agentic AI and new governance capabilities to scale AI agents.

FAQs

How does the Unity Gateway help control AI cost?

It enforces rate limits per endpoint, user, or group, supports budgets and spend alerts, and logs usage to system tables for per-endpoint and per-user cost attribution.

When should I use pay-per-token versus provisioned throughput?

Use pay-per-token endpoints for experimentation, demos, and proofs of concept, and provisioned throughput for production workloads that need consistent performance and throughput SLAs.

How do I balance cost and quality across models?

Route across models through one governed API, measure output quality with MLflow evaluation, track spend in system tables, and use Smart Routing (in preview) to route each request to an appropriate model by task complexity.