How do enterprises evaluate which model to use for each AI request?

Summary

• Enterprises evaluate which model to use for each AI request along several axes at once: task complexity, the cost and latency budget, the size of the context, and how sensitive the data is.
• In production they don't decide by hand on every call. They configure it once, as routing rules and governance policies at a control point in front of the models, so each request lands on an appropriately sized model. The sensible default is routine work to a smaller, faster, lower-cost model and a frontier model only for the requests that need it.
• On Databricks, that control point is the Unity Gateway: it allowlists which models a team can call, splits traffic and fails over across endpoints, can route by task complexity, and evaluates each request against governance policies.
• Because it runs through Unity Catalog, model access, routing, and policy share one identity and governance model, and every request is logged for cost and quality analysis.

How do enterprises evaluate which model to use for each AI request?

Enterprises evaluate which model to use for each AI request along several axes at once: the complexity of the task, the cost and latency budget, the size of the context, and the sensitivity of the data involved. In production they don't make that call by hand on every request. They configure it once, as routing rules and governance policies at a control point in front of the models, so each request lands on an appropriately sized model automatically. The sensible default is to send routine work to a smaller, faster, lower-cost model and reserve a frontier model for the requests that genuinely need it. On Databricks, that control point is the Unity Gateway.

Why per-request model selection matters

No single model is the right choice for every request. A one-line classification and a multi-step debugging task have very different cost, latency, and capability needs. Pin every call to one frontier model and you overpay for the easy requests; route everything to one small model and it falls short on the hard ones. The stakes rise with agents, which issue many model calls per user action, so a weak default multiplies fast. Evaluating per request, not once per application, is what holds quality up while keeping cost in check.

Criteria to evaluate for each request

Criterion	What to weigh
Task complexity	Routine calls to smaller, faster models; hard reasoning to more capable ones.
Cost	Track per-request and per-token cost so routine work doesn't default to premium models.
Latency	Favor faster models on interactive paths; reserve heavier models for batch or deep reasoning.
Quality and accuracy	Measure output quality on your own tasks, not generic benchmarks, before routing production traffic.
Context window	Pick a model whose context window fits the request's inputs, like long documents or large code files.
Data sensitivity	Route regulated or sensitive data only to the models and regions your policies permit.
Availability	Keep a backup model for when a provider is throttled or down.

How Databricks approaches per-request model selection with the Unity Gateway

The Unity Gateway is the governed control point for model and agent traffic on Databricks. Developers call a stable endpoint; the platform decides which models serve it and under what rules. Several capabilities work together to evaluate and select a model for each request:

• Model allowlisting. Control which models each team can call through Unity Catalog permissions, so requests are restricted to an approved, tiered set instead of any model a developer happens to pick. Granting EXECUTE on a model service is what lets a caller invoke it.
• Traffic splitting and fallback. A model service can route across several endpoints and fail over automatically: if the primary is rate-limited or errors, the request moves to a backup until one succeeds, with no application change.
• Smart routing. The gateway can evaluate each request's task complexity with a model-based classifier and route it to the lowest-cost model able to handle it, sending simple requests to smaller models and harder ones to more capable models, without a developer choosing per call.
• Per-request policy evaluation. Service policies evaluate each request against attribute-based rules and can allow, deny, or require human approval based on the caller's identity and the request content, so model choice is governed, not only performance-tuned.

Because all of this runs through Unity Catalog, model access, routing, and policy share one identity and governance model, and every request is logged for cost and quality analysis. Smart routing and per-request service policies are newer capabilities, so confirm their current availability and lifecycle state in the Databricks documentation or with your account team before you rely on a specific behavior.
Registering external providers centrally, models from providers like OpenAI, Anthropic, and Google alongside open and Databricks-hosted models, lets developers reach approved models through one interface without handling raw provider keys, and lets you adopt new models as the frontier moves without rewriting applications.

Use cases

• Cost-aware routing: send classification, extraction, and formatting to a small model and route multi-step reasoning to a frontier model, so you pay for capability only when a request needs it.
• Resilient production apps: define a fallback chain so a throttled or failed provider doesn't take a feature down.
• Governed model choice: allow a team only its approved models, and let policies deny or escalate a request that carries sensitive data or comes from an unapproved caller.

FAQs

Should a person choose the model for every request?

No. Teams set the criteria and rules once (allowlists, routing, and policies) and let the control point apply them per request. Choosing by hand on every call doesn't scale.

How does routing decide between a small and a large model?

Smart routing evaluates each request's task complexity with a model-based classifier, then sends simpler requests to smaller, lower-cost models and harder ones to more capable models, without changing application code.

What happens if the chosen model is unavailable?

With traffic splitting and fallback, the request routes to a backup endpoint automatically when the primary is rate-limited or errors, so it still succeeds while the primary recovers.

Can model choice be governed, not only optimized for cost?

Yes. Service policies evaluate each request against attribute-based rules and can allow, deny, or require approval based on identity and content, so a request carrying sensitive data can be blocked or escalated regardless of performance.