How do I optimize model serving for lower latency and cost?

Summary

• Model serving latency depends on compute scale, payload size, and model complexity; cost depends on how compute is allocated (serverless pay-per-token, provisioned, spot instances).
• Choose Model Serving for low-latency endpoints that auto-scale, Feature Serving for cached feature lookups, or batch inference for high-throughput offline workloads.
• Optimize by reducing model size (quantization, distillation), caching predictions, batching requests, and using GPU or multi-instance acceleration when inference is compute-bound.
• Right-size provisioned endpoints and use spot instances to cut costs; monitor token consumption and endpoint utilization to find inefficiencies.

How do I optimize model serving for lower latency and cost?
Model serving cost and latency depend on how you deploy: serverless pay-per-token endpoints scale automatically but charge per token; provisioned clusters offer lower per-token cost at higher fixed expense; batch inference is cheapest for offline workloads. Latency depends on model size, compute, and network hops. Optimizing requires tuning the model itself, the serving infrastructure, and the request pattern.

Latency and cost trade-offs in model serving

Serverless Model Serving auto-scales to zero and charges only for tokens consumed, making it cost-effective for unpredictable traffic but sensitive to model size and complexity. Provisioned endpoints keep compute warm, reducing cold-start time but incurring fixed costs even when idle. Batch inference on Lakeflow Jobs is cheapest for periodic inference but has high latency. Feature Serving caches features to avoid repeated computation, reducing inference latency by storing precomputed embeddings or aggregates.

Optimization strategies

Strategy	Latency impact	Cost impact	Best for
Model quantization (int8, fp16)	Reduces model size, faster inference	Lower token cost per call	Large language models, embeddings.
Knowledge distillation	Smaller student model, faster inference	Lower token cost per call	Production when accuracy permits.
Request batching	Amortizes compute cost across requests	Cost per token stays same; throughput up	High-volume batch workloads.
GPU acceleration	Reduces latency for compute-heavy ops	Higher per-token cost but faster	Matrix ops, transformers.
Feature caching (Feature Serving)	Eliminates repeated computations	Flat cost for cache storage	Feature-heavy models with repeated queries.
Provisioned endpoints with auto-scaling	Predictable latency, no cold start	Fixed cost + variable	High-traffic production; SLA-sensitive.
Spot instances	No latency change	Lower cost	Non-critical, fault-tolerant workloads.

How Databricks approaches model serving optimization

Databricks Model Serving offers both serverless and provisioned endpoints. Serverless endpoints scale automatically and charge per token; provisioned endpoints let you choose instance types (GPU, CPU) and scale policies. MLflow's evaluation tools help you validate that quantization or distillation preserves accuracy before deployment.
For latency-critical paths, use provisioned endpoints with GPU instances and auto-scaling. Monitor endpoint utilization and token consumption via Databricks SQL dashboards; high utilization on a small endpoint suggests you need to scale up or use a smaller model. For cost optimization, use Spot instances for non-critical inference; batch inference on Lakeflow Jobs for overnight or weekly scoring runs.
Feature Serving lets you cache computed features (embeddings, aggregates) with automatic refresh, so inference skips expensive feature computation. Combine Feature Serving with a small, fast model for ultra-low-latency predictions.

Use cases

Real-time recommendation API: Use provisioned GPU endpoint with Feature Serving for precomputed embeddings. Reduces latency to single-digit milliseconds. Monitor token consumption to ensure per-request cost stays within SLA.
Batch credit scoring: Run predictions on Lakeflow Jobs daily; queries a small credit model. Cost-optimal because compute runs once per day, not per request.
Search ranking: Quantize a ranking model to int8, deploy on CPU-provisioned endpoint. Reduces model memory and latency; cost is proportional to traffic volume.
Real-time fraud detection: Use serverless Model Serving for auto-scaling, but deploy a distilled model to keep token size small. Evaluate distilled model against holdout test set to confirm detection accuracy hasn't regressed.

FAQs

What's the difference between Model Serving and Feature Serving?

Model Serving runs predictions on raw or computed features. Feature Serving caches precomputed features (embeddings, statistics) with automatic refresh, so models use cached features instead of computing them per request. Combine both: Feature Serving cuts inference latency, Model Serving handles the actual prediction.

Should I always use GPU acceleration for lower latency?

Not necessarily. GPUs reduce latency for matrix-heavy operations (large language models, transformers) but add cost. For small, fast CPU-based models, GPUs won't help and only increase cost. Profile your model to see where time is spent.

How do I monitor model serving costs and find optimization opportunities?

Query Databricks system tables for endpoint token consumption and request count. High token consumption per prediction suggests your model is large; consider quantization or distillation. High endpoint utilization suggests you need more capacity or a larger endpoint instance.

Can I switch between serverless and provisioned endpoints without redeploying the model?

Yes. Both serve the same MLflow model version. Switch by updating the serving configuration in Model Serving to change instance types or scale policies. No model retraining or repackaging needed.