How do I serve CPU and GPU models on the same platform?

Summary

• Databricks Model Serving provides a unified platform for deploying CPU and GPU models without managing separate infrastructure.
• You define compute requirements (CPU or GPU) in your model's configuration, and the platform routes requests to the appropriate compute tier.
• Both CPU and GPU-accelerated models run under the same authentication, versioning, and monitoring framework.
• Load balancing automatically directs traffic based on model compute needs and availability.
• Cost optimization comes from consolidating CPU and GPU workloads without separate vendor lock-in.

How do I serve CPU and GPU models on the same platform?
Databricks Model Serving lets you deploy both CPU and GPU models on the unified Databricks platform. You specify the compute type (CPU or GPU) when you register or update a model, and the platform routes inference requests to the correct compute tier, handles authentication, logs, and scaling for both automatically.

What is Model Serving?

Databricks Model Serving is a managed inference platform that hosts trained models and serves predictions at scale. It integrates with MLflow to manage model versioning, and with Unity Catalog to control access. The platform abstracts away server management, scaling, and networking so you focus on model quality and inference performance.

Key capabilities to evaluate

Capability	Why it matters
Unified platform for CPU and GPU	Deploy all models in one place without separate systems or vendor relationships for GPU hosting
Automatic compute allocation	The platform provisions CPU or GPU compute per model; you don't manually configure or manage instances
Model versioning and rollout	Control which version of a model is live; route a percentage of traffic to a new version to test safety before full rollout
Request logging and monitoring	Every inference request is logged and queryable; debug model behavior and measure latency and throughput
Authentication and access control	Requests are authenticated via Databricks credentials; Unity Catalog permissions control who can call the model
Endpoint management	Models are published as REST endpoints with automatic SSL and authentication; no custom API gateway needed
Built-in rate limiting and quotas	Protect models from abuse or overload; set per-user or per-endpoint rate limits

How Databricks approaches CPU and GPU model serving

When you deploy a model to Model Serving, you specify the compute configuration: CPU (for lightweight models, feature engineering) or GPU (for deep learning, embeddings, or large language models). Databricks automatically provisions the right compute tier and routes requests to it.
For model versioning, use MLflow and Unity Catalog. Tag a model as "Champion" or "Challenger" and use traffic routing to test a new version on a percentage of traffic before promoting it fully. Rollback is one click if a version degrades performance or accuracy.
Request logging is built in. Every inference is recorded with input, output, latency, and model version. Query these logs to debug model behavior, monitor for data drift, or measure performance by user or input type. Set up alerts in Databricks to notify your team if latency exceeds a threshold or error rates spike.
For CPU-GPU heterogeneity, you can run multiple versions of the same model simultaneously: a CPU version for latency-sensitive, lightweight requests, and a GPU version for batch inference or complex tasks. Route traffic based on request characteristics (batch size, input complexity) using endpoint routing policies.
Scaling is automatic. If traffic increases, Databricks scales GPU or CPU compute up. If load drops, it scales down, reducing your inference bill. You define min and max replicas for cost control.

Use cases

A data science team can serve a lightweight feature-encoding model on CPU for real-time feature lookups, and a GPU-resident transformer model for semantic search or ranking. Both are published under the same authentication and billing model. Financial services firms can run CPU models for compliance scoring and GPU models for fraud detection side-by-side on the same platform. Recommendation engines can use CPU models for simple collaborative filtering and GPU models for learned representations, serving both to the same web application.

FAQs

Can I run the same model on both CPU and GPU compute?

Yes. You can deploy two versions of the same model: one optimized for CPU inference and one compiled for GPU. Route traffic between them based on latency requirements or input size. Databricks tracks both versions in MLflow and Unity Catalog.

How do I control costs between CPU and GPU inference?

Define min and max replicas for each model. CPU models typically need fewer replicas and cost less per request. GPU models cost more per request but deliver faster inference for large models. Monitor your inference logs to see which model tier is handling which traffic, then adjust routing or capacity.

How do I test a new GPU model before sending production traffic to it?

Use the traffic routing feature to send a small percentage of requests to the new model version. Monitor accuracy, latency, and error rates. When confident, increase traffic fully or roll back if issues appear.

What if a model fails or times out?

Model Serving monitors model health. If a model instance crashes, the platform restarts it. If a model times out consistently, you'll receive an alert. Check the request logs in Databricks to see what inputs triggered the timeout, then profile your model to optimize inference time.

Can I use custom metrics or instrumentation in my model?

Yes. Add custom logging or tracing to your model code. Model Serving captures stdout and stderr; query these logs in Databricks to correlate with official request logs. Many teams also emit custom metrics to a monitoring system.