Which model serving platforms automatically scale with traffic?

Summary

• Autoscaling only helps if you know what scales, how fast it scales up and down, whether it can scale to zero, and what happens when a spike outruns capacity. Ask those four questions of any model serving platform.
• Databricks Model Serving runs on serverless compute and automatically scales up or down to meet demand changes.
• On custom model endpoints you choose a compute scale-out size, or minimum and maximum provisioned concurrency, and the endpoint scales within that range: up almost immediately, and down every five minutes.
• Scale to zero is optional and causes a cold start, so Databricks advises against it for production. When a spike outruns capacity, requests queue and then receive HTTP 429, so set a sufficient minimum and retry with exponential backoff.

Which model serving platforms automatically scale with traffic?

Databricks Model Serving does. It runs on serverless compute, and the service automatically scales up or down to meet demand changes. Whatever platform you evaluate, judge it on the same points: what unit scales, how quickly capacity moves in each direction, whether it can scale to zero, and what happens when a spike outruns capacity. The sections below show how Databricks answers each one.

What should you look for in autoscaling model serving?

Question	Why it matters
What scales?	Concurrency, replicas, or tokens per second decide how you size and cap an endpoint
How fast does it scale up and down?	Slow scale-up queues requests during a spike, and slow scale-down pays for idle capacity
Can it scale to zero?	It saves idle cost but adds cold-start latency to the first request
What happens at the limit?	Requests may queue or be rejected, so clients need retries
Can you cap the maximum?	A ceiling controls spend
Does it cover your hardware?	CPU and GPU endpoints can scale at different speeds

How does Databricks Model Serving scale?

Custom models. In the Serving UI you pick a Small, Medium, or Large compute scale-out size. Through the REST API (POST /api/2.0/serving-endpoints) you set min_provisioned_concurrency, max_provisioned_concurrency (multiples of 4), and scale_to_zero_enabled. Provisioned concurrency is how many requests the endpoint can process at once, and you estimate it as queries per second multiplied by model execution time in seconds.

{
  "name": "uc-model-endpoint",
  "config": {
    "served_entities": [
      {
        "name": "ads-entity",
        "entity_name": "catalog.schema.my-ads-model",
        "entity_version": "3",
        "min_provisioned_concurrency": 4,
        "max_provisioned_concurrency": 12,
        "scale_to_zero_enabled": false
      }
    ]
  }
}

Scaling behavior. Endpoints scale up almost immediately as traffic rises and scale down every five minutes to match reduced traffic. New nodes serve traffic only after the model downloads and passes health checks, so model size and load time matter. GPU autoscaling takes longer than CPU autoscaling.
Scale to zero. An endpoint with scale to zero enabled shrinks to zero after 30 minutes of inactivity, and the first request afterward pays a cold start with no SLA on latency. Databricks does not recommend it for production endpoints because capacity is not guaranteed.
Traffic spikes. Autoscaling needs time to detect load, so a sudden surge can queue requests. Endpoints allow temporary queuing and return HTTP 429 beyond a defined threshold. To reduce queuing, set a minimum concurrency that covers baseline traffic plus typical bursts, enable route optimization (custom model endpoints only), and retry with exponential backoff in clients.
Foundation Model APIs. Pay-per-token endpoints are preconfigured in your workspace, and priority mode, set per request, admits a request ahead of best-effort traffic for more consistent latency. Provisioned throughput endpoints scale automatically up to the maximum tokens per second you configure, and on-demand capacity has no term commitment.

How do you tune an endpoint?

1. Estimate concurrency as queries per second multiplied by model execution time.
2. Set the minimum high enough for baseline traffic plus typical bursts.
3. Set the maximum to accommodate spikes while controlling cost.
4. Load test to measure real latency and confirm the settings.
5. Turn scale to zero off for latency-sensitive production traffic.

FAQs

Does Model Serving have cold starts?

After scale to zero, yes. The first request waits while the endpoint scales back up from zero. When an endpoint grows from a nonzero minimum, new nodes still need time to download the model and pass health checks.

Can I cap what autoscaling spends?

Yes. The maximum provisioned concurrency caps capacity, and system.billing.usage shows serving cost, with endpoint starts after scale to zero under a separate SKU. For LLM services, Unity Gateway rate limits enforce requests-per-minute and tokens-per-minute limits to manage capacity and cost.

Do GPU endpoints scale as fast as CPU endpoints?

No. Databricks documents that autoscaling for GPU serving takes longer than for CPU serving.

Do I choose between autoscaling and serverless?

No. Model Serving is serverless compute that scales automatically. Your choices are the concurrency range, whether scale to zero is on, and for LLMs, pay-per-token or provisioned throughput.