What is a cold start?

Summary

• A cold start is the extra latency that occurs when a compute resource or model serving endpoint has to initialize from a suspended or scaled-to-zero state, so the first request waits while capacity spins up and the model or runtime loads.
• It is most visible with scale-to-zero: after an endpoint or compute suspends to save cost, the first request that arrives pays the startup cost before the resource is ready.
• On Databricks Model Serving, scaling up from zero typically takes about 10 to 20 seconds, and can take minutes for large GPU models because bringing a pod up takes time that grows with model size.
• Cold starts cannot be fully optimized away — there is a physical floor to how fast capacity can come up. The way to avoid the first-request delay is to keep a baseline of capacity ready with minimum provisioned concurrency.
• Databricks reduces cold-start latency with serverless warm pools and fast container startup that begin compute in seconds, plus a performance-optimized mode for serverless jobs and pipelines when startup speed matters.

What is a cold start?

A cold start is the added latency you experience when a compute resource or model serving endpoint has to initialize from a suspended or zero state. Instead of a warm, already-running resource answering immediately, the first request has to wait while capacity is provisioned and the runtime or model is loaded into memory. Cold starts are most noticeable with scale-to-zero: when an endpoint or compute automatically suspends during idle periods to save cost, the next request that arrives pays that startup cost before the resource is ready to serve.

How Databricks handles and reduces cold starts

• Serverless warm pools and fast startup. Serverless compute on the Databricks Platform starts in seconds because Databricks maintains warm pools of instances ready for immediate use and uses lazy-loading container images that fetch only the metadata needed to start, then load remaining blocks on demand. This brings startup down to seconds rather than a long provisioning wait.
• Scale-to-zero model serving. With Model Serving endpoints that scale to zero, the first request after the endpoint has scaled down is a cold start. Scaling up from zero typically takes about 10 to 20 seconds and can take minutes for large models, since a pod must come up and load the model before it can serve.
• Minimum provisioned concurrency. Because you cannot optimize cold starts away entirely, the way to avoid the first-request delay is to keep a baseline of capacity ready. Setting a minimum provisioned concurrency floor keeps pods fully ready with the model loaded, so the first request is served immediately with no queuing. A useful sizing rule is: provisioned concurrency = QPS multiplied by model execution time in seconds.
• Performance modes for serverless jobs and pipelines. Serverless jobs and pipelines offer a performance-optimized mode that prioritizes startup and execution time, alongside a standard mode that prioritizes cost and accepts a slightly higher launch latency. Choose the performance-optimized mode for latency-sensitive workloads where a cold start would be disruptive.

Getting started

• For latency-sensitive Model Serving endpoints, set a minimum provisioned concurrency floor so a baseline of capacity stays ready and avoids cold-start delay on the first request.
• Size that floor using the rule provisioned concurrency = QPS multiplied by model execution time in seconds.
• For serverless jobs and pipelines, pick the performance-optimized mode when fast startup matters and the standard mode when cost matters more than launch latency.
• Keep scale-to-zero enabled for intermittent workloads where a short cold start on the first request is an acceptable trade for paying only when the resource is in use.

FAQs

What is a cold start in simple terms?

It is the delay before a suspended or scaled-to-zero resource can serve its first request, because capacity has to be provisioned and the runtime or model loaded before work can begin.

How long is a cold start on Databricks Model Serving?

Scaling up an endpoint from zero typically takes about 10 to 20 seconds, and can take minutes for large models because bringing a pod up and loading the model takes time that grows with model size.

How do I avoid cold starts?

Keep a baseline of capacity ready by setting a minimum provisioned concurrency floor, which keeps pods loaded and serving immediately so the first request does not wait.