How do I reduce startup on my Spark clusters?

Summary

• With serverless compute for jobs, notebooks, and Lakeflow Pipelines, you don't provision clusters at all: Databricks allocates and manages the resources, which speeds up start-up and scaling times.
• On serverless jobs, keep the Performance optimized setting on for time-sensitive work; standard mode uses less compute but tolerates higher startup latency.
• For classic compute, instance pools keep idle, ready-to-use instances, which reduces cluster start and autoscaling times, and a preloaded Databricks Runtime version speeds launches further.
• Sharing one job compute resource across a job's tasks means the cluster starts once per run, which the docs say can reduce latency associated with start-up times.
• Init scripts run on every node during startup; Databricks recommends compute policies, not init scripts, for installing libraries.

How do I reduce startup on my Spark clusters?

Start fewer classic clusters, and make the ones you still start find capacity waiting. Move supported jobs, notebooks, and pipelines to serverless compute, where Databricks allocates and manages the resources, which speeds up start-up and scaling. For workloads that stay on classic compute, back clusters with instance pools, share one job cluster across a job's tasks, and keep init scripts and library installs off the startup path.

Where does classic cluster startup time go?

A classic cluster gets instances from your cloud provider and loads the Databricks Runtime version onto them. Init scripts then run on each node before the Spark driver or executor JVM starts, and Databricks installs the cluster's libraries. To see which phase dominates, open the compute's Event log tab and compare the timestamps of lifecycle events such as STARTING, INIT_SCRIPTS_STARTED, INIT_SCRIPTS_FINISHED, and RUNNING.

Which levers shorten startup?

Lever	What it changes	Trade-off
Serverless compute	Databricks allocates and manages compute, which speeds up start-up and scaling	No init scripts; jobs that need large amounts of memory or many tasks can see longer startup
Performance optimized (serverless jobs)	The job starts and runs faster	Turning it off selects standard mode, which uses fewer DBUs and accepts higher startup latency
Instance pools	Cluster start and autoscaling times are reduced when nodes come from idle instances	Idle instances still incur cloud provider charges
Preloaded runtime on a pool	Launches speed up further	Helps only clusters that select the preloaded version
Shared job compute	One cluster serves several tasks in a job run	It stays on, idle, while tasks that don't use it run

How do serverless jobs and performance modes help?

Serverless compute for jobs runs your job without configuring or deploying infrastructure, and Databricks recommends serverless compute for all job tasks. It supports notebook, Python script, dbt, Python wheel, and JAR tasks. For time-sensitive jobs, leave Performance optimized on, and the job starts and runs faster. With it off, standard mode uses less compute and suits workloads that can tolerate higher startup latency. Install dependencies through serverless environments, which cache installed packages to reduce startup latency on later runs. Pin package versions too, since an unpinned package can resolve to a new version that has to be installed first.

How do instance pools speed up classic clusters?

A pool keeps idle, ready-to-use instances. When cluster nodes are created from them, start and autoscaling times are reduced; if the pool has no idle instances, it allocates a new one from the cloud provider. To get the most from a pool:

• Keep Minimum Idle Instances at or above the cluster size you request. Larger requests start like clusters without a pool.
• Choose a Preloaded Databricks Runtime version that matches your clusters. With None, the runtime downloads on demand.
• Create a pool for each instance type and runtime you use often, and use on-demand instances for short jobs with strict execution-time requirements.

Then point a shared job cluster at the pool:

resources:
  jobs:
    nightly_etl:
      name: nightly_etl
      job_clusters:
        - job_cluster_key: shared_cluster
          new_cluster:
            spark_version: '15.4.x-scala2.12'
            instance_pool_id: '<worker-pool-id>'
            driver_instance_pool_id: '<driver-pool-id>'
            num_workers: 2
      tasks:
        - task_key: ingest
          job_cluster_key: shared_cluster
          notebook_task:
            notebook_path: ../src/ingest.ipynb
        - task_key: transform
          job_cluster_key: shared_cluster
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ../src/transform.ipynb

The shared cluster starts with the first task that uses it and terminates after the last one, which the docs say can reduce latency associated with start-up times. It's scoped to a single job run, so other jobs and other runs of this job can't reuse it.

How do I keep init scripts from slowing startup?

Init scripts run during startup of each node, before Spark starts. Databricks recommends built-in platform features over init scripts whenever possible. Manage library installation with compute policies rather than init scripts, and use compute policies to set Spark configuration and environment variables. Init scripts on DBFS have reached end-of-life, so keep any you still need in Unity Catalog volumes or workspace files.

FAQs

Is a pool the same as keeping a cluster running?

No. A pool holds idle instances that clusters attach to when they start. Databricks doesn't charge DBUs for idle pool instances, but your cloud provider still bills for them. Setting Minimum Idle Instances to 0 avoids paying for running instances that aren't doing work; the trade-off is a possible wait when a cluster needs a new instance.

Should production jobs run on an always-on all-purpose cluster?

Databricks doesn't recommend all-purpose compute for production jobs. Use serverless compute for jobs or classic jobs compute. If you keep an all-purpose cluster for notebooks, set automatic termination so it shuts down after a period of inactivity.

Does serverless always start quickly?

Not always. Because serverless provisions enough resources to run the workload, a job that needs large amounts of memory or includes many tasks can take longer to start. Standard performance mode also accepts higher startup latency in exchange for using less compute.