How do I reduce cluster idle?

Summary

• Idle compute keeps accumulating DBU and cloud instance charges until it terminates, so the levers are shorter inactivity limits, compute that ends with the work, and serverless.
• Set auto termination on all-purpose compute, enforce it with a compute policy, and set Auto Stop on SQL warehouses.
• Run production jobs on job compute or serverless. Job compute terminates after the task runs, but a job compute shared across tasks stays on and idle while tasks that do not use it run.
• Pools keep idle instances for faster starts. Databricks does not charge DBUs for idle pool instances, but your cloud provider still bills them, and Databricks recommends serverless instead of pools where it fits.
• Find idle classic compute with the compute system tables, then terminate it sooner, resize it or move it to serverless.

How do I reduce cluster idle?

End idle compute quickly. Set auto termination on all-purpose compute, run jobs on job compute or serverless so the compute ends with the work, enforce limits with compute policies, and use the system tables to find compute that sits unused. Idle compute continues to accumulate DBU and cloud instance charges during the inactivity period before it terminates.

Where does idle time come from, and what controls it?

Compute	Where idle time comes from	Control
All-purpose compute	Runs until the inactivity period after the last command expires	Auto termination, set in minutes when you create the compute
Job compute for one task	None between runs: it starts when the task run begins and terminates after the task runs	Use job compute instead of all-purpose compute for production jobs
Job compute shared across tasks	Stays on and idle while tasks that do not use it run	Share it only between tasks that run back to back, or move tasks to serverless
SQL warehouse	Idle until Auto Stop fires	Auto Stop minutes; the default is 45 for pro and classic warehouses and 10 for serverless
Pool	Minimum idle instances never terminate, and the cloud provider bills idle instances	Set the minimum idle instances only as high as you need; Databricks recommends serverless instead of pools
Serverless compute	Databricks minimizes idle time, and billing follows DBU consumption, not cluster uptime	Move supported workloads

How do I set auto termination?

Specify an inactivity period in minutes when you create the compute. Databricks terminates the compute when the time since the last command exceeds that period. Compute counts as inactive once all commands, including Spark jobs, Structured Streaming, JDBC calls and web terminal activity, have finished. The period must be between 10 and 10,000 minutes, and clearing the checkbox or entering 0 turns auto termination off. DStreams do not report activity, so turn auto termination off for compute that runs them or move to Structured Streaming.
Enforce a limit with a compute policy so users cannot skip it. This policy attribute fixes the value and hides the setting; 30 is an example, so choose your own threshold:

{
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30,
    "hidden": true
  }
}

Terminating stores the compute's configuration. When you start it again, Databricks re-creates the compute with the same ID, installs libraries and reattaches notebooks. A terminated compute that is neither pinned nor restarted is deleted 30 days after termination.

How do I find idle compute?

The compute system tables record classic all-purpose, jobs and Lakeflow Pipelines compute. system.compute.node_timeline holds minute-level utilization, and system.compute.clusters holds each configuration, including auto_termination_minutes.

SELECT cluster_id,
       AVG(cpu_user_percent + cpu_system_percent) AS avg_cpu_pct,
       AVG(mem_used_percent) AS avg_mem_pct
FROM system.compute.node_timeline
WHERE start_time >= '<start-time>'
GROUP BY cluster_id
ORDER BY avg_cpu_pct

Compute with persistently low CPU and memory use is a candidate for earlier termination, a smaller size or serverless.

What else reduces idle?

• Autoscaling. Databricks adds and removes workers during a run, which makes it easier to achieve high utilization because you do not need to provision the compute to match a workload.
• Single-node compute. It is intended for jobs with small amounts of data or non-distributed work.
• Event triggers. Lakeflow Jobs can start a run on a table update or when files arrive in a Unity Catalog location, so the run and its compute start when the data is there.
• Serverless. For most new workloads Databricks recommends serverless compute, which requires no configuration.

FAQs

Which jobs should use serverless?

Serverless compute for jobs is the default for tasks that support it. Check the compute recommendation table first, because JAR and Spark Submit tasks are recommended for classic jobs compute, and review the serverless limitations before you move a job.

Is an auto termination period enough on its own?

No. It only bounds idle time after the last command. Pair it with job compute for scheduled work, a policy for consistency and a utilization check in the system tables.

Can a scheduled job reuse a terminated all-purpose compute?

Yes. A job scheduled on a terminated all-purpose compute autostarts it, but Databricks does not recommend all-purpose compute for production jobs.