How do I reduce time on managing clusters?

Summary

• Serverless compute removes most cluster management for jobs, notebooks and Lakeflow Pipelines: Databricks manages instance types, memory and scaling, and upgrades the runtime for you.
• Cluster creation permission is not required for serverless jobs or pipelines, so all workspace users can use serverless compute to run their workflows.
• Serverless has trade-offs: no Spark UI or Spark logs, a short list of supported Spark properties, and no init scripts, instance pools or compute policies. Check the limitations before you move a workload.
• For workloads that must stay on classic compute, cut the work with compute policies, autoscaling, standard access mode, a long-term support runtime and fewer init scripts and hardcoded Spark settings.

How do I reduce time on managing clusters?

Move supported workloads to serverless compute, and simplify the classic compute you keep. Classic compute is deployed in your cloud account and is not managed by Databricks, while serverless compute is deployed and managed by Databricks. For most new workloads, Databricks recommends serverless compute, which requires no configuration.

What cluster management work does serverless remove?

Task	Classic compute	Serverless compute
Choose instance types and sizes	You pick the driver and worker instance types and the number of workers	Databricks continuously optimizes instance types, memory and processing engines for jobs
Scale	You set the minimum and maximum workers for autoscaling	Autoscaling is enabled automatically for jobs
Select a runtime	You choose a Databricks Runtime version; long-term support versions suit operational jobs	Databricks upgrades the runtime automatically
Grant access	Users need the unrestricted cluster creation entitlement or a policy	Cluster creation permission is not required for serverless jobs and pipelines
Install dependencies	Compute-scoped libraries and init scripts	Serverless environments; init scripts are not supported
Debug	Spark UI and Spark logs	Query profile; the Spark UI and Spark logs are not available

How do I move work to serverless?

• Jobs. Serverless compute for jobs is the default for every task that supports it. Databricks says most classic workloads can migrate with minimal or no code changes. A quick test is to run the job on classic compute with standard access mode and Databricks Runtime 14.3 or above: if it succeeds, it can migrate without code changes.
• Notebooks. Serverless compute is available by default in most workspaces, and workspaces with Unity Catalog have access automatically.
• Lakeflow Pipelines. New pipelines default to serverless, and Databricks recommends serverless for new pipelines. Serverless pipelines always use Unity Catalog and remove most infrastructure configuration.
• SQL warehouses. For serverless SQL warehouses, Databricks handles capacity management, patching, upgrades and performance optimization.

What do I give up, and what is still my job?

Serverless compute does not support R, Scala in notebooks, RDD APIs, DataFrame caching calls, Databricks Container Services, compute policies, instance pools or init scripts. It supports only a short list of Spark properties, and the platform terminates a run that exceeds 7 days. Streaming jobs must use supported triggers such as Trigger.AvailableNow().
Some work stays with you. Query design is still yours, and the query profile shows execution details and metrics for each query. Cost monitoring is yours too: read the billable usage system table, where usage can take up to 24 hours to appear.

How do I reduce the work on classic compute that has to stay?

• Use compute policies. They preconfigure compute for purposes such as personal compute, shared compute and jobs, which limits the decisions users make.
• Standardize the basics. Use standard access mode, the latest long-term support runtime and autoscaling.
• Stop hardcoding. Avoid fixed Spark settings such as spark.executor.memory or spark.dynamicAllocation.*, because they override built-in optimizations.
• Drop init scripts and compute-scoped libraries. Add libraries to policies, use %pip install in notebooks or define an environment spec. Databricks notes this also makes workloads easier to migrate to serverless.
• Skip pools where serverless fits. Databricks recommends serverless instead of pools for workloads that support it.

FAQs

Do I still need to optimize my code?

Yes. Serverless manages the compute, not the query. Use the query profile for execution details and metrics, because the Spark UI is not available.

How do I keep serverless spend in check?

Serverless billing is based on DBU consumption, not cluster uptime. By default, serverless jobs have no query execution timeout, so set one with spark.databricks.execution.timeout if you want to bound query run time. Watch spend through the billable usage system table and account budget alerts.

Which workloads should stay on classic compute?

Workloads that depend on R or Scala notebooks, Spark Submit tasks, init scripts, Databricks Container Services or runs longer than 7 days stay on classic compute while those limitations apply.

How do I install libraries without init scripts?

Use a serverless environment. Databricks caches the environment, so dependencies do not reinstall each time, and job tasks that share a dependency set also benefit within a run.