Which compute should I use for my Spark workloads?

Summary

• Databricks recommends serverless compute for most new Spark workloads. It requires no configuration, uses Spark Connect, and for jobs it turns on autoscaling and Photon automatically.
• Choose classic compute when serverless can't run the workload, for example when you need R, Spark RDD APIs, init scripts, Databricks Container Services, instance pools or Spark properties outside the supported list.
• On classic compute, use standard access mode unless the workload needs something it doesn't support. Dedicated access mode covers ML Runtime on GPU, RDD APIs and R.
• Run production jobs on job compute or serverless, not all-purpose compute. One Lakeflow Job can mix compute types by task, so serverless and classic tasks can live in the same job.

Which compute should I use for my Spark workloads?
Use serverless compute unless your workload needs something it doesn't support. Databricks recommends serverless for most new workloads, and for Lakeflow Jobs it is the default for every task that supports it. Classic compute is the fallback for R, RDD APIs, init scripts, Databricks Container Services, or Spark settings that serverless doesn't allow. For production jobs, choose job compute over all-purpose compute.

What are the Spark compute options?

Option	What it is	Best fit
Serverless compute	Databricks-managed compute for notebooks, jobs and Lakeflow Pipelines. It needs no configuration and is versionless.	Most new batch and interactive work using DataFrame, SQL and Python APIs
Classic jobs compute	Compute you configure for a job. It starts when the task run begins and terminates after the task runs, and tasks in one job can share it.	Workloads serverless can't run, plus JAR and Spark Submit tasks, where Databricks recommends classic jobs
Classic all-purpose compute	Compute you create and manage, in standard or dedicated access mode.	Interactive analysis in notebooks, but not production jobs
SQL warehouses	Serverless, pro or classic compute for SQL queries.	SQL and dbt tasks, ETL and BI
Lakeflow Pipelines compute	Serverless by default, or classic compute you configure.	Declarative batch and streaming pipelines

How do I decide?

Pick serverless when you want Databricks to manage the infrastructure. Serverless compute for jobs enables autoscaling and Photon automatically, and all workspace users can run it because cluster creation permission isn't required. Databricks Connect lets you run workloads on serverless from your local machine.
Pick classic jobs compute when a task needs a feature serverless lacks. Serverless doesn't support R, RDD APIs, compute policies, compute-scoped init scripts, instance pools, Databricks Container Services, or most Spark properties. Databricks also lists classic jobs as the recommended compute for JAR and Spark Submit tasks.
Pick all-purpose compute for interactive analysis in notebooks when you need classic features. Databricks does not recommend all-purpose compute for production jobs.
On classic compute, Databricks recommends standard access mode, which supports Python, SQL and Scala for multiple users with data isolation. Use dedicated access mode only when the workload hits a standard limitation, such as ML Runtime on GPU, RDD APIs, or R.

What about startup time and cost?

Pricing for Lakeflow Jobs is tied to the compute that runs the tasks. Serverless jobs have two performance modes. Performance optimized starts and runs faster for time-sensitive workloads. Standard consumes fewer DBUs and suits jobs that can tolerate a startup latency of four to six minutes. On classic compute, instance pools reserve compute from your cloud provider to shorten cluster start time, and sharing job compute across tasks can reduce start-up latency, although the compute stays on and idle while tasks that don't use it run.

What about streaming and machine learning?

Serverless notebooks and jobs support Structured Streaming with Trigger.AvailableNow. Time-based triggers such as Trigger.ProcessingTime aren't supported, so always-on streams belong in Lakeflow Pipelines in continuous mode. On classic compute, autoscaling has limits when scaling down for Structured Streaming, and Databricks recommends Lakeflow Pipelines with enhanced autoscaling for streaming workloads.
For machine learning, Databricks Runtime for ML and GPU-enabled compute aren't supported on standard compute. Select an ML runtime only if the workload uses GPUs, distributed ML training or AutoML.

FAQs

Can one job use more than one compute type?

Yes. You define compute for each task, and you can specify different compute types for tasks in a job, which may be required when a task type isn't supported on serverless.

Can I switch an existing job to serverless?

You can switch an existing job to serverless for supported task types when you edit it. Check compatibility first: serverless uses Spark Connect, which defers analysis and name resolution to execution time and may change how your code behaves, and ANSI mode is on by default.

Is Photon on by default?

Photon is enabled on serverless compute and SQL warehouses. Databricks also enables it by default on classic all-purpose and jobs compute, but if you create those resources with the Clusters API or Jobs API, you must set runtime_engine to PHOTON.

Which Databricks Runtime should I choose?

Serverless compute doesn't use Databricks Runtime versions. You write code against an environment version, and Databricks upgrades the underlying runtime. On classic compute, Databricks recommends the latest long-term support (LTS) version.