What cluster should I use for ETL?

Summary

• For most new ETL workloads, Databricks recommends serverless compute, which requires no configuration. The recommended compute depends on the task type: serverless jobs for notebooks, Python scripts and Python wheels, a serverless SQL warehouse for SQL and dbt tasks, and a serverless pipeline for Lakeflow Pipelines.
• Use classic compute only when a task or feature rules serverless out. Databricks lists classic jobs compute as the recommended option for JAR and Spark Submit tasks, and classic compute is where init scripts, instance pools and custom Spark configurations are available.
• Size a classic cluster by total executor cores, total executor memory and local storage, not just worker count. Simple batch ETL runs on instances with lower memory and storage requirements, while complex joins and unions call for fewer, larger workers that shuffle less data.
• Don't run production ETL on all-purpose compute. Databricks does not recommend it for production jobs, so use job compute or a serverless option instead.

What cluster should I use for ETL?

Start with serverless and move to classic compute only when a task needs something serverless doesn't offer. Databricks recommends serverless compute for most new workloads, and its Lakeflow Jobs documentation names a recommended option for each task type. When classic compute is required, run production ETL on job compute, not all-purpose compute.

Which compute does each ETL task need?

ETL task	Recommended compute	Other supported compute
Notebook, Python script, Python wheel	Serverless jobs	Classic jobs, classic all-purpose
SQL	Serverless SQL warehouse	Pro SQL warehouse
dbt	Serverless SQL warehouse	Pro SQL warehouse
Lakeflow Pipelines	Serverless pipeline	Classic pipeline
JAR	Classic jobs	Classic all-purpose
Spark Submit	Classic jobs	None

The table follows the compute recommendations in the Lakeflow Jobs documentation. Serverless compute for jobs is the default for every task that supports it. The serverless jobs page also lists JAR among the task types serverless supports, so check the current docs for JAR workloads.

When does ETL need classic compute?

Serverless compute has documented limits, so check them before you commit:

• APIs and languages: only Spark Connect APIs are supported. R and Spark RDD APIs are not.
• Compute features: compute policies, compute-scoped init scripts, instance pools and most Spark configurations are not supported.
• Streaming: serverless notebooks and jobs support Structured Streaming with Trigger.AvailableNow, not time-based triggers. For always-on streams, use Lakeflow Pipelines in continuous mode.
• Pipelines: Databricks says to use classic pipeline compute only when serverless cannot run the pipeline, such as with the legacy Hive metastore, unsupported private networking, or a region without serverless.

How do I size a classic ETL cluster?

Databricks sizing guidance looks at three things. Total executor cores set the maximum parallelism. Total executor memory sets how much data fits in memory before spilling to disk. Executor local storage is used mainly for spills during shuffles and caching. Two workers with 16 cores and 128 GB of RAM each have the same compute and memory as eight workers with 4 cores and 32 GB each.

• Basic batch ETL: for jobs without wide transformations such as joins or aggregations, use instances with lower memory and storage requirements, which might cost less.
• Complex batch ETL: for unions and joins across multiple tables, Databricks recommends fewer workers to reduce the amount of data shuffled, with larger instances to compensate. If you see significant spill to disk or out-of-memory errors, add memory to the instances.
• Scaling and cost: configure autoscaling so long-running tasks can add and remove workers, use spot instances when latency requirements are lax, and consider instance pools to shorten launch times.
• Spark settings: avoid hardcoding properties such as spark.executor.memory. Databricks says hardcoded values override built-in optimizations and often lead to wasted spend or degraded performance.
• Photon: it helps most with joins, aggregations and scans of large tables. Simple batch ETL may see minimal impact.

How do Lakeflow Pipelines and SQL warehouses fit?

New Lakeflow pipelines default to serverless. On serverless, Databricks says there are no clusters to size, configure, secure, or grant permissions for, and you get capabilities classic compute cannot offer, such as incremental refresh and vertical autoscaling. A serverless SQL warehouse performs well with ETL, business intelligence and exploratory analysis. A pro SQL warehouse fits when serverless warehouses aren't available in a region or you need custom-defined networking.

FAQs

Can I run production ETL on all-purpose compute?

You can configure tasks that way, but Databricks does not recommend all-purpose compute for production jobs.

Do I have to size a cluster for Lakeflow Pipelines?

Not on serverless. Databricks says serverless pipelines have no clusters to size or configure.

What if an ETL job runs out of memory on classic compute?

Databricks says to increase the memory available on your instances when you see significant spill to disk or out-of-memory errors. The stage detail page in the Spark UI reports shuffle spill to memory and disk, which helps confirm the cause.

Can streaming ETL run on serverless compute?

Yes, within limits. Serverless notebooks and jobs support Trigger.AvailableNow and reject time-based triggers. Lakeflow Pipelines in continuous mode on serverless covers always-on streams.