How do I improve speed for my Spark job?

Summary

• Measure before you tune. On classic compute, the Spark UI stage detail page shows task duration, GC time, shuffle read size and spill. On serverless compute, where the Spark UI isn't available, and on SQL warehouses, use the query profile.
• Adaptive query execution (AQE) is on by default. It coalesces small shuffle partitions, can switch a sort-merge join to a broadcast join, and splits skewed partitions in sort-merge and shuffle hash joins, so check the plan before hand-tuning.
• Read less data. Liquid clustering is recommended for new tables, Delta data skipping uses per-file statistics to skip files, and auto compaction, optimized writes and OPTIMIZE address small files.
• Photon speeds up supported SQL and DataFrame work on longer-running queries. It does not support UDFs, RDD APIs or Dataset APIs, and it gives no meaningful gain to queries that finish in under two seconds.
• Change compute size last, and avoid hardcoding Spark settings such as spark.executor.memory, which Databricks says can override built-in optimizations.

How do I improve speed for my Spark job?
Find the slow stage first, work out why it's slow (a skewed join, small files, a large shuffle, spill to disk), and then change the one thing that addresses it. On Databricks some common fixes are already on: adaptive query execution (AQE) is enabled by default, and Photon is enabled on serverless compute. AQE applies to non-streaming queries that contain at least one exchange (usually a join, aggregate or window) or a subquery.

Where is the time going?

Tool	Where it works	What it shows
Spark UI	Classic compute, from the compute's Spark UI tab. Not available on serverless.	Stages and tasks, with task duration, GC time, shuffle read size and shuffle spill to memory and disk
Query profile	SQL warehouses and serverless compute	Execution details and metrics per query, plus the share of task time spent in Photon
EXPLAIN or DataFrame.explain()	In SQL or code	The query plan, in modes such as simple, extended, cost and formatted

Which bottleneck do you have?

Symptom	Likely cause	What the docs suggest
A few tasks in a join stage run far longer than the rest	Skewed join keys	Rely on AQE skew handling. By default a partition counts as skewed when it is larger than 5 times the median partition size and larger than 256 MB. It covers sort-merge and shuffle hash joins, never broadcast joins.
Many tiny shuffle tasks	Too many shuffle partitions	AQE coalesces small partitions after a shuffle. spark.sql.shuffle.partitions defaults to 200 on classic compute and to auto on serverless.
A large shuffle before a join with a small table	Sort-merge join	Hint a broadcast join. Apache Spark broadcasts tables up to 10 MB by default, and the AQE runtime threshold on Databricks defaults to 30 MB.
Slow scans with selective filters	Data layout	Use liquid clustering, which replaces partitioning and ZORDER. Data skipping uses per-file minimum, maximum and null-count statistics.
Many small files	Frequent small writes	Use auto compaction and optimized writes, or OPTIMIZE. Predictive optimization runs OPTIMIZE for managed tables.
Spill to disk or out-of-memory errors	Instances short on memory	On classic compute, add memory to the instances.

from pyspark.sql.functions import broadcast

result = big_df.join(broadcast(small_df), "key")
result.explain(mode="formatted")

A broadcast hint still earns its place when AQE is on. A statically planned broadcast join usually performs better than one AQE plans at runtime, because AQE might not switch until after it shuffles both sides.

How do Photon and serverless compute fit?

Photon is the Databricks-native vectorized engine for SQL, DataFrame calls, ETL pipelines and stateless streaming. It helps most on longer-running queries over large datasets. It doesn't support UDFs, RDD APIs or Dataset APIs, and it falls back to the Spark runtime for those operations. Serverless compute for jobs enables Photon and autoscaling automatically. The Spark UI isn't available there, so use the query profile, and df.cache() and df.persist() raise exceptions.

How should I size compute?

Size last. Databricks sizing guidance looks at total executor cores, which set the maximum parallelism, total executor memory, and executor local storage, which is used mainly for spills and caching. If you see significant spill to disk or out-of-memory errors, increase the memory available on your instances. Databricks generally recommends against configuring most Spark properties because legacy settings can override newer defaults that optimize workloads. For a manual starting point, the Spark tuning guide recommends two to three tasks per CPU core.

FAQs

Is repartition(N) always a good idea?

No. repartition hash-partitions the data and adds a shuffle step. coalesce narrows partitions without a shuffle, but a drastic coalesce, such as to one partition, can leave the work on fewer nodes.

Does caching help?

Databricks caches remote Parquet data files, including Delta tables, on local storage automatically on first read, and recommends this disk cache, which is enabled by default on certain node types. The Apache Spark cache needs code changes. On serverless compute, DataFrame cache and persist calls aren't supported.

Why did my job slow down without code changes?

Compare the slow run with an earlier one in the Spark UI or query profile. AQE helps when statistics are missing or stale, and predictive optimization collects statistics when data is written to a managed table.