How do I debug a slow Spark stage in the Spark UI?

Summary

• Open the slow stage's detail page and classify the problem before changing any configuration.
• Compare the maximum task duration with the 75th percentile in the summary metrics. A long tail means skew; uniformly slow tasks point to parallelism, memory, I/O or expensive operators.
• Read where task time goes: GC time, task deserialization time, shuffle read fetch wait time and spill each point to a different fix.
• Map the stage to its operators through the stage DAG and the SQL tab, whose per-operator metrics (number of output rows, spill size, peak memory) show which part of the query is slow.

How do I debug a slow Spark stage in the Spark UI?

Open the stage's detail page and read it in a fixed order. First, the task duration quantiles show whether a few tasks or all of them are slow. Then the time breakdown shows where task time goes, the executor table shows whether one executor is the problem, and the stage DAG ties the stage to the operators in your query. Classify the slowness before you change any configuration.

Find the stage worth debugging

The web UI runs at http://<driver-host>:4040 while the application is alive. On the Jobs tab, open the slow job and sort its stages by duration. The stage list's Input, Output, Shuffle read and Shuffle write columns show whether the stage mostly reads, writes or shuffles. Check the task count too: a long stage with a single task is a warning sign (see the last FAQ). Click the stage's description to open its detail page.

Read the stage detail page in this order

1. Summary metrics for all tasks. If the maximum duration sits far above the 75th percentile, a few tasks take much longer than the rest, which is skew. When the quantiles are close together, the cause is uniform.
2. Time breakdown. The same table reports GC time, scheduler delay (time a task waits to be scheduled), task deserialization time, Shuffle Read Fetch Wait Time (time blocked on remote shuffle data), Shuffle Write Time, result serialization time and getting result time. Find the metric that dominates task duration.
3. Memory. Shuffle spill (memory) and Shuffle spill (disk) show whether tasks ran low on execution memory; peak execution memory is the most that shuffle, aggregation and join data structures used.
4. Aggregated metrics by executor. The same metrics per executor show whether slow tasks cluster on one executor. The locality level summary shows how close tasks ran to their data.
5. Tasks table and the Executors tab. The tasks table lists each task, with attempt numbers for failures and executor log links. While the application runs, the Executors tab opens each executor's live thread dump, heap histogram and flame graph.
6. The stage DAG. Nodes are labelled with their operation scope (BatchScan, WholeStageCodegen, Exchange), and codegen IDs let you cross-reference the SQL tab, where each operator shows metrics such as number of output rows, spill size, peak memory and data sent to Python workers.

# Print the physical plan with node details to compare against the SQL tab graph
df.explain(mode="formatted")

Match the pattern to the fix

Symptom on the stage page	Likely cause	Fix
Maximum duration far above the 75th percentile	Skewed join or grouping key	For sort-merge and shuffled hash joins keep AQE skew handling on (spark.sql.adaptive.skewJoin.enabled, default true); otherwise change the key distribution
Spill on many tasks	Partitions too large for execution memory	Raise spark.sql.shuffle.partitions (default 200) so each task's input is smaller; AQE uses it as the initial partition count unless you set one
High GC time	Object churn, or cached data crowding the heap	Use data structures with fewer objects, cache in serialized form, or cache less by lowering spark.memory.fraction
High Shuffle Read Fetch Wait Time	Tasks blocked waiting for shuffle data from remote machines	For a join with a small side, keep AQE on so it can switch to a broadcast join and read shuffle files locally (spark.sql.adaptive.localShuffleReader.enabled, default true)
High task deserialization time	Large objects captured in task closures	Turn large read-only lookups into broadcast variables
Slow stage with little I/O	Slow UDFs, exploding or Cartesian joins, many small files	Check operator metrics in the SQL tab; replace UDFs with built-in functions; compact files

How Databricks approaches slow stages

On Databricks, open the Spark UI from the compute's Spark UI tab. Databricks publishes a step-by-step Spark UI diagnosis guide that follows the same path: jobs timeline, longest stage, skew and spill, I/O, then other causes. Databricks recommends setting spark.sql.shuffle.partitions to auto so Spark picks the shuffle partition count, and Photon can speed up read-heavy and write-heavy stages. For queries on SQL warehouses, the query profile shows per-operator time, rows and memory, plus a list of the longest-running operators.

FAQs

How do I see the Spark UI after the application ends?

Set spark.eventLog.enabled to true (and spark.eventLog.dir) before the application starts, then run ./sbin/start-history-server.sh. The History Server rebuilds the UI from the event log at http://<server-url>:18080 by default.

Why do tasks spill when the cluster has free memory?

Memory pressure follows each task's working set, not total cluster memory. The Spark tuning guide notes that one task's working set, such as a reduce task in groupByKey, can be too large, and its simplest fix is more parallelism so each task's input set is smaller.

Should I raise spark.sql.shuffle.partitions?

Only when spill or oversized tasks show partitions are too big. With AQE coalescing on, the Spark docs say you don't need a shuffle partition number that fits your dataset: set a large enough spark.sql.adaptive.coalescePartitions.initialPartitionNum and Spark picks the count at runtime.

What does a single-task stage mean?

Only one CPU is working while the rest of the cluster may sit idle. Look for an unsplittable file such as gzip, multiLine JSON or CSV reads, schema inference over a large file, a window function without PARTITION BY, or an explicit repartition(1) or coalesce(1).