How do I read the Spark UI to find a bottleneck?

Summary

• Read the Spark UI top down: the Jobs tab shows which job dominates the run, the job page shows which stage inside it, and the stage page shows why that stage is slow.
• On the stage page, the summary metrics table is the diagnostic. A maximum task duration far above the 75th percentile means skew, non-zero shuffle spill means tasks ran short of execution memory, and high GC time points at JVM memory pressure.
• The SQL tab ties a slow stage back to your query, with metrics such as number of output rows, spill size and shuffle bytes written on each operator.
• Gaps in the jobs timeline mean no Spark work was scheduled: the cluster sat idle between queries, or the driver was busy building a plan or running non-Spark code.
• The live UI disappears when the application ends; event logs plus the History Server let you inspect finished jobs.

How do I read the Spark UI to find a bottleneck?

Work from the outside in: application, job, stage, task, operator. The Jobs tab shows where the time went, the longest stage shows what kind of work it was, and that stage's task metrics show why it was slow: skew, spill, garbage collection or shuffle waits. The SQL tab then names the operator responsible.
While an application runs, the driver serves its UI at http://<driver-host>:4040 by default (spark.ui.port changes it). It goes away when the application exits. To read it afterward, set spark.eventLog.enabled to true and spark.eventLog.dir to a shared directory before the job starts, then run ./sbin/start-history-server.sh with spark.history.fs.logDirectory pointing at the same place. It listens on port 18080 by default.

Step 1: Find the expensive job in the Jobs tab

The event timeline lays out jobs and executor events in time order. A long bar is your target job, so open it. Gaps where no job runs mean no Spark work was scheduled. Databricks' Spark UI guide lists the usual reasons: no queries submitted, the driver building a complex plan (for example, withColumn() called in a loop), plain Python or other non-Spark code, or an overloaded driver.

Step 2: Open the longest stage

On the job page, sort the stage list by duration. The Input, Output, Shuffle read and Shuffle write columns show what the stage did: reading storage, writing results, or exchanging data for a join, aggregation or repartition. Check the task count too. A long stage with one task can leave the rest of the cluster idle; common causes include a window function without PARTITION BY, an unsplittable file such as gzip, and repartition(1) or coalesce(1).

Step 3: Read the stage's summary metrics

The stage page shows summary metrics for all tasks, the same metrics per executor, and a per-task table.

Symptom on the stage page	Likely cause	Fix to try first
Maximum task duration far above the 75th percentile	Data skew: a few tasks process an outsized share of the data	Keep AQE skew-join handling on (spark.sql.adaptive.skewJoin.enabled), which splits skewed partitions in sort-merge joins
Shuffle spill (memory) and Shuffle spill (disk) above zero	Tasks ran low on execution memory during shuffles, joins, sorts or aggregations	Raise parallelism so each task's input set is smaller
High GC time	JVM garbage collection pressure	Measure GC first, then cut object churn or cached data, as the tuning guide describes
High Shuffle Read Fetch Wait Time	Tasks blocked waiting for shuffle data from remote executors	Shrink or avoid the shuffle, for example by broadcasting the small side of a join
High Scheduler delay on short tasks	Many tiny tasks, each paying scheduling overhead	Let AQE coalesce post-shuffle partitions

Step 4: Map the stage to an operator in the SQL tab

Every DataFrame action appears in the SQL tab. The plan graph shows each operator with its metrics inline, and stage DAGs carry operation-scope labels such as WholeStageCodegen and Exchange, so you can match a stage to its query. An output row count that jumps after a join signals an exploding join. Large shuffle bytes written on an Exchange marks an expensive shuffle. Spill size and peak memory on Sort or HashAggregate confirm memory pressure, and data sent to Python workers measures Python UDF serialization. The Details link expands the parsed, analyzed, optimized and physical plans; with AQE, runtime statistics appear as Statistics(..., isRuntime=true).

Step 5: Check executors and configuration

The Executors tab lists every executor with its memory, disk, cores, task counts, shuffle totals and GC time, plus a Thread Dump link with live thread data for checking a task that looks stuck. The Environment tab is where you verify the application is running with the configuration you expect, for example that a spark.sql.shuffle.partitions change took effect.

How Databricks approaches Spark UI debugging

On classic Databricks compute, open the Spark UI from the compute page; Databricks' step-by-step guide follows the same order as above, from the jobs timeline to the longest stage, skew or spill, and I/O-bound stages. Serverless compute doesn't expose the Spark UI. There, and on SQL warehouses, the query profile shows each operator's time spent, rows and memory, with a Top operators panel listing the longest running ones.

FAQs

How do I tell skew from a uniformly slow stage?

Compare the maximum task duration with the 75th percentile in the stage's summary metrics. If they're close, every task is slow and the cause is the work itself (I/O, spill or GC); if the maximum is far higher, a few partitions are oversized.

Where do Python UDF costs show up?

In the SQL tab, Python UDF operators report data sent to Python workers and data returned from Python workers, the bytes serialized between the JVM and Python.

How can I tell whether AQE changed the plan?

AQE plans have an AdaptiveSparkPlan root whose isFinalPlan flag turns true when the query completes. Compare the final plan's join and shuffle nodes with the initial plan's.