Why is my job stuck on the last task (skew)?

Summary

• A Spark job that hangs on its last task usually has data skew: a hot key sends far more rows to one shuffle partition, and a single task has to process that partition alone.
• Confirm it in the Spark UI first: on the stuck stage, check whether the slow task's Shuffle Read Size / Records dwarfs the others'. Normal input with high GC time or spill means memory pressure instead.
• Adaptive Query Execution (AQE), on by default since Spark 3.2.0, splits skewed sort-merge join partitions, but only those larger than both spark.sql.adaptive.skewJoin.skewedPartitionFactor (default 5.0) times the median and spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (default 256MB).
• Windows, grouped pandas functions and coalesce(1) writes fall outside the skew-join rule and need a code change, such as a finer partition key or a REBALANCE hint.
• Speculative execution doesn't rescue a skewed task, because the relaunched copy has to process the same oversized partition.

Why is my job stuck on the last task (skew)?

Because one partition holds a disproportionate share of the data. In Spark a single task operates on a single partition, and the job can't move past a stage until every task in it finishes, so when a hot key funnels most rows into one shuffle partition, that task runs long after its peers are done. The fix depends on which operator built the partition: AQE splits skewed sort-merge joins itself, but windows, grouped pandas functions and single-partition writes need code changes.

Confirm it's skew, not a lookalike

In the Spark UI, open Stages and click the stage that's still running. Its page shows summary metrics for all tasks plus a per-task table. Sort tasks by Shuffle Read Size / Records and compare the slow one with the rest; in the summary, skew shows up as a maximum duration far above the 75th percentile. Then match what you see:

What the stuck stage shows	Likely cause	Fix
One task's shuffle read dwarfs the rest, under a join	Hot join key	Let AQE split it (check its thresholds), or broadcast the small side
The same, under a window function or applyInPandas	Every row for one key must sit in one task	Partition by a finer key, or reduce rows before that step
The stage has exactly one task	coalesce(1), or a window with no PARTITION BY	Remove the single-partition step, or add a partition spec
Normal input, but high GC time or shuffle spill	Memory pressure, not skew	Smaller partitions or more memory per task
Normal input, and only one executor is slow	A slow or overloaded node	Turn on spark.speculation

To name the hot keys, count rows per key on the input side:

SELECT customer_id, COUNT(*) AS row_count
FROM orders
GROUP BY customer_id
ORDER BY row_count DESC
LIMIT 20

Join skew: check what AQE is doing

AQE has been on by default since Apache Spark 3.2.0, and its skew-join rule (spark.sql.adaptive.skewJoin.enabled) defaults to true. It splits, and replicates if needed, sort-merge join partitions that are larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (default 5.0) times the median partition size and also larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (default 256MB). A partition that misses either test is left alone, as is a plan where splitting would add a shuffle. Lower the thresholds, or force the rule:

spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "128MB")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "3")
spark.conf.set("spark.sql.adaptive.forceOptimizeSkewedJoin", "true")

The tuning guide advises keeping the byte threshold above spark.sql.adaptive.advisoryPartitionSizeInBytes (default 64 MB). If one side of the join is small, a BROADCAST hint suggests that Spark broadcast that side instead of shuffling both sides by the join key, so no hot shuffle partition forms.

Skew the join rule doesn't touch

Window functions. A window partitioned by customer_id needs every row for a customer in one partition, so one hot customer means one long task. A window with no partition spec is worse: Spark logs "No Partition Defined for Window operation! Moving all data to a single partition" and runs the whole window in one task. Partition by a finer key when the logic allows it, or aggregate first so the window sees fewer rows.
Grouped pandas functions. The PySpark docs warn that the pandas.DataFrame form of applyInPandas loads each group fully into memory, risking OOM when skewed groups don't fit; the iterator form mitigates this.
Writes. A drastic coalesce(1) can put the whole computation on one node; repartition() adds a shuffle but keeps upstream work parallel. To even out output files, use the REBALANCE hint. With AQE on, Spark splits skewed partitions under it (spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled, default true):

INSERT INTO daily_orders
SELECT /*+ REBALANCE(order_date) */ * FROM staged_orders

How Databricks approaches skew

Databricks enables AQE by default, and its docs state that skew handling covers both sort merge join and shuffle hash join. To confirm the rule fired, look in the query plan for a SortMergeJoin node with isSkew set to true. Setting spark.sql.shuffle.partitions to auto turns on auto-optimized shuffle, which sets the partition count from the query plan and input size. For diagnosis, Databricks publishes a Spark UI guide to skew and spill. See AQE on Databricks.

FAQs

Will speculative execution fix a skewed task?

No. spark.speculation (default false) relaunches tasks that run slowly, which helps when a node is the problem. The relaunched copy still has to process the same oversized partition.

Why doesn't raising spark.sql.shuffle.partitions help?

All values for a single key must be co-located to compute a join or aggregation, so every row of a hot key lands in one partition however many partitions you configure. Extra partitions only shrink the tasks that were already fast.

Can skew cause failures, not just slow tasks?

Yes. An oversized partition can exhaust execution memory and spill to disk, and grouped pandas functions can fail with OOM errors when one group doesn't fit in memory.

What does forceOptimizeSkewedJoin trade off?

It defaults to false. Setting it to true applies the skew-join rule even when that introduces an extra shuffle, so you accept the extra shuffle to avoid the straggler task.