repartition versus coalesce: which do I use?

Summary

• Use coalesce(n) to reduce partitions without a shuffle: it merges existing partitions through a narrow dependency.
• Use repartition to increase partitions, rebalance data, or hash-partition rows by columns; it always shuffles.
• coalesce never adds partitions: if you ask for more than you have, the count stays the same.
• For a drastic cut such as one partition, repartition(1) adds a shuffle but keeps upstream work parallel, while coalesce(1) can run it on one node.
• Adaptive query execution, on by default since Spark 3.2.0, also coalesces small post-shuffle partitions by itself.

repartition versus coalesce: which do I use?

Use coalesce when you only need fewer partitions, and repartition when you need more partitions, a fresh spread of rows, or rows grouped by key. coalesce merges existing partitions through a narrow dependency: going from 1000 partitions to 100, "there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions." repartition "always shuffles all data over the network," which means disk I/O, serialization and network I/O, but it returns exactly the partition count you ask for.

How do repartition and coalesce differ?

Aspect	coalesce	repartition
Shuffle	No; it "results in a narrow dependency"	Yes; it "always shuffles all data over the network"
Increasing partitions	Not possible; the count "will stay at the current number of partitions"	Returns exactly the number you request
Arguments	numPartitions only	numPartitions, columns, or both
Data placement	Each new partition claims several existing ones	Rows are reshuffled; with columns, the result "is hash partitioned"
Count when only columns are given	Not applicable	spark.sql.shuffle.partitions (default 200)
Reducing to one partition	"may result in your computation taking place on fewer nodes than you like"	Adds a shuffle step, but upstream partitions still run in parallel
SQL equivalent	/*+ COALESCE(3) */ hint	/*+ REPARTITION(3, c) */ hint, or DISTRIBUTE BY

When should I use coalesce?

• After a selective filter. The RDD guide calls coalesce "useful for running operations more efficiently after filtering down a large dataset."
• To write fewer output files, as long as the partitions you keep still give enough parallelism. Spark's partitioning hints exist partly to "control the number of output files."
• Not for a drastic cut on top of heavy work. coalesce(1) can pull the upstream computation onto one node. The docs' answer is repartition(1), which "will add a shuffle step, but means the current upstream partitions will be executed in parallel."

When should I use repartition?

• To raise parallelism, because coalesce can't add partitions.
• To group rows by key. df.repartition(7, "id") hash-partitions on id, "the same operation as "DISTRIBUTE BY" in SQL."
• To partition by ranges. repartitionByRange returns a range-partitioned result. It samples the data to estimate ranges, so "the output may not be consistent," and rows aren't sorted within each partition.
• Skew is a separate problem. For skewed sort-merge joins, adaptive query execution "dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks."

How do I write each one?

from pyspark.sql import functions as sf

df = spark.range(0, 64, 1, 9)    # 9 partitions

df.coalesce(3)                   # 3 partitions, no shuffle
df.coalesce(20)                  # still 9: coalesce never adds partitions
df.repartition(20)               # 20 partitions, full shuffle
df.repartition(7, "id")          # 7 partitions, hash partitioned by id
df.repartition(sf.col("id"))     # count comes from spark.sql.shuffle.partitions

# One output file without collapsing upstream parallelism
df.repartition(1).write.format("parquet").save("/tmp/single_file")

In SQL, the same choices are hints: SELECT /*+ COALESCE(3) */ * FROM t or SELECT /*+ REPARTITION(3, c) */ * FROM t.

How Databricks approaches partitioning

On Databricks, adaptive query execution is enabled by default and "dynamically coalesces partitions (combine small partitions into reasonably sized partitions) after shuffle exchange." Setting spark.sql.shuffle.partitions to auto turns on auto-optimized shuffle, and auto is the default on serverless compute. For writes, optimized writes "improve file size as data is written," and if you use them, Databricks recommends that you "don't run coalesce(n) or repartition(n) just before a write to control the number of files written." Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables.

FAQs

Does coalesce ever shuffle?

Not on a DataFrame, where coalesce creates a narrow dependency. On an RDD, coalesce(numPartitions, shuffle=False) takes a shuffle flag, and setting it to True adds a shuffle step. That's why the RDD guide lists coalesce among operations that "can cause a shuffle."

Can I repartition on multiple columns?

Yes. df.repartition(3, "name", "age") hash-partitions on both columns into three partitions, and numPartitions is optional when you pass columns.

Is the writer's partitionBy the same thing?

No. partitionBy on the writer "creates a directory structure" in storage, so it "has limited applicability to columns with high cardinality." repartition changes how rows are split across tasks before the write.

Do I still need either with adaptive query execution?

Less often. When spark.sql.adaptive.enabled and spark.sql.adaptive.coalescePartitions.enabled are true (both default to true), Spark merges small post-shuffle partitions, so "you do not need to set a proper shuffle partition number to fit your dataset." By default spark.sql.adaptive.coalescePartitions.parallelismFirst is true, which ignores the 64MB advisory size and respects only the 1MB minimum. For evenly sized output files, the REBALANCE hint asks for partitions "not too small and not too big," and it's ignored when AQE is off.