How do I choose between repartition and coalesce?

Summary

• Use coalesce(n) to reduce the number of partitions. On a DataFrame it results in a narrow dependency: going from 1000 to 100 partitions involves no shuffle, because each new partition claims 10 existing ones.
• Use repartition(n) to increase partitions or to redistribute data. It uses a shuffle, and the RDD guide says it "always shuffles all data over the network".
• A DataFrame coalesce never raises the partition count, and a drastic coalesce (for example to one partition) can leave the computation on fewer nodes. The docs suggest repartition() in that case.
• On Databricks, if you use optimized writes, don't run coalesce(n) or repartition(n) just before a write to control the number of files.

How do I choose between repartition and coalesce?

Use coalesce to reduce the number of partitions, for example after filtering down a large dataset. Use repartition when you need more partitions or want the data redistributed. The RDD programming guide says "Spark will run one task for each partition of the cluster," and it describes coalesce as "Useful for running operations more efficiently after filtering down a large dataset" and repartition as reshuffling "the data in the RDD randomly to create either more or fewer partitions and balance it across them."

How do they compare?

Aspect	coalesce(n)	repartition(n)
Shuffle	A narrow dependency on a DataFrame, so no shuffle	Uses a shuffle to redistribute data
Raising the partition count	No: a larger request "will stay at the current number of partitions"	Yes: "Can increase or decrease the level of parallelism"
Partitioning columns	None	Optional; with columns the result is hash partitioned
Drastic reduction, such as to 1	The computation may run on fewer nodes than you like	Adds a shuffle step, but upstream partitions run in parallel
shuffle argument	Only on RDD.coalesce(numPartitions, shuffle=False)	Not applicable

How does coalesce behave?

DataFrame.coalesce(numPartitions) "results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions." With a drastic coalesce, such as numPartitions = 1, "this may result in your computation taking place on fewer nodes than you like." The docs suggest calling repartition() instead, which adds a shuffle step "but means the current upstream partitions will be executed in parallel."
DataFrame.coalesce has no shuffle argument. The RDD version does: RDD.coalesce(numPartitions, shuffle=False), where shuffle means "whether to add a shuffle step."

How does repartition behave?

RDD.repartition(numPartitions) returns an RDD with exactly that many partitions. The docs add: "Internally, this uses a shuffle to redistribute data." On a DataFrame, repartition(numPartitions, *cols) accepts partitioning columns, and the docs say: "The resulting DataFrame is hash partitioned."

from pyspark.sql import functions as sf

df = spark.range(0, 64, 1, 9).withColumn("age", sf.col("id") - 32)

df.coalesce(3)            # 9 partitions down to 3, no shuffle
df.repartition(10)        # exactly 10 partitions, with a shuffle
df.repartition(7, "age")  # 7 partitions, hash partitioned by age

df.repartition(10).select(
    sf.spark_partition_id().alias("partition")
).distinct().sort("partition").show()

How do I fix skewed data or too many small output files?

For a result with reasonably sized partitions, Spark SQL offers a REBALANCE hint. It is best effort: "if there are skews, Spark will split the skewed partitions," and it is useful when you write a query result to a table "to avoid too small/big files." The hint is ignored if adaptive query execution (AQE) is not enabled. The COALESCE and REPARTITION hints control the number of output files in the same way the Dataset methods do.

SELECT /*+ COALESCE(3) */ * FROM t;
SELECT /*+ REPARTITION(3) */ * FROM t;
SELECT /*+ REBALANCE */ * FROM t;

Spark also coalesces shuffle partitions on its own. With spark.sql.adaptive.enabled and spark.sql.adaptive.coalescePartitions.enabled both true (the defaults), it merges contiguous shuffle partitions to avoid too many small tasks. spark.sql.shuffle.partitions defaults to 200.

What does Databricks recommend?

Delta optimized writes "improve file size as data is written." They are enabled by default for MERGE, UPDATE with subqueries and DELETE with subqueries, plus CTAS and INSERT on SQL warehouses. The Databricks docs say: "If you use optimized writes, Databricks recommends that you don't run coalesce(n) or repartition(n) just before a write to control the number of files written." Writes can still take longer because data is shuffled before it is written.
Standard compute and serverless compute don't support RDD APIs, so df.rdd.getNumPartitions() is not available there. Use spark_partition_id(), as in the example above, to inspect partitions.

FAQs

Should I always use coalesce to reduce partitions?

No. A drastic coalesce can concentrate the computation on fewer nodes, and the docs point to repartition() for that case.

What happens if I call coalesce with a number above the current partition count?

The docs say it "will stay at the current number of partitions." Use repartition to increase the count.

How many partitions should I aim for?

The RDD guide says "Typically you want 2-4 partitions for each CPU in your cluster. Normally, Spark tries to set the number of partitions automatically based on your cluster."

How do I check how many partitions a DataFrame has?

Where RDD APIs are available, df.rdd.getNumPartitions() returns the count. Otherwise, select spark_partition_id() and take the distinct values, as in the example above.