What are the best practices for partitioning data in Spark?

Summary

• Spark partitions data in two places: in memory, where each partition becomes a task, and in storage, where partitionBy() writes one directory per value. Most advice applies to only one of them.
• Partition a table on disk only by a column that most queries filter on and that has few distinct values. The Delta Lake docs suggest partitioning only when each partition will hold at least 1 GB of data.
• For high-cardinality keys, use bucketing on persistent tables or Delta Lake liquid clustering instead of directory partitioning.
• Let adaptive query execution, on by default since Spark 3.2.0, size shuffle partitions. The Spark docs suggest turning parallelismFirst off on busy clusters.
• On Databricks, most tables don't need partitioning: liquid clustering is recommended for all managed tables, and predictive optimization handles file maintenance.

What are the best practices for partitioning data in Spark?

Partition data on storage only when one column is both low-cardinality and present in most query filters, leave in-memory partition counts to adaptive query execution, and keep the file count in check on every write. The catch is that "partition" means two different things in Spark, and a fix for one layer can hurt the other. Getting either layer wrong ends the same way: piles of small files and tiny tasks.

Two layers of partitioning

In-memory partitions are the units of parallel work. For file reads, spark.sql.files.maxPartitionBytes (default 128 MB) caps the bytes in one partition. For joins and aggregations, spark.sql.shuffle.partitions (default 200) sets the partition count. repartition() and coalesce() change it inside a job.
Storage partitions are directories. partitionBy("col") or PARTITIONED BY (col) writes Hive-style col=value folders, and filters on that column let queries skip folders that can't match. This layout outlives the job that wrote it, so it deserves more care.

Choosing a storage partition column

Criterion	What to look for
Filter usage	Most queries filter on the column. If they rarely do, partitioning only adds directories
Cardinality	Few distinct values. Spark's docs note that partitionBy has limited applicability to columns with high cardinality
Data per partition	The Delta Lake best-practices guide suggests partitioning only if you expect at least 1 GB of data in each partition
Write pattern	New data arrives by that column, such as one date per daily load, so writes touch few partitions
Column count	One column, occasionally two. Every extra column multiplies the directories
Granularity	A coarse value such as a date rather than a raw timestamp. Spark infers partition values as numeric, date, timestamp or string

The Delta Lake guide names date as the most commonly used partition column. A user ID is the classic mistake: a directory per user, each holding a sliver of data.

Keep in-memory partitions sized by the engine

Adaptive query execution has been on by default since Spark 3.2.0. With spark.sql.adaptive.coalescePartitions.enabled (also true by default), Spark merges contiguous small shuffle partitions at runtime, so you don't need a hand-tuned shuffle count for every dataset. One default surprises people: spark.sql.adaptive.coalescePartitions.parallelismFirst is true, so Spark ignores the advisory size (spark.sql.adaptive.advisoryPartitionSizeInBytes, 64 MB) and respects only the minimum partition size (1 MB by default) to maximize parallelism. The Spark docs recommend setting it to false on a busy cluster:

spark.conf.set("spark.sql.adaptive.coalescePartitions.parallelismFirst", "false")

AQE starts from spark.sql.adaptive.coalescePartitions.initialPartitionNum, which defaults to spark.sql.shuffle.partitions, so raise that if it's too low to coalesce from. To cut partitions without a shuffle, coalesce(n) creates a narrow dependency, but a drastic reduction can leave the work on very few nodes, and repartition(n) is the safer option then, at the cost of a shuffle.

Write partitioned data without small files

• Group rows before the write. repartition() on the partition column, or the SQL REBALANCE hint (it needs AQE), keeps each directory from receiving a file from every task.
• Cap file size by rows. spark.sql.files.maxRecordsPerFile splits very large partitions into several files. Its default, 0, sets no limit.
• Rewrite only what changed. Dynamic partition overwrite (partitionOverwriteMode set to dynamic) replaces just the partitions present in the new data.
• Compact on a schedule. The Delta Lake guide shows reading a table, repartitioning it and rewriting it with dataChange set to false, which rearranges files without changing the data.

When not to partition

Skip directory partitioning for small tables and for high-cardinality keys. For persistent tables, bucketBy() spreads rows across a fixed number of buckets and works when the number of unique values is unbounded. Delta Lake 3.1.0 and above offers liquid clustering (CLUSTER BY), which lets you redefine clustering columns without rewriting existing data; it can't be combined with partitioning on the same table. Apache Iceberg uses hidden partitioning: transforms such as day(ts) produce partition values automatically, readers don't add extra filters, and the layout can evolve.

How Databricks approaches partitioning

Databricks' partitioning guide says most tables with less than 100 TB of data don't need partitioning. It recommends liquid clustering for all managed tables, clusters unpartitioned Delta tables by ingestion time automatically, and suggests at least 1 GB per partition when you do partition. CLUSTER BY AUTO lets Databricks choose clustering keys for Unity Catalog managed tables from the query workload. Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, so compaction and statistics don't need their own jobs.

FAQs

How many partitions should a Spark job have?

There's no fixed number. Start from a large enough shuffle partition count, let adaptive query execution coalesce at runtime, and check task sizes in the Spark UI.

Should I partition by a timestamp column?

No. Derive a date and partition by that, or use a table-format transform such as Iceberg's day(ts). For Delta liquid clustering, the Delta docs recommend clustering on the original column instead of a generated, lower-cardinality one.

Is bucketing still worth using?

For persistent tables repeatedly joined or filtered on a high-cardinality key, yes. Spark's bucketing uses its own hash function and isn't compatible with Hive's bucketing.