How should I lay out partitions in an Iceberg table?

Summary

• Partition on the columns your queries filter by, using Iceberg's hidden partitioning transforms: year, month, day or hour for timestamps, bucket(N, col) to hash a key into N buckets, truncate(L, col) for prefixes and ranges, and plain columns (identity) for categories.
• Iceberg derives partition values from the source column and skips files automatically; nobody maintains a separate partition column.
• Avoid grains that leave many small files: a data file can't span a partition boundary, and small files add metadata and file-open cost. The rewrite_data_files procedure compacts them.
• Layouts evolve in place with ALTER TABLE ... ADD PARTITION FIELD and related commands, a metadata-only change; existing data keeps its old layout.
• On Databricks, managed Iceberg tables don't support expression transforms such as bucket(), and their partition columns act as liquid clustering keys.

How should I lay out partitions in an Iceberg table?

Partition on the columns your queries filter by, and let Iceberg's hidden partitioning compute the partition values. The Iceberg docs' own example is a logs table whose queries usually include a time range: partition it by day(event_time) and the categorical level column, and Iceberg uses the date to skip files for other dates. Match the grain to your filters, avoid layouts that split writes into many small files, and evolve the layout later if needs change; evolution doesn't rewrite existing data.

Which partition transforms does Iceberg support?

Transform in Spark DDL	What it produces	Use it when queries filter on
A column name (identity)	The source value, unmodified	A category, like level in the docs' logs example
year(ts), month(ts)	Years or months from 1970	Years or months
day(ts) or date(ts)	A date	Days or date ranges
hour(ts) or date_hour(ts)	Hours from 1970-01-01 00:00:00 (timestamps only)	Hours
bucket(N, col)	A hash of the value, mod N	A key with many distinct values, such as an ID
truncate(L, col)	Strings cut to length L; integers and longs cut into bins, so truncate(10, i) gives 0, 10, 20	Prefixes or numeric ranges

The spec also defines a void transform that always produces null; it can be used to drop a partition field in v1 tables. The older forms years(ts), months(ts), days(ts) and hours(ts) still work for compatibility.

How do you create a partitioned table?

CREATE TABLE prod.db.sample (
    id bigint,
    data string,
    category string,
    ts timestamp)
USING iceberg
PARTITIONED BY (day(ts), category);

Queries then filter on ts and category directly. There's no derived date column to populate, which removes the Hive-style failure mode where a wrong format or source column "produces silently incorrect results, not query failures". By default, Iceberg's Spark writers also request a hash shuffle on the partition value before writing (write.distribution-mode is hash, the default since Iceberg 1.2.0), so rows for the same partition land in the same Spark task.

How do you avoid too many small files?

A data file can't span a partition boundary, and a Spark task that writes to several partitions produces even smaller files. So a grain finer than your data volume supports, such as hour on a table that receives little data each hour, leaves many small files. The docs say small files cause "an unnecessary amount of metadata and less efficient queries from file open costs". Two fixes:

• Choose a coarser transform, such as day instead of hour, if your filters allow it.
• Compact with the rewrite_data_files procedure, which bin-packs small files by default.

How do you change the layout later?

Partition evolution is "a metadata operation and does not eagerly rewrite files". Old data keeps its earlier spec, new data uses the new one, and queries get split planning across both layouts. In Spark, the ALTER TABLE ... PARTITION FIELD commands come from the Iceberg SQL extensions; CALL procedures need the extensions on Spark 3.x but run natively on Spark 4.0:

ALTER TABLE prod.db.sample ADD PARTITION FIELD bucket(16, id);
SELECT * FROM prod.db.sample.partitions;
CALL catalog_name.system.rewrite_data_files('db.sample');

• DROP PARTITION FIELD removes a field, and REPLACE PARTITION FIELD ... WITH ... swaps one in a single metadata update.
• Moving from daily to hourly, you don't need to drop the daily field; keeping it keeps existing metadata-table queries working.
• Dynamic partition overwrite behavior changes when partitioning changes. To overwrite explicitly, use the DataFrameWriterV2 API.
• The partitions metadata table reports record_count, file_count and total_data_file_size_in_bytes per partition, so you can check how evenly data lands.
• rewrite_data_files writes to the current partition spec by default (its output-spec-id option), so files it rewrites land in the new layout. Its rewrite-all option forces every file to be rewritten.

How Databricks approaches this

Iceberg tables you create in Unity Catalog are managed Iceberg tables. Specify USING iceberg, or Databricks creates a Delta Lake table by default. The Databricks docs list layout differences that matter here:

• Expression-based partition transforms such as years() and bucket() aren't supported for managed Iceberg tables, so partition them by plain columns.
• Managed Iceberg tables use partition definitions as liquid clustering keys.
• Partition evolution works through external Iceberg engines using the Iceberg REST Catalog, not through Databricks SQL.
• Partitioning by BINARY type isn't supported.

Unity Catalog manages lifecycle tasks such as snapshot expiration and file compaction on managed Iceberg tables, and Databricks recommends predictive optimization for all Unity Catalog managed tables.

FAQs

Should I add my own date column to partition by?

No. Iceberg produces partition values from the source column, for example with day(ts), and the docs note that producers and consumers "wouldn't even see" the derived value.

Can I change the number of buckets later?

Yes. The Iceberg spec says changing the number of buckets as a table grows is possible by evolving the partition spec.

Does evolving the partition spec rewrite my data?

No. It's a metadata operation. Run rewrite_data_files when you want existing files in the new layout.

How do I see the current layout?

Query the partitions metadata table, for example SELECT * FROM prod.db.sample.partitions. For unpartitioned tables, it has no partition or spec_id fields.