How do I fix small-file problems in an Iceberg table?

Summary

• Compact small files with Iceberg's rewrite_data_files Spark procedure, which combines small files into larger ones to cut metadata overhead and file open cost.
• Control which files are rewritten with the options map (for example target-file-size-bytes and min-input-files) and the where argument.
• Compaction commits a new snapshot, so run expire_snapshots afterward to remove the old files.
• Reduce new small files at the source with a longer streaming trigger interval and a write distribution mode.
• On Databricks, predictive optimization runs OPTIMIZE and VACUUM on Unity Catalog managed tables, and Unity Catalog manages compaction for managed Iceberg tables.

How do I fix small-file problems in an Iceberg table?

Compact the small files with Iceberg's rewrite_data_files Spark procedure, then expire old snapshots and fix the write pattern that created the small files. The Iceberg docs explain the cost: more data files means more metadata in manifest files, and small data files cause an unnecessary amount of metadata and less efficient queries from file open costs. Compaction (the rewriteDataFiles action in the Java API) combines small files into larger ones to reduce metadata overhead and runtime file open cost.

Why do small files pile up in an Iceberg table?

Cause	What happens	What to do
Frequent streaming commits	Data written by a streaming process is typically small, and every batch also produces a snapshot	Use a longer trigger interval (the docs recommend 1 minute at the minimum) and maintain the table regularly
Many partitions per Spark task	Spark cannot write a file larger than a Spark task, and a file cannot span an Iceberg partition boundary	Use a write distribution mode or repartition the data before writing
Row-level deletes	Tables with position delete files can hold small delete files too	Run rewrite_position_delete_files

For Spark writes, write.distribution-mode accepts none (no shuffle), hash (the default) and range (the default when a table is created with a sort order).

How do I compact with rewrite_data_files?

Call the procedure through your Iceberg catalog's system namespace. On Spark 3.x, stored procedures need the Iceberg SQL extensions. Spark 4.0 supports them natively, and procedure names are case-sensitive there.

-- Compact with defaults (the binpack strategy)
CALL my_catalog.system.rewrite_data_files('db.sample');

-- Pass options and limit which files are considered
CALL my_catalog.system.rewrite_data_files(
  table => 'db.sample',
  options => map('min-input-files', '2'),
  where => 'event_date = "2026-10-01"'
);

The arguments are table, strategy (binpack by default, or sort), sort_order, options and where. Option names use hyphens and go inside the options map. The where predicate selects every file that may contain matching rows.

Option	Default	What it does
target-file-size-bytes	536870912 (512 MB)	Target output file size
min-file-size-bytes	Three quarters of the target	Files under this size are rewrite candidates regardless of other criteria
max-file-size-bytes	Set above the target	Files over this size are rewrite candidates too
min-input-files	5	A file group with this many files or more is rewritten regardless of other criteria
max-concurrent-file-group-rewrites	5	Maximum number of file groups rewritten at the same time
partial-progress.enabled	false	Commit groups of files before the whole rewrite completes
rewrite-all	false	Rewrite all provided files, overriding the other options

How do I check that compaction worked?

The procedure returns rewritten_data_files_count, added_data_files_count, rewritten_bytes_count, failed_data_files_count and removed_delete_files_count. For before-and-after numbers, query the partitions metadata table. Use snapshots, not history, to see the commit:

SELECT partition, file_count, total_data_file_size_in_bytes
FROM my_catalog.db.sample.partitions;

SELECT committed_at, operation, summary
FROM my_catalog.db.sample.snapshots
ORDER BY committed_at DESC;

The Iceberg spec defines the replace operation as data and delete files added and removed without changing table data, and names compaction as an example. The history table has only made_current_at, snapshot_id, parent_id and is_current_ancestor.

What should I run after compaction?

Compaction does not free storage by itself. Each compaction produces a new snapshot, and Iceberg keeps the old data and metadata for snapshot isolation and time travel. expire_snapshots removes older snapshots and the files only they need. Its older_than default is 5 days ago and retain_last defaults to 1.

CALL my_catalog.system.expire_snapshots(table => 'db.sample', retain_last => 5);
CALL my_catalog.system.remove_orphan_files(table => 'db.sample', dry_run => true);

rewrite_manifests optimizes scan planning, which matters because streaming appends can leave many small manifest files. remove_orphan_files finds files that table metadata no longer references, such as leftovers from failed jobs. Its older_than default is 3 days ago, and dry_run previews the result.

How does Databricks handle small files?

On Databricks, predictive optimization automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, both Delta Lake and Iceberg. OPTIMIZE improves query performance by optimizing file sizes. For managed Iceberg tables, Unity Catalog manages lifecycle tasks such as snapshot expiration and file compaction, and these tables can only be created when predictive optimization is enabled for table maintenance. If you run Iceberg tables yourself on open-source Spark, use the procedures above.

FAQs

Will compaction delete my data?

No. A compaction commit is a replace operation, which leaves table data unchanged. Earlier snapshots keep referencing the old files until you expire them.

Will queries fail while compaction runs?

Readers always use a consistent snapshot of the table without holding a lock. Writers use optimistic concurrency: if another writer commits first, the failed writer retries against the new table state.

Can I compact only part of the table?

Yes. Pass a where predicate, or set max-files-to-rewrite to cap how many eligible files one run rewrites.

What happens if I cancel the job?

With the default partial-progress.enabled of false, groups of files are not committed before the whole rewrite completes, so the table keeps its current snapshot. Files already written can be left unreferenced, and remove_orphan_files cleans them up.

What is the difference between rewrite_data_files and expire_snapshots?

rewrite_data_files rewrites data files into larger ones. expire_snapshots removes old snapshots and the files only they reference.