How do I resolve Iceberg metadata or manifest bloat?

Summary

• Iceberg keeps history by design. Each change writes a new table metadata file and each write creates a snapshot, and both pile up until you clean them.
• expire_snapshots removes old snapshots and the files only they need. Setting write.metadata.delete-after-commit.enabled prunes old metadata JSON files.
• Use rewrite_manifests when manifests don't match how you query, rewrite_data_files for small data files, and remove_orphan_files for leftovers from failed jobs.
• On Databricks, Unity Catalog manages snapshot expiration and file compaction for managed Iceberg tables. For Delta tables with Iceberg reads, VACUUM removes old Iceberg metadata.

How do I resolve Iceberg metadata or manifest bloat?

Match the cleanup to the layer that grew. The Iceberg maintenance docs recommend regularly expiring snapshots "to delete data files that are no longer needed, and to keep the size of table metadata small". They group expiring snapshots, removing old metadata files and deleting orphan files as recommended maintenance, and list compacting data files and rewriting manifests as optional maintenance.

Which layer is growing?

Layer	Why it grows	Fix
Snapshots	"Each write to an Iceberg table creates a new snapshot", and snapshots accumulate until they are expired	expire_snapshots
Table metadata files (JSON)	"Each change to a table produces a new metadata file", and old ones "are kept for history by default"	write.metadata.delete-after-commit.enabled
Manifests (Avro)	Manifests are compacted in the order they are added. When the write pattern doesn't align with the query pattern, metadata can be rewritten to re-group data files into manifests	rewrite_manifests
Data files	"More data files leads to more metadata stored in manifest files"	rewrite_data_files
Orphan files	Task or job failures can leave files that no metadata references	remove_orphan_files

How do I check which layer is the problem?

Add a metadata table name after the table name:

SELECT * FROM catalog_name.db.sample.snapshots;
SELECT * FROM catalog_name.db.sample.manifests;
SELECT * FROM catalog_name.db.sample.metadata_log_entries;

The .manifests table lists each current manifest with its length and its added, existing and deleted data file counts. The .files table helps you spot small data files.

How do I expire snapshots?

Iceberg procedures run through CALL. On Spark 3.x they need the Iceberg SQL extensions, and Spark 4.0 supports them natively.

CALL catalog_name.system.expire_snapshots(
  table => 'db.sample',
  older_than => TIMESTAMP '2021-06-30 00:00:00.000',
  retain_last => 100
);

The procedure removes old snapshots and the data files "uniquely required by those old snapshots", and "will never remove files which are still required by a non-expired snapshot". older_than defaults to 5 days ago and retain_last to 1. If you omit both, the table's expiration properties apply, such as history.expire.max-snapshot-age-ms and history.expire.min-snapshots-to-keep. Snapshots that are still referenced by branches or tags are not removed. Add stream_results => true, which the docs recommend "to prevent Spark driver OOM from large file size", and clean_expired_metadata => true to clean up partition specs and schemas that no snapshot references.

How do I prune old metadata files?

ALTER TABLE catalog_name.db.sample SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '20'
);

Tables with frequent commits, like those written by streaming jobs, "may need to regularly clean metadata files". The defaults are false and 100. With deletion on, the writer deletes the oldest tracked metadata file after each commit. It does not delete orphaned metadata files, which are cleaned up as part of orphan file deletion.

How do I rewrite manifests?

CALL catalog_name.system.rewrite_manifests(table => 'db.sample', sort_by => array('category'));

The procedure rewrites manifests "to optimize scan planning" and runs in parallel as a Spark job. sort_by clusters manifests by the partition transforms you list, and choosing frequently queried ones "can reduce planning time by skipping unnecessary manifests". It invalidates cached Spark plans that reference the table. Iceberg also merges manifests on write by default (commit.manifest-merge.enabled is true), once commit.manifest.min-count-to-merge (default 100) manifests accumulate.

How do I compact small data files?

CALL catalog_name.system.rewrite_data_files('db.sample');

This bin-packs small files into larger ones, which the docs say reduces "metadata overhead and runtime file open cost".

How do I remove orphan files safely?

CALL catalog_name.system.remove_orphan_files(table => 'db.sample', dry_run => true);

Start with dry_run. The default older_than is 3 days ago. The docs warn that using a retention interval shorter than the time any write needs to complete "might corrupt the table".

How does Databricks handle Iceberg metadata?

• Managed Iceberg tables. "Unity Catalog manages lifecycle tasks like snapshot expiration and file compaction on these tables." Predictive optimization automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables.
• Delta tables with Iceberg reads (UniForm). Iceberg metadata accumulates in the metadata/ directory, and "Only VACUUM removes it". On Databricks Runtime 17.2 and above, VACUUM table_name FULL deletes untracked files in metadata/ while keeping the metadata that clients need to read current table versions. Run it after the delta.deletedFileRetentionDuration period has elapsed. With predictive optimization on, Databricks handles this for you.
• Foreign Iceberg tables. They are read-only in Databricks, so maintain them from the engine and catalog that own them.

FAQs

Will expiring snapshots break time travel?

Yes, for the snapshots it removes. "Expiring old snapshots removes them from metadata, so they are no longer available for time travel queries."

How often should I run maintenance?

The docs say to expire snapshots regularly and to run orphan file removal periodically, but give no fixed interval. Use the metadata tables above to decide.

Do I always need rewrite_manifests?

No. The docs list it as optional maintenance, useful when the write pattern doesn't align with the query pattern.