Why does my Iceberg table have orphan files?

Summary

• An orphan file sits under the table location but no Iceberg metadata file references it. The Iceberg docs say task or job failures in Spark and other distributed engines can leave such files, and normal snapshot expiration cannot always tell that a file is no longer needed.
• Files that old snapshots still reference are not orphans. They stay until expire_snapshots removes the snapshots that need them, so check which problem you have before deleting anything.
• List candidates with remove_orphan_files and dry_run => true. The default older_than is 3 days ago, and a retention interval shorter than your longest write is dangerous.
• If the path strings in metadata differ from the storage listing, the docs warn that cleanup can cause data loss. The equal_schemes, equal_authorities and prefix_mismatch_mode arguments handle that case.
• On Databricks, managed Iceberg tables in Unity Catalog are cleaned with VACUUM, which predictive optimization can run automatically. External Iceberg clients cannot run orphan file removal on them.

Why does my Iceberg table have orphan files?

Orphan files are files under a table's location that no Iceberg metadata file references. The Iceberg docs name the cause: "In Spark and other distributed processing engines, task or job failures can leave files that are not referenced by table metadata, and in some cases normal snapshot expiration may not be able to determine a file is no longer needed and delete it." A failed or killed write job can therefore leave files behind.

Are they really orphans?

Growing storage has other causes. Each write, update, delete, upsert or compaction produces a new snapshot and keeps the old data and metadata for time travel, so a file that an unexpired snapshot references is not an orphan.

Symptom	Likely cause	Fix
Storage keeps growing after deletes or overwrites	Old snapshots still reference the files	expire_snapshots
Unreferenced files after a failed or killed job	Task or job failure left files behind	remove_orphan_files, dry run first
Many old JSON metadata files	Each table change writes a new metadata file	write.metadata.delete-after-commit.enabled, write.metadata.previous-versions-max
Cleanup stops with a prefix mismatch error	Path strings in metadata differ from the storage listing	equal_schemes, equal_authorities, prefix_mismatch_mode

How do I find and remove orphan files?

Call the procedure through a Spark catalog that has the Iceberg SQL extensions enabled. List the candidates first:

CALL catalog_name.system.remove_orphan_files(table => 'db.sample', dry_run => true);

Each output row has an orphan_file_location. When the list looks right, run the same call without dry_run. From Java, SparkActions.get().deleteOrphanFiles(table).execute() runs the action. The main arguments:

• older_than: removes only files created before this timestamp. It defaults to 3 days ago.
• dry_run: when true, nothing is removed. It defaults to false.
• location: the directory to scan. It defaults to the table's location.
• stream_results: the docs recommend true to prevent a Spark driver out-of-memory error with many files.

What can go wrong?

The docs warn: "It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete because it might corrupt the table if in-progress files are considered orphaned and are deleted." Keep older_than longer than your longest-running write.
Iceberg also "uses the string representations of paths when determining which files need to be removed." If paths change over time, for example when HDFS authorities change, the docs state that "This will lead to data loss when RemoveOrphanFiles is run." By default a prefix mismatch raises an error (prefix_mismatch_mode is ERROR). Setting it to DELETE deletes the mismatched files, so map equivalent schemes and authorities with equal_schemes and equal_authorities instead.

How do I keep orphans under control?

Failures can leave files behind, so treat cleanup as routine maintenance. The Iceberg docs list three tasks: expire snapshots ("Regularly expiring snapshots is recommended to delete data files that are no longer needed, and to keep the size of table metadata small"), remove old metadata files, and delete orphan files. For orphan removal the docs say to run it periodically, but "you may not need to execute this often", and it can take a long time when the data and metadata directories hold many files.

How does Databricks approach orphan files?

For Unity Catalog managed Iceberg tables, Databricks runs the maintenance. The docs say Unity Catalog "manages lifecycle tasks like snapshot expiration and file compaction on these tables," and that predictive optimization "automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables." For an Iceberg table, VACUUM "recursively removes unreferenced files from the directories associated with the Apache Iceberg table," and the default retention is 7 days. To preview what it would remove:

VACUUM main.sales.events DRY RUN;

DRY RUN returns a list of up to 1000 files to be deleted. Do not run Iceberg's remove_orphan_files against these tables from an external engine: "External Iceberg clients can't run table maintenance operations, such as expiring snapshots and removing orphan files." On Delta tables with Iceberg reads (UniForm) enabled, VACUUM also cleans up Iceberg metadata for older table versions. For Iceberg tables whose maintenance you own, use the Iceberg procedures above.

FAQs

Does expire_snapshots remove orphan files?

No. It removes old snapshots and the data files that only those snapshots require, and it never removes files a non-expired snapshot still needs. Files that no snapshot references are what remove_orphan_files is for.

Why does the procedure fail with a prefix mismatch?

Iceberg compares the path strings in metadata with the storage listing. When schemes or authorities differ, the default prefix_mismatch_mode of ERROR throws an exception instead of guessing. If the paths are equivalent, declare them with equal_schemes or equal_authorities; the equal_schemes default already treats s3a and s3n as s3. IGNORE skips the mismatched files.