What are the best practices for compaction and maintenance on Delta or Iceberg?

Summary

• Compact small files regularly, and delete unreferenced files only after a retention window longer than your slowest writer. Delta Lake uses OPTIMIZE and VACUUM. Apache Iceberg uses the rewrite_data_files, expire_snapshots and remove_orphan_files procedures.
• Cleanup ends time travel. VACUUM in Delta Lake and expire_snapshots in Iceberg remove the versions they clean up, so set retention to match your recovery needs.
• Documented defaults: Delta VACUUM keeps files for 7 days, Iceberg expire_snapshots removes snapshots older than 5 days, and remove_orphan_files only touches files older than 3 days.
• On Databricks, predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed Delta Lake and Iceberg tables.

What are the best practices for compaction and maintenance on Delta or Iceberg?

Compact small files on a schedule, expire old versions deliberately, and delete unreferenced files only after a window longer than your longest-running write. In Delta Lake that means OPTIMIZE and VACUUM. In Apache Iceberg it means rewrite_data_files, expire_snapshots and remove_orphan_files. Both cleanup steps also end time travel to the versions they remove.

How do I compact and clean up a Delta Lake table?

OPTIMIZE coalesces small files into larger ones (Delta Lake 1.2.0 and above). It is idempotent, and readers keep a snapshot view while it runs. VACUUM deletes files that are no longer referenced and are older than the retention threshold, 7 days by default. Open-source Delta Lake never runs it automatically, and it deletes data files only: log files are removed after checkpoints, 30 days by default (delta.logRetentionDuration). To keep time travel longer, raise delta.deletedFileRetentionDuration and delta.logRetentionDuration.

OPTIMIZE my_table WHERE date >= '2026-10-01';  -- optional partition predicate
VACUUM my_table DRY RUN;                       -- lists files, deletes nothing
VACUUM my_table;

-- liquid clustering
CREATE TABLE events (id INT, department STRING) USING DELTA CLUSTER BY (department);
OPTIMIZE events FULL;                          -- reclusters existing data

Liquid clustering (available from Delta Lake 3.1.0, with the preview flag removed in 3.2) improves on partitioning and ZORDER, and a table can't use clustering together with either. Run OPTIMIZE to cluster incrementally, and OPTIMIZE ... FULL (Delta Lake 3.3 and above) after you change clustering columns. For frequent small writes, auto compaction (Delta Lake 3.1.0 and above) runs after a successful write once a partition has enough small files; enable it with the delta.autoOptimize.autoCompact table property.

How do I compact and clean up an Apache Iceberg table?

Many small data files mean more manifest metadata and less efficient queries. These procedures need the Iceberg SQL extensions on Spark 3.x. Spark 4.0 supports them natively, with case-sensitive names.

CALL catalog_name.system.expire_snapshots(table => 'db.sample', older_than => TIMESTAMP '2026-10-01 00:00:00.000', retain_last => 10);
CALL catalog_name.system.remove_orphan_files(table => 'db.sample', dry_run => true);
CALL catalog_name.system.rewrite_data_files(table => 'db.sample');
CALL catalog_name.system.rewrite_data_files(table => 'db.sample', strategy => 'sort', sort_order => 'zorder(c1,c2)');

expire_snapshots removes snapshots older than older_than (default: 5 days ago) while keeping retain_last (default 1). remove_orphan_files deletes files no metadata references; its older_than defaults to 3 days ago. rewrite_data_files bin-packs by default. With strategy => 'sort' it accepts column sorts or zorder(...), and hilbert(...) was added for Spark 4.1 in Iceberg 1.12.0. For tables with frequent commits, set write.metadata.delete-after-commit.enabled (default false) to prune old metadata files.

Which maintenance task maps to which command?

Task	Delta Lake	Apache Iceberg
Compact small files	OPTIMIZE	rewrite_data_files
Cluster by columns	CLUSTER BY, or ZORDER BY	rewrite_data_files with a sort strategy
Drop old versions	Log entries expire per delta.logRetentionDuration; VACUUM removes data files	expire_snapshots
Delete unreferenced files	VACUUM	remove_orphan_files
Preview deletions	VACUUM ... DRY RUN	remove_orphan_files with dry_run
Check file counts	DESCRIBE DETAIL (numFiles)	SELECT * FROM prod.db.table.files

What mistakes should I avoid?

• A VACUUM interval shorter than your longest job. The docs recommend at least 7 days, because old snapshots and uncommitted files can still be in use by concurrent readers or writers. Delta blocks shorter intervals unless spark.databricks.delta.retentionDurationCheck.enabled is false.
• Other files in the table directory. VACUUM removes files from directories Delta doesn't manage, ignoring those that begin with _. Keep Structured Streaming checkpoints in a name such as _checkpoints.
• Expiring snapshots you may still need. Expired snapshots are no longer available for time travel.
• A short orphan-file window. An older_than shorter than the time any write takes can delete in-progress files and corrupt the table.

How does Databricks automate table maintenance?

Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, Delta Lake and Iceberg, using serverless compute for jobs. It identifies tables that would benefit and queues the work, and the docs say you don't need to run VACUUM manually in most cases. It does not run ZORDER. For managed Iceberg tables, Unity Catalog also manages snapshot expiration and file compaction. When it runs VACUUM, it keeps data files for at least 7 days, so set delta.deletedFileRetentionDuration first if you need longer time travel.

FAQs

How often should I run OPTIMIZE?

It is a cost and performance trade-off. Databricks recommends starting with a daily run, preferably at night when spot prices are low, then adjusting the frequency.

Does OPTIMIZE block other jobs?

Readers aren't interrupted. Writers are different: Delta's conflict table says compaction can conflict with UPDATE, DELETE, MERGE INTO and other compactions, but not with INSERT. The docs' remedy is to partition by the columns the operations filter on, so they touch disjoint files.

Can I run rewrite_data_files without a sort order?

Yes. The default is bin-packing, which combines small files and splits large ones to the table's write size.