How do I vacuum old files from a Delta table safely?

Summary

• VACUUM deletes data files that a Delta table no longer references and that are older than the retention threshold, which defaults to 7 days. Open-source Delta Lake never runs it automatically, and it doesn't delete transaction log files.
• The retention threshold is your safety margin. It has to exceed your longest-running job or reader and the longest time a streaming query on the table might stay stopped, and it caps time travel: versions older than it can't be read after a vacuum.
• Preview with VACUUM <table> DRY RUN, set retention on the table with delta.deletedFileRetentionDuration, then run VACUUM with the default threshold.
• Leave spark.databricks.delta.retentionDurationCheck.enabled on. Delta Lake warns that deleting files that concurrent readers or writers still use can make readers fail or corrupt the table.
• On Databricks, predictive optimization runs VACUUM automatically on Unity Catalog managed tables.

How do I vacuum old files from a Delta table safely?

Run a dry run, keep the retention threshold at the 7-day default or longer, and only then run VACUUM. When you update, delete, overwrite or compact a Delta table, the old data files are only logically deleted; they stay in storage so concurrent readers, long-running writers and time travel can still use them. VACUUM is the step that physically removes them once they're older than the retention threshold. Make that window shorter than something that still reads or writes the table and, in Delta Lake's own warning, concurrent readers can fail or, worse, the table can be corrupted.

What controls VACUUM

Two table properties and one session setting govern vacuum and time travel:

• delta.deletedFileRetentionDuration (default interval 1 week) is how long logically deleted data files are kept before VACUUM can remove them. This is the 7-day threshold.
• delta.logRetentionDuration (default interval 30 days) is how long table history is kept. Old log entries are cleaned up automatically when checkpoints are written, never by VACUUM.
• spark.databricks.delta.retentionDurationCheck.enabled is Delta Lake's safety check against a dangerous VACUUM. Setting it to false lets you vacuum below the recommended minimum.

Because time travel needs both the log and the data files for a version, the shorter of the two windows sets how far back you can query.

A safe VACUUM, step by step

1. Size the window from your workloads. Delta Lake's guidance is that retention should exceed the longest possible job when VACUUM runs alongside concurrent readers or writers, and should exceed the longest time any streaming query reading the table might be stopped, because a restarted stream must still read old files.
2. Set it on the table, so every vacuum uses the same value. To keep 30 days of history readable, match the default log retention, then preview and run:

ALTER TABLE events SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = 'interval 30 days'
);

VACUUM events DRY RUN;
VACUUM events;

3. Read the dry run. It returns the list of files that would be deleted and removes nothing. Anything unexpected in that list is a reason to stop.
4. Keep other files out of the table directory. VACUUM removes files from directories Delta Lake doesn't manage, while ignoring directories whose names begin with _. A Structured Streaming checkpoint under the table path is at risk unless it sits in a directory such as _checkpoints.
5. Run it from code if you prefer. The Python API takes the retention in hours:

from delta.tables import DeltaTable

events = DeltaTable.forName(spark, "events")
events.vacuum()     # default retention period
events.vacuum(720)  # keep files needed by versions from the last 720 hours

Which VACUUM to run

Command	When to use it	Trade-off
VACUUM events DRY RUN	Before the first run, and after any retention change	Lists files only; deletes nothing
VACUUM events (FULL mode, the default)	Routine cleanup	Lists the whole table directory, so it also removes files the table never referenced
VACUUM events LITE (Delta 3.3 and above)	Large tables you vacuum regularly	Uses the transaction log instead of listing the directory; raises DELTA_CANNOT_VACUUM_LITE if the log has been pruned
VACUUM events RETAIN <n> HOURS	A one-off threshold override	Below the table's retention it needs the safety check off and risks concurrent readers and writers
VACUUM events USING INVENTORY <table>	You already keep a listing of the table's storage	The inventory must hold each file's path, size, type and last modification time

Setting spark.databricks.delta.vacuum.parallelDelete.enabled to true makes Spark delete files in parallel, based on the number of shuffle partitions.

How Databricks approaches VACUUM

Predictive optimization automatically runs VACUUM, along with OPTIMIZE and ANALYZE, on Unity Catalog managed tables, and it's enabled by default for accounts created on or after November 11, 2024. For tables you vacuum yourself, the same retention logic applies, with documented specifics: in Databricks Runtime 18.0 and above, the deletedFileRetentionDuration table property controls retention and logRetentionDuration must be greater than or equal to it. LITE mode is in Public Preview on Databricks Runtime 16.4 LTS and above and requires at least one successful VACUUM within the log retention threshold. VACUUM commits audit information to the transaction log, controlled by spark.databricks.delta.vacuum.logging.enabled, and DESCRIBE HISTORY shows it.

FAQs

What is the default VACUUM retention in Delta Lake?

Seven days for data files, set by delta.deletedFileRetentionDuration. Table history in the log is kept for 30 days by default, so you can time travel up to 30 days back only until a vacuum removes the data files those versions need.

Can I run VACUUM with RETAIN 0 HOURS?

Only after setting spark.databricks.delta.retentionDurationCheck.enabled to false, which Delta Lake reserves for when you're certain no operation on the table takes longer than the retention you specify. Zero retention also makes every earlier version unreadable by time travel.

Does VACUUM delete the transaction log?

No. VACUUM deletes only data files. Log files are deleted automatically and asynchronously after checkpoint operations, based on delta.logRetentionDuration.

Does VACUUM run automatically?

Not in open-source Delta Lake, where vacuum is never triggered on its own. On Databricks, predictive optimization runs it for Unity Catalog managed tables.