Why is my Delta VACUUM deleting files I still need?

Summary

• VACUUM deletes data files that a Delta table stopped referencing longer ago than its retention threshold. That threshold is delta.deletedFileRetentionDuration, which defaults to interval 7 days, and anything that still reads those files after the window breaks.
• One surprise is time travel. Delta keeps the transaction log for 30 days by default, so DESCRIBE HISTORY can list versions whose data files VACUUM has already removed.
• Turning off spark.databricks.delta.retentionDurationCheck.enabled and passing a short RETAIN can delete files that concurrent readers, uncommitted writers or lagging streams still use, which can corrupt the table.
• In its default FULL mode, VACUUM also deletes files in the table directory that Delta doesn't track, unless they sit in a directory whose name starts with _. Shallow clones break when their source table is vacuumed.
• Run VACUUM <table> DRY RUN first and set retention longer than your longest time-travel need, transaction and stream lag. Delta can't bring back files VACUUM has deleted.

Why is my Delta VACUUM deleting files I still need?

VACUUM is doing what your retention setting tells it to do. Something you depend on is simply older than that setting. In open-source Delta Lake, VACUUM removes files that the table no longer references and that are older than the retention threshold, 7 days by default. If a time-travel query, a RESTORE, a long-running writer, a stream or a shallow clone still needs one of those files, it's gone. The fix is to size retention to your longest real dependency and leave Delta's safety check on.

How VACUUM decides what to delete

Two kinds of files qualify. The first is data files that a commit replaced or removed, through a DELETE, MERGE, overwrite or OPTIMIZE rewrite, more than delta.deletedFileRetentionDuration ago. The clock starts when the file is logically removed from the transaction log, not when it was written. The second kind, in the default FULL mode, is every file in the table directory that the table doesn't reference. VACUUM skips directories whose names begin with _, which is why _delta_log survives.
VACUUM never deletes log files. Delta cleans those up after checkpoints, once they pass delta.logRetentionDuration (default interval 30 days). That gap explains the "missing file" errors on older versions: by default, a table's history outlives the data files behind its older versions.

Match the symptom to the dependency you missed

Symptom	Likely cause	Fix
Time travel or RESTORE to a version listed in DESCRIBE HISTORY fails	The log outlived the data files (30 days of history, 7 days of files)	Raise delta.deletedFileRetentionDuration to the window you need and keep delta.logRetentionDuration at least as long
Readers fail, or the table is corrupted, after a short RETAIN	The safety check was off and VACUUM deleted files a running reader or an uncommitted writer still used	Turn the check back on and keep retention longer than your longest transaction
A stream reading the table fails after VACUUM	The stream fell further behind the table than the retention window	Keep retention longer than the worst-case stream lag
Checkpoints or side files in the table folder disappeared	They weren't in a _-prefixed directory	Store them under a path such as <table>/_checkpoints, or outside the table
A shallow clone throws FileNotFoundException	VACUUM on the source removed files the clone points to	Repair it with CREATE OR REPLACE TABLE ... SHALLOW CLONE
Change data feed reads of older versions fail	Files in _change_data follow the table's retention	Raise retention to cover the window your CDF readers need

Make the next VACUUM safe

Preview first, check the history, then set retention on the table itself:

VACUUM events DRY RUN;

DESCRIBE HISTORY events;

ALTER TABLE events SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = 'interval 30 days',
  'delta.logRetentionDuration' = 'interval 30 days'
);

VACUUM events;

DRY RUN returns the list of files VACUUM would delete without deleting them. DESCRIBE HISTORY shows each version with its timestamp, so you can see which versions fall outside the window.
A RETAIN <n> HOURS clause overrides the table setting for one run. With spark.databricks.delta.retentionDurationCheck.enabled at its default of true, a value shorter than the table's delta.deletedFileRetentionDuration raises an error instead of running. Keep it that way. The Delta docs give the rule for picking an interval: longer than the longest-running concurrent transaction and longer than the longest period any stream can lag behind the most recent update to the table.
VACUUM events LITE (Delta 3.3 and above) finds candidates in the transaction log instead of listing the directory. Only the default FULL mode also deletes untracked files in the table directory.

How Databricks approaches VACUUM

On Databricks, predictive optimization automatically runs VACUUM on Unity Catalog managed tables, and the VACUUM window comes from the delta.deletedFileRetentionDuration table property. Two Databricks Runtime 18.0 changes (12.2 and above for Unity Catalog managed tables) close the history-versus-data gap: time travel queries are blocked if they request a version older than deletedFileRetentionDuration, and logRetentionDuration must be greater than or equal to deletedFileRetentionDuration. For Unity Catalog shallow clones (Public Preview), Unity Catalog tracks which source files each clone uses, so running VACUUM on the source doesn't break the clone. See Remove unused data files with vacuum.

FAQs

What is the default VACUUM retention in Delta Lake?

Seven days. delta.deletedFileRetentionDuration defaults to interval 7 days, and the Delta docs recommend at least 7 days because old snapshots and uncommitted files can still be in use by concurrent readers or writers.

Can I recover files that VACUUM deleted?

Not through Delta. A RESTORE to a version whose data files were vacuumed fails, although setting spark.sql.files.ignoreMissingFiles to true lets a partial restore go through.

Is RETAIN 0 HOURS ever safe?

Only when nothing reads, writes or streams from the table during the run, and only if you accept losing all time travel. It deletes every data file that doesn't belong to the latest version, which is why the safety check blocks it by default.