Why do I get a concurrent-modification or commit conflict on a Delta table?

Summary

Delta Lake uses optimistic concurrency control: a write reads the latest table version, stages new files, then checks at commit time whether another writer changed what it read. If so, it fails with a concurrent modification exception instead of corrupting the table. Conflicts come from UPDATE, DELETE, MERGE and compaction operations that touch the same files, from concurrent metadata changes, and from streaming queries that share a checkpoint location.

Why do I get a concurrent-modification or commit conflict on a Delta table?

Delta Lake uses optimistic concurrency control. Each write has three stages: it reads the latest available version of the table to identify which files to rewrite, stages its changes by writing new data files, then validates and commits. At commit time, Delta "checks whether the proposed changes conflict with any other changes that may have been concurrently committed since the snapshot that was read." If they do, "the write operation fails with a concurrent modification exception rather than corrupting the table." So the error means Delta protected your data. Whether two operations conflict "depends on whether they operate on the same set of files."
In the open-source conflict table, INSERT cannot conflict with INSERT or compaction, but can conflict with UPDATE, DELETE and MERGE INTO. UPDATE, DELETE and MERGE INTO can conflict with each other and with compaction, and compaction can conflict with other compaction. Here compaction means a file compaction written with dataChange set to false.

Which exception am I seeing?

Exception	Documented cause	Fix
ConcurrentAppendException	A concurrent operation adds files in the same partition (or anywhere in an unpartitioned table) that your operation reads	Make the separation explicit in the operation condition
ConcurrentDeleteReadException	A concurrent operation deleted a file that your operation read, commonly a DELETE, UPDATE or MERGE that rewrites files	Make the operations touch different files
ConcurrentDeleteDeleteException	A concurrent operation deleted a file that your operation also deletes, for example two compactions rewriting the same files	Run one compaction at a time on the same files
MetadataChangedException	A concurrent transaction updated the table metadata, commonly ALTER TABLE or a write that updates the schema	Keep schema changes apart from writes
ConcurrentTransactionException	A streaming query using the same checkpoint location is started multiple times concurrently	Never run two streaming queries with the same checkpoint location at once
ProtocolChangedException	The table is upgraded to a new protocol version, or several writers create or replace a table, or write to an empty path, at the same time	Upgrade your Delta Lake version, and create the table once before concurrent writers start

How do I make concurrent writes stop conflicting?

Partition the table by the columns your operation conditions use, then put those columns in the condition. The docs say you can make two sets of files disjoint "by partitioning the table by the same columns as those used in the conditions of the operations," and their example adds explicit partition filters to a MERGE condition:

from delta.tables import DeltaTable

target = DeltaTable.forPath(spark, "/data/events")

def merge_partition(updates, date, country):
    (target.alias("t")
        .merge(updates.alias("s"),
               "s.user_id = t.user_id AND s.date = t.date AND s.country = t.country"
               f" AND t.date = '{date}' AND t.country = '{country}'")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute())

Give each streaming query its own checkpointLocation. The docs say "You should never have two streaming queries use the same checkpoint location and run at the same time."

events_a.writeStream.format("delta").outputMode("append") \
    .option("checkpointLocation", "/tmp/delta/events_a/_checkpoints/").toTable("events")
events_b.writeStream.format("delta").outputMode("append") \
    .option("checkpointLocation", "/tmp/delta/events_b/_checkpoints/").toTable("events")

Can I just retry?

If you cannot make the file sets disjoint, re-running the failed operation is an application-level option, not a Delta feature. A conflicting commit fails instead of applying, and the Read stage uses the latest table version, so a re-run starts from the other writer's commit. The exception classes live in delta.exceptions:

import time
from delta.exceptions import ConcurrentAppendException, ConcurrentDeleteReadException

for attempt in range(3):
    try:
        merge_partition(updates, "2026-10-01", "US")
        break
    except (ConcurrentAppendException, ConcurrentDeleteReadException):
        if attempt == 2:
            raise
        time.sleep(2 ** attempt)

How Databricks approaches this

Databricks documents two isolation levels. WriteSerializable is the default and "Ensures that only write operations (not reads) are serializable." Serializable is described as "The strongest isolation level." Set it with ALTER TABLE my_table SET TBLPROPERTIES ('delta.isolationLevel' = 'Serializable'). Row-level concurrency "reduces conflicts between concurrent write operations by detecting changes at the row level," and it needs Databricks Runtime 14.3 LTS or above, a table without partitions, and deletion vectors enabled. To stop overlapping runs of the same job, Lakeflow Jobs has a maximum concurrent runs setting. Its default is 1, and Databricks skips a run when the job has reached its limit.

FAQs

Does Serializable isolation avoid conflicts?

No, it can add them. In the Databricks conflict matrix, INSERT cannot conflict with UPDATE, DELETE or MERGE INTO under WriteSerializable, but can under Serializable. The open-source conflict table lists that pair as "Can conflict."

Can two appends conflict?

The docs list INSERT with INSERT as "Cannot conflict," so jobs that only append do not need to coordinate with each other.

Does partitioning always fix it?

It helps when the partition columns appear in the operation conditions. The docs' example: UPDATE table WHERE date > '2010-01-01' and DELETE table WHERE date < '2010-01-01' conflict if the table is not partitioned by date, because both can attempt to modify the same set of files.