Delta Lake versus Parquet: what do I gain?

Summary

• Parquet is an open source, column-oriented data file format. Delta Lake "extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling."
• With Delta Lake you gain ACID transactions (a conflicting concurrent write fails with an exception instead of corrupting the table), schema validation on write, time travel, and DELETE, UPDATE and MERGE.
• A Delta table is a batch table as well as a streaming source and sink. Spark's file source can also stream a directory of Parquet files.
• CONVERT TO DELTA converts a Parquet table in place. Afterwards, route all writes through Delta Lake.
• On Databricks, Delta Lake is the default format for all operations, and Unity Catalog adds governance.

Delta Lake versus Parquet: what do I gain?

Parquet is a file format. Delta Lake builds on Parquet files: the Databricks docs say it "extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling." What you gain is that log and the table behavior built on it: ACID transactions, schema validation on write, queryable table versions, and row-level DELETE, UPDATE and MERGE. Plain Parquet stays a good fit when you only need portable files that many tools can read.

What is the difference between Parquet and Delta Lake?

Apache Parquet is "an open source, column-oriented data file format designed for efficient data storage and retrieval," supported in many programming languages and analytics tools. Each Parquet file carries its own schema: Spark's Parquet support "automatically preserves the schema of the original data," and Delta's conversion command infers a table schema "by reading the footers of all Parquet files."
Delta Lake is "an open source project that enables building a Lakehouse architecture on top of data lakes." A Delta table is a directory of Parquet data files plus a log. Log entries are JSON files in the _delta_log directory, and readers use the log to choose which data files to process.

What do I gain with Delta Lake?

Capability	Parquet data files alone	Delta Lake
Concurrent writes	The Delta docs say a conflicting write on a Parquet table can corrupt it	ACID transactions: a conflicting write fails with a concurrent modification exception
Schema	Each file has its own schema; Spark merges compatible schemas on read only when schema merging is enabled	Writes are validated against the table schema; evolution is opt-in
Table versions	No transaction log, so no versions to query	Time travel with VERSION AS OF or TIMESTAMP AS OF, plus DESCRIBE HISTORY and RESTORE
Row-level changes	Not defined by the Parquet file format	DELETE, UPDATE and MERGE INTO
Streaming	Spark's file source can stream a directory of Parquet files	A table is a batch table as well as a streaming source and sink

How do transactions and schema checks work?

Delta writes use optimistic concurrency control in three stages: read the latest table version, write new data files, then validate and commit. If a transaction fails after writing data files, those files don't corrupt the table and don't become part of it. VACUUM deletes them.
On write, "Delta Lake automatically validates that the schema of the DataFrame being written is compatible with the schema of the table." Evolution is opt-in: set the mergeSchema option on a write, or on Delta Lake 4.3 and above use WITH SCHEMA EVOLUTION in a SQL INSERT.

How do I use Delta Lake from SQL?

On open source Spark, add the Delta Lake package and set spark.sql.extensions to io.delta.sql.DeltaSparkSessionExtension and spark.sql.catalog.spark_catalog to org.apache.spark.sql.delta.catalog.DeltaCatalog. Then:

-- Convert an existing Parquet directory in place
CONVERT TO DELTA parquet.`<path-to-table>`;

-- Row-level changes
UPDATE people10m SET gender = 'Female' WHERE gender = 'F';
DELETE FROM people10m WHERE birthDate < '1955-01-01';

-- Time travel and history
SELECT * FROM people10m VERSION AS OF 1;
DESCRIBE HISTORY people10m;

The conversion happens in place: it "lists all the files in the directory, creates a Delta Lake transaction log that tracks these files," and infers the schema. Avoid updating or appending data files during the conversion, add PARTITIONED BY for partitioned data, and afterwards make sure all writes go through Delta Lake.

When is plain Parquet enough?

When you need portable files and nothing more. A Delta table is more than its data files: readers use the log to pick the files in the current version, and files removed from it stay in the directory until VACUUM deletes them (the default retention threshold is 7 days). A reader that only understands Parquet doesn't use the log to make that choice.

How does Databricks approach Delta Lake?

"Delta Lake is the default format for all operations on Databricks," and Databricks "originally developed the Delta Lake protocol and continues to actively contribute to the open source project." Streaming tables and materialized views that pipelines create are stored as Delta tables by default. Unity Catalog is "the unified governance layer for data and AI built into Databricks," covering access control, lineage and auditing. Predictive optimization automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables. Iceberg reads (UniForm) generate Iceberg metadata so Iceberg clients can read Delta tables, and that client support is read-only.

FAQs

How far back can I time travel?

By default up to 30 days, unless you run VACUUM or change the retention properties. delta.logRetentionDuration controls how long table history is kept, with a default of 30 days.

Is Delta Lake only for Databricks?

No. Delta Lake is "an independent open-source project and not controlled by any single company," and its governing Delta Lake Project is a sub-project of the Linux Foundation Projects. The Delta docs list connectors that read and write Delta tables from engines like Apache Spark, Apache Flink and Apache Hive.

Do Delta tables still use Parquet files?

Yes. Delta Lake "extends Parquet data files" with the log, and each log entry records which data files to logically add and remove.