Lakehouse versus data lake?

Summary

• A data lake is "a central location that holds a large amount of data in its native, raw format." A lakehouse keeps data in the same kind of object storage and adds a table layer (Delta Lake or Apache Iceberg) plus governance on top.
• The table layer is the difference. Delta Lake and Iceberg add ACID transactions, schema handling and time travel to files that have none of those on their own.
• Databricks lists the shortcomings of a plain data lake as "no support for transactions, no enforcement of data quality or governance, and poor performance optimizations."
• A lakehouse is built on top of a data lake rather than replacing it. The raw layer still exists, and in a medallion design it is the bronze layer.
• Choose by workload: raw landing areas can stay file-based, and shared production tables belong on a table format.

Lakehouse versus data lake?

A lakehouse is a data lake with a table layer and governance on top. Both keep data in object storage. The lakehouse adds transactions, schema handling and time travel, so SQL, streaming and machine learning can share the same tables. Databricks defines a lakehouse as "a data management system that combines the benefits of data lakes and data warehouses", and the Delta Lake docs describe an open source project that "enables building a Lakehouse architecture on top of data lakes."

What is a data lake?

Databricks defines a data lake as "a central location that holds a large amount of data in its native, raw format." The data usually sits in files such as Apache Parquet, "an open source, column-oriented data file format designed for efficient data storage and retrieval." On their own, files have no table layer. The Delta Lake docs show the consequence for concurrent writes: a conflicting write to a Delta table fails with a concurrent modification exception "rather than corrupting the table as would happen with the write operation on a Parquet table."

What does the table layer add?

Open table formats track which files make up a table, so readers see a consistent version.

• Delta Lake keeps table state in a _delta_log directory of JSON commit files and Parquet checkpoint files. The project describes itself as independent and "a sub-project of the Linux Foundation Projects."
• Apache Iceberg keeps table state in metadata files. Every change creates a new metadata file and replaces the old one with an atomic swap. A snapshot is the state of a table at some time, tracked through a manifest list and manifests. Data files can be Parquet, ORC or Avro.

How do a data lake and a lakehouse compare?

Aspect	Data lake (files only)	Lakehouse (Delta Lake or Iceberg)
Transactions	"no support for transactions" (Databricks' description)	ACID transactions with serializable isolation
Concurrent writes	A conflicting write can corrupt a Parquet table	Delta Lake fails the conflicting write, and Iceberg asks the writer to retry on the new version
Data quality	"no enforcement of data quality or governance" (Databricks' description)	Delta Lake schema enforcement, Iceberg schema and partition spec evolution, catalog-level governance
Table history	Not provided by the file format	Time travel and rollback
Streaming	Not provided by the file format	A Delta table is a streaming source and sink, and Iceberg reads and writes with Spark Structured Streaming

When should you use each?

• File-based zone: raw exports and landing areas that you re-read as files. A table format is optional here.
• Table format: data that several teams read and write, or that needs transactions, schema enforcement, rollbacks or access control.

Delta Universal Format (UniForm) lets Iceberg and Hudi clients read Delta tables.

How does Databricks approach the lakehouse?

Delta Lake is "the default format for all operations on Databricks." Unity Catalog is the governance layer, with access control, lineage, auditing and data classification, and it also supports managed Iceberg tables. Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines, a declarative framework for batch and streaming pipelines in SQL and Python. The medallion architecture, which Databricks calls "a recommended best practice but not a requirement", puts raw ingestion in bronze, cleaning and validation in silver, and dimensional modeling and aggregation in gold.

FAQs

Can I migrate from a data lake to a lakehouse?

Yes. In Delta Lake, CONVERT TO DELTA creates a transaction log that tracks the files already in a Parquet directory and infers the schema from their footers. In Iceberg, the Spark migrate procedure replaces a Hive or Spark table with an Iceberg table loaded with the source's data files, while add_files adds files to an existing Iceberg table.

-- Delta Lake
CONVERT TO DELTA parquet.`<path-to-table>`

-- Iceberg
CALL catalog_name.system.migrate('db.sample');

Do I have to replace my data lake?

No. The Delta Lake docs describe building a lakehouse on top of data lakes, and CONVERT TO DELTA builds its transaction log by listing the files already in the directory.

Is a lakehouse a data warehouse?

Databricks defines it as a system that combines the benefits of both. Whether it replaces a separate warehouse depends on your workloads. Test your heaviest queries and governance needs on the lakehouse before you retire anything.