Is the lakehouse hype over?

Summary

• The 2021 CIDR paper that set out the architecture defines a lakehouse as "a data management system based on low-cost and directly-accessible storage" that adds warehouse features such as ACID transactions, data versioning, auditing, indexing, caching and query optimization.
• Its building blocks now ship as documented open-source features: Delta Lake provides ACID transactions and time travel on object storage, and Apache Iceberg provides atomic commits, schema evolution and snapshot time travel that Spark, Trino, Flink and other engines use on the same tables.
• Commercial platforms document the same blocks. Snowflake documents Apache Iceberg tables in Snowflake storage or external cloud storage that you manage, and Databricks supports both Delta Lake and Apache Iceberg tables in Unity Catalog.
• Whether the buzz fades matters less than fit. A lakehouse earns its keep when several engines, teams or workloads (BI, streaming, machine learning) need one governed copy of data; a single team with data that fits one database may not need one.
• Decide with criteria: count the engines that touch your data, check portability and governance needs, and budget for table maintenance such as compaction and cleaning up old files.

Is the lakehouse hype over?

Set the hype cycle aside and look at what ships. The lakehouse is now a set of documented, open components: table formats that bring ACID transactions to files on object storage, catalogs that several engines share, and engines that read the same tables. The useful question is whether those components fit your data. If several engines or teams need one governed copy of data, they usually do. If one team runs one tool on data that fits a single database, a lakehouse may add machinery you don't need.

What a lakehouse is, precisely

The architecture was set out in a 2021 CIDR paper, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics." It pairs low-cost, directly accessible storage with warehouse management features, using open, direct-access formats such as Apache Parquet and ORC and first-class support for machine learning and data science. Databricks' docs put it more briefly: "A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses."
That means three layers. Data sits in an open table format (Delta Lake, Apache Iceberg or Apache Hudi) as data files, such as Parquet, plus a transaction log or metadata layer. A catalog tracks tables and permissions. Engines read and write through the format's protocol, so no single engine owns the data.

The building blocks, as each project documents them

Capability	Delta Lake	Apache Iceberg
Transactions	ACID guarantees between reads and writes on S3, ADLS, GCS and HDFS; optimistic concurrency between writers	"A linear history of atomic table updates"; optimistic concurrency for concurrent writers
Readers during writes	A consistent snapshot while the table changes	"A consistent snapshot of the table without holding a lock"
Schema changes	Schema validated on write; automatic schema update during DML	Full schema evolution: add, rename and reorder columns
History	Time travel to older table snapshots	Time travel and rollback to earlier snapshots
Streaming	"Unifies streaming and batch data processing"	Incremental streaming reads and streaming writes with Spark Structured Streaming
Engines the project names	Apache Spark, Apache Flink, Apache Hive, Trino, Presto and more	Spark, Trino, Flink, Presto, Hive and Impala
Upkeep	VACUUM removes files no longer referenced by the table	Expire snapshots, delete orphan files, compact data files

The catalog layer has opened up as well: Unity Catalog OSS is Apache 2.0 licensed, a sandbox project at LF AI & Data, and implements the Iceberg REST Catalog APIs. Warehouse products document open formats too: Snowflake lets you keep its Apache Iceberg tables in Snowflake storage or in "external cloud storage that you manage."

How to decide if a lakehouse fits

Work through five questions:

1. How many engines touch the same data? Several engines, such as Spark for pipelines, a SQL engine for BI and Python tools for data science, are the case open table formats were built for.
2. Do BI, streaming and machine learning need the same tables? Keeping one copy avoids exporting data between systems and reconciling copies.
3. Does portability matter? Any engine that implements the Delta Lake or Iceberg spec can read the tables, which keeps the option to change engines.
4. Is governance split across tools? A shared catalog puts permissions, lineage and discovery in one place.
5. Can you run the upkeep? Tables need compaction and cleanup of old files and snapshots. Managed platforms automate some of it; self-managed stacks schedule it.

Mostly one tool, one team and modest data? A single database or warehouse may be simpler. Many engines and workloads? A lakehouse fits.

How Databricks approaches the lakehouse

The Databricks lakehouse uses Apache Spark, Delta Lake and Unity Catalog, and Unity Catalog supports both formats: managed Delta Lake tables and managed Iceberg tables, with an Iceberg REST Catalog API that lets external Iceberg engines read and write managed Iceberg tables. Delta tables can turn on Iceberg reads (UniForm), which generates Iceberg metadata without rewriting data files, so Iceberg clients read the same copy. For upkeep, predictive optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables in either format. Databricks open-sourced Unity Catalog under Apache 2.0, and its Lakeflow Pipelines extend and interoperate with Apache Spark Declarative Pipelines.

FAQs

Where does the lakehouse definition come from?

From a 2021 CIDR paper, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics," which defines it as directly accessible storage with warehouse features such as ACID transactions and versioning.

Do I have to choose between Delta Lake and Apache Iceberg?

Not necessarily. Unity Catalog governs both formats, and Delta tables with Iceberg reads enabled can be read by Iceberg clients from the same files.

What work does a self-managed lakehouse involve?

Running a catalog, managing access, and scheduling table maintenance such as compaction, snapshot expiry and file cleanup.