What is a lakehouse?

Summary

• A lakehouse keeps data once, in open files on low-cost storage, and layers database guarantees on top: ACID commits, table versions and audit history, plus indexes, caching and query optimization for speed.
• The core mechanism is a transactional metadata layer over open files: data stays in formats such as Apache Parquet on object storage or HDFS, and a table format records which files make up each version of a table.
• In open source, that layer is a table format (Delta Lake, Apache Iceberg or Apache Hudi), paired with a catalog such as Unity Catalog OSS and engines such as Apache Spark.
• Because the files and the table metadata are open, SQL, streaming and machine learning workloads can share one copy of the data, without a second copy that has to be synchronized.
• Databricks builds its lakehouse on Apache Spark, Delta Lake and Unity Catalog, which also manages Apache Iceberg tables and serves them to other engines through the Iceberg REST Catalog API.

What is a lakehouse?

A lakehouse stores your data in open file formats on low-cost storage and puts a transactional table layer on top, so one copy of the data can serve SQL analytics, streaming and machine learning with database-style guarantees. A 2021 CIDR paper by Databricks researchers defines a lakehouse as "a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization." It replaces the two-tier pattern the paper describes: a lake plus a warehouse that take continuous engineering to keep consistent.

The building blocks

An open-source lakehouse has five parts:

Concept	What it is	Why it matters
Open file format	Columnar files such as Apache Parquet or ORC, on object storage or HDFS	Any engine or library can read the bytes directly
Table format	Delta Lake, Apache Iceberg or Apache Hudi: a log or metadata tree that lists the files in each table version	Turns a folder of files into a table with ACID commits, snapshots and schema enforcement
Catalog	A service that maps table names to table metadata, such as Unity Catalog OSS or a Hive metastore	Lets several engines find, share and govern the same tables
Engines	Apache Spark, Apache Flink, Apache Hive and others with a connector for the table format	Batch, streaming, SQL and ML run against one copy of the data
Performance features	Caching, statistics and indexes, and data layout optimizations such as compaction	Improves SQL performance without changing the open file format

How the table layer makes files behave like a database

The paper's central idea is to keep data in a standard file format but "implement a transactional metadata layer on top of the object store that defines which objects are part of a table version." Delta Lake and Apache Iceberg do it differently.
Delta Lake keeps an ordered transaction log in a _delta_log directory next to the data. A writer stages its new data files first and then commits a log entry, and that entry becomes the next atomic version of the table. Readers use the log to select the files that make up one consistent snapshot.
Apache Iceberg keeps table state in metadata files. Each commit writes a fresh metadata file, and an atomic swap makes it the current one. The metadata tracks the schema, partitioning and snapshots, and each snapshot points to a manifest list that leads to the data files.
Either way, readers never see a half-finished write, and older versions stay queryable until cleaned up. With Delta Lake configured in Spark, that history is plain SQL:

CREATE TABLE IF NOT EXISTS events (id BIGINT, kind STRING) USING DELTA;
INSERT INTO events VALUES (1, 'click'), (2, 'view');

DESCRIBE HISTORY events;
SELECT * FROM events VERSION AS OF 1;

What changes in practice

A plain data lake is a set of files with no notion of a table version, so on its own it can't offer atomic commits or snapshot isolation. The lakehouse adds those guarantees while the files stay open, so you don't copy data into a separate system to get them. The paper notes the design suits clouds with separate compute and storage, since applications such as a GPU cluster for machine learning can run on separate nodes against the same data.
Lakehouses often organize tables in layers that refine data step by step (a medallion architecture), but that's a design pattern, not a requirement.

How Databricks approaches the lakehouse

Databricks is built on Apache Spark and adds two key technologies: Delta Lake, a storage layer that supports ACID transactions and schema enforcement, and Unity Catalog, a unified, fine-grained governance solution for data and AI.
The platform isn't tied to one table format. Unity Catalog treats Iceberg tables created in it as managed Iceberg tables, and its Iceberg REST Catalog API lets external Iceberg engines read and write them. Delta Lake tables can also generate Iceberg metadata, so Iceberg clients read the same Parquet files without a rewrite. Where predictive optimization is enabled, it handles OPTIMIZE, VACUUM and ANALYZE for managed tables in either format.

FAQs

Is a lakehouse a product or an architecture?

An architecture. You can assemble one from open-source parts (a table format, a catalog and an engine) or run it on a managed platform that operates those parts for you.

Do I need Apache Spark to build a lakehouse?

No. Delta Lake and Apache Iceberg are open specifications with connectors for several engines. Spark is one option, and it covers batch, streaming, SQL and machine learning in a single engine.

Can I run Unity Catalog outside Databricks?

Yes. Unity Catalog OSS is an Apache 2.0 project with an OpenAPI spec, hosted as a sandbox project by the LF AI & Data Foundation, and it speaks both the Hive metastore API and Apache Iceberg's REST catalog API.