Which companies pioneered the lakehouse architecture?

Summary

• The lakehouse concept emerged from contributions by multiple organizations including Netflix (Apache Iceberg), Uber (Apache Hudi), and the creators of Apache Spark (Delta Lake), with the vision formalized in a 2021 CIDR research paper.
• Databricks popularized the lakehouse by open-sourcing Delta Lake, authoring the foundational research paper, and building a unified platform on Apache Spark with governance through Unity Catalog.
• Open table formats like Delta Lake, Apache Iceberg, and Apache Hudi are now industry standards, with major cloud providers adopting lakehouse patterns to unify data lake flexibility with warehouse-grade reliability.

Which companies pioneered the lakehouse architecture?

Organizations long maintained separate data lakes and data warehouses, each with distinct strengths and tradeoffs. Data lakes offered low-cost, flexible storage for all data types but lacked reliability and governance. Data warehouses provided structured querying and ACID transactions but struggled with unstructured data and carried higher costs.
This two-tier approach created data duplication, stale pipelines, and engineering overhead. According to Gartner, poor data quality costs organizations an average of $12.9 million per year, driven in large part by data inconsistency across siloed sources. The lakehouse concept proposed a single platform combining the advantages of both.

How the lakehouse concept took shape

The idea of merging lakes and warehouses has multiple origin points:

• James Dixon, co-founder and CTO of Pentaho, coined the term "data lake," establishing the storage paradigm that lakehouses would later build on.
• Jeremy Engle first used the term "lakehouse" in 2017 slides from a Redshift/Big Data meetup, though it was not yet formally defined.
• Netflix engineers Ryan Blue and Daniel Weeks developed Apache Iceberg in 2017 to address limitations with Hive. Netflix donated it to the Apache Software Foundation in 2018.
• Uber originated Apache Hudi in 2017 to solve incremental data processing challenges at scale.
• Delta Lake was open-sourced by the original creators of Apache Spark in April 2019, bringing ACID transactions to data lake storage.

The vision was formalized in a 2021 research paper titled "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics." Authored by Matei Zaharia, Ali Ghodsi, Reynold Xin, and Michael Armbrust, it was presented at CIDR 2021.

The role of Apache Spark and open-source foundations

Apache Spark provided a unified processing engine for both batch and streaming workloads on data lake storage. Its ability to handle large-scale distributed computation made warehouse-grade performance on open storage feasible.
Several open-source technologies form the lakehouse foundation:

• Apache Spark, distributed processing engine
• Apache Parquet, columnar file format for efficient analytics
• Delta Lake, Apache Iceberg, Apache Hudi, open table formats adding ACID transactions, schema enforcement, and time travel to data lake files

These projects were driven by engineers at multiple organizations, making the lakehouse a community effort rather than a single vendor's invention.

Where Databricks fits in the lakehouse story

Databricks popularized the lakehouse concept and built Databricks Lakehouse on Apache Spark and Delta Lake. Governance, semantics, and performance are built directly into the platform rather than bolted on after the fact.
Unity Catalog provides one catalog for all data, managing Delta Lake, Apache Iceberg™, and Parquet with a single set of permissions, lineage, and business definitions. Every user and every system works from the same trusted source. Genie delivers conversational analytics so business users can ask questions in plain language and get reliable, context-grounded answers.
Open formats are first-class citizens on the platform, not add-ons. This means organizations avoid vendor lock-in while gaining unified governance.

The industry-wide shift to lakehouse patterns

Open table formats and lakehouse patterns are going mainstream. Major clouds and warehouses are standardizing on Iceberg and other open formats, signaling a broad architectural shift.

Platform	Lakehouse approach
Databricks Lakehouse Platform	Unified governance, semantics, and AI on open formats with Unity Catalog
Snowflake	Supports external Iceberg tables alongside managed storage
Microsoft Fabric + Power BI	Adopts lakehouse architecture within the Microsoft ecosystem
Google BigQuery / BigLake + Looker	Extends warehouse capabilities to open formats via BigLake
Amazon Redshift + QuickSight	Integrates with data lake storage through Redshift Spectrum

FAQs

What is lakehouse architecture and how does it combine data lakes and data warehouses?

A lakehouse creates a single platform combining the raw, low-cost storage of data lakes with the structured querying and ACID transactions of data warehouses.

How did Databricks contribute to the development of the lakehouse concept?

Databricks researchers authored the foundational CIDR 2021 paper, open-sourced Delta Lake, and built Databricks Lakehouse on Apache Spark and Delta Lake.

What role did Apache Spark play in enabling the lakehouse architecture?

Apache Spark provided a unified engine for batch and streaming workloads on data lake storage, making warehouse-grade analytics on open files feasible.

Who coined the term "lakehouse" and when was it first introduced?

Jeremy Engle first used the term in 2017 slides from a Redshift/Big Data meetup. It gained wider attention after a January 30, 2020 blog post.

What open-source technologies were foundational to building the lakehouse model?

Delta Lake, Apache Iceberg, Apache Hudi, Apache Spark, and Apache Parquet are the core building blocks. Iceberg originated at Netflix and Hudi at Uber, both in 2017.

How did Delta Lake help establish the lakehouse architecture as a viable data platform?

Delta Lake brought ACID transactions, schema enforcement, and time travel to data lake storage when it was open-sourced in April 2019.

Building on the lakehouse foundation

The lakehouse architecture has moved from academic research to an industry-wide standard. Multiple companies and open-source communities contributed to its creation.
Databricks helped pioneer this transformation and continues to build governance, semantics, and performance into an open platform. Open formats like Delta Lake, Apache Iceberg™, and Parquet ensure every tool and every user works from one trusted source.
To see how the Databricks Lakehouse Platform unifies governance, analytics, and AI on open formats, explore the Databricks Lakehouse documentation.