What is a data lakehouse?

Summary

• A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses into a unified architecture that provides scalable storage, querying, and governance for multiple workloads.
• A lakehouse eliminates the need to maintain separate data silos by providing a single source of truth for data engineers, data scientists, analysts, and production systems, reducing operational cost and complexity.
• The core innovation enabling a lakehouse is a transactional storage layer powered by open table formats like Delta Lake, Apache Iceberg, and Apache Hudi, which add ACID transaction support directly to object storage.
• Data lakehouses typically follow a medallion architecture pattern with bronze (raw), silver (cleaned), and gold (curated) layers that progressively refine data while maintaining a single source of truth.
• Databricks lakehouse is built on Apache Spark, Delta Lake, and Unity Catalog, providing integrated support for analytics, machine learning, data engineering, real-time streaming, and batch processing.

What is a data lakehouse?

A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses into a unified architecture. It provides the low-cost, flexible storage and processing capabilities of a data lake with the query performance, data management, and governance features of a data warehouse, all on a single system. Rather than maintaining separate data repositories, a lakehouse enables data engineers, data scientists, analysts, and production systems to all work from one source of truth.
The primary motivation for a lakehouse is practical: organizations need to avoid the expense and complexity of maintaining multiple isolated systems. A data warehouse handles business intelligence queries but requires loading data through an expensive extraction and curation process. A data lake stores raw data cheaply but lacks governance, performance, and consistency guarantees. A lakehouse provides both.

The transactional storage layer

The core technology enabling a lakehouse is a transactional storage layer based on open table formats. Formats like Delta Lake, Apache Iceberg, and Apache Hudi add ACID transaction support directly to cloud object storage (S3, Azure Blob, GCS). This means:

• ACID guarantees: Every write operation either fully succeeds or fully rolls back, preventing data corruption from concurrent writes.
• Consistent snapshots: Readers always see a consistent view of data while writers update it.
• Data reliability: Concurrent pipeline jobs cannot corrupt each other's data, and failed jobs do not leave partial writes that contaminate downstream results.

Delta Lake, the most widely adopted lakehouse format, stores data in Parquet files on cloud object storage and maintains a transaction log recording every schema change, insert, update, and delete. This enables capabilities like time travel (query any historical snapshot), automatic file optimization, and data skipping indexes that accelerate query performance.

Medallion architecture pattern

Data lakehouses typically use a medallion architecture design pattern that progressively improves data quality:

Layer	Purpose	Data Quality
Bronze	Raw data ingestion from source systems	Unmodified source data
Silver	Filtered, cleaned, and augmented data	Deduplicated, validated, quality-checked
Gold	Business-level aggregates and curated, consumption-ready data	Optimized for specific use cases and consumers

This layered approach enables progressive refinement while maintaining a single source of truth. Analysts and applications consume from gold; data engineers can trace lineage back through silver to bronze.

Unified governance and data management

The lakehouse centralizes data governance through a unified catalog that manages access control, lineage tracking, data classification, and discovery across all tables and assets. Role-based access policies apply consistently regardless of which query engine or tool accesses the data. This unified governance extends from raw ingestion through to model serving without gaps or governance silos.
On Databricks, this governance layer is powered by Unity Catalog, which provides a central registry of all data, models, and notebooks. Unity Catalog enforces fine-grained access control at the table and column level, tracks data lineage across pipelines and tools, and enables auditing of all data access.

Support for diverse workloads

A lakehouse architecture supports multiple data types and use cases on one platform, rather than being optimized for a single type of workload. This includes:

• Batch analytics: SQL queries for business intelligence and exploratory data analysis.
• Machine learning: Direct access to data in standard formats with indexing protocols optimized for feature engineering and model training.
• Real-time streaming: Ingest and process event streams with exactly-once semantics.
• Data engineering: Build and orchestrate data pipelines that transform raw data into curated, actionable datasets.
• Advanced analytics: Statistical analysis, predictive modeling, and custom workloads using Python, Scala, and SQL.

The lakehouse handles structured data, semi-structured data (JSON, nested records), and unstructured data (documents, images) in a single platform. This eliminates the data movement and duplication that arises from managing separate systems for BI, ML, and data engineering.

Use cases

Unified Analytics Platform: Replace separate data warehouse and data lake with a single system that serves dashboards, ad hoc queries, and data science workloads.
Real-time Decision Making: Ingest streaming data, perform immediate analysis, and feed results into production applications without batch delays.
Data-Driven ML Pipelines: Build feature pipelines that compute ML features from raw data, train models using those features, and serve predictions with unified governance.
Cross-functional Collaboration: Data engineers own data quality in the lakehouse, analysts run self-service analytics, and data scientists build models, all from the same data source.

FAQs

How is a lakehouse different from a data warehouse?

A data warehouse is optimized for BI queries and requires loading curated data through an expensive extraction process. A lakehouse supports multiple workloads (analytics, ML, engineering, streaming) and queries data directly from cloud storage, eliminating the extraction step and reducing latency.

How is a lakehouse different from a data lake?

A data lake stores raw data cheaply but lacks transactional consistency, data governance, and query performance. A lakehouse adds transaction support through open table formats, unified governance through a catalog, and query optimization, making data both accessible and trustworthy.

Can I query a lakehouse with SQL?

Yes. Databricks lakehouse supports SQL through SQL warehouses and Jupyter notebooks, enabling both self-service BI and programmatic access. SQL queries can access any table in the lakehouse governed by Unity Catalog, with results reflecting the latest committed data.

What are open table formats and why do they matter?

Open table formats like Delta Lake and Apache Iceberg add ACID transactions, time travel, and governance to cloud object storage, eliminating vendor lock-in. They enable portability across different query engines and tools while maintaining data consistency and performance.