Open lakehouse in 2026

Summary

• An open lakehouse puts table-level transactions and metadata on top of open files in a data lake. Its layers are file formats such as Apache Parquet, table formats (Delta Lake, Apache Iceberg, Apache Hudi), catalogs with open APIs, and engines such as Apache Spark.
• As of October 2026, the official release notes list Apache Spark 4.2.0 (July 14, 2026), Delta Lake 4.4.0, which adds Spark 4.2 support, and Apache Iceberg 1.12.0 (September 29, 2026).
• The layers interoperate. UniForm lets Iceberg and Hudi clients read Delta tables, Unity Catalog OSS implements the Iceberg REST Catalog APIs, and Apache XTable (Incubating) translates table metadata between Hudi, Delta Lake and Iceberg.
• Spark added Spark Declarative Pipelines in 4.1.0, and Spark 4.2.0 adds Auto CDC to it. Spark Connect, introduced in 3.4, gained a lightweight Python client in 4.0.0.
• Databricks documents Iceberg reads for Delta tables, managed Iceberg tables in Unity Catalog and an Iceberg REST catalog for external clients.

Open lakehouse in 2026

A lakehouse puts table-level transactions and metadata on top of files in a data lake. It is open when each layer has a public specification or an open-source implementation that more than one product can use: files, a table format, a catalog with an open API, and engines such as Apache Spark. Release facts below are as of October 2026.

What are the layers of an open lakehouse?

Layer	Open projects and specs	What the project's own docs say
File format	Apache Parquet, Apache ORC	Parquet is an "open source, column-oriented data file format designed for efficient data storage and retrieval"
Table format	Delta Lake, Apache Iceberg, Apache Hudi	Delta Lake enables "building a Lakehouse architecture on top of data lakes"; Iceberg lets engines "safely work with the same tables, at the same time"; Hudi "pioneered the concept of 'transactional data lakes'"
Catalog	Unity Catalog OSS, Iceberg REST catalog	Unity Catalog OSS is Apache 2.0 licensed, with an OpenAPI specification, server and clients, and implements the Iceberg REST Catalog APIs
Engine	Apache Spark, Apache Flink	Spark is a "multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters"

What do the latest releases add?

Project	Release	What the release notes list
Apache Spark	4.2.0 (July 14, 2026)	Geospatial types, Change Data Capture (a SQL CHANGES clause, plus Auto CDC in Spark Declarative Pipelines), Data Source V2 transaction management, metric views, a modernized Web UI, and building and running Spark on Java 25
Delta Lake	4.4.0 (4.4.1 has since been released)	Adds Apache Spark 4.2 support, expands integration with the UC Delta Table API, adds identity-column and generated-column support to Delta Spark
Apache Iceberg	1.12.0 (September 29, 2026)	Removes Spark 3.4 and Flink 2.0 support, adds Flink 2.2 and 2.3 support, adds a Hilbert-curve clustering strategy for rewrite_data_files on Spark 4.1

The Iceberg specification has versions 1, 2 and 3 complete and adopted.

How do the table formats interoperate?

• UniForm. "Delta Universal Format (UniForm) allows you to read Delta tables with Iceberg and Hudi clients." In open-source Delta Lake the Iceberg path needs column mapping, a Hive Metastore catalog and writes from Delta Lake 3.1 or above, the Hudi path needs 3.2 or above, and those clients can only read. On Databricks, Iceberg reads covers Iceberg clients.
• Metadata translation. Apache XTable (Incubating) reads a table's existing metadata and writes metadata for other formats, so the same data files can be read as Delta, Hudi or Iceberg.
• Catalog APIs. The Iceberg REST Catalog API is an OpenAPI specification, and Unity Catalog OSS implements it to interoperate with a broad ecosystem.

How does Apache Spark fit in?

Spark Declarative Pipelines (SDP) is "a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark." The 4.1.0 release notes introduce it as a new component. Its objects are streaming tables, materialized views, temporary views, flows and sinks, and Python pipelines import from pyspark import pipelines as dp.
Spark Connect introduced "a decoupled client-server architecture" in Spark 3.4. It supports PySpark and Scala applications and most of the Structured Streaming API. Spark 4.0.0 added a lightweight Python client, a Java client with full API compatibility, the spark.api.mode configuration and a Swift client.

How does Databricks approach the open lakehouse?

Databricks says "Delta Lake is the default format for all operations on Databricks," and the Delta Lake transaction log "has a well-defined open protocol that can be used by any system to read the log." For other formats and engines it documents three paths:

• Iceberg reads. Delta tables generate Iceberg metadata automatically so Iceberg clients can read them without rewriting files. Iceberg clients are read-only.
• Managed Iceberg tables. Iceberg tables you create in Unity Catalog are managed Iceberg tables, and Databricks supports Iceberg spec versions 1, 2 and 3 with Parquet.
• Iceberg REST catalog. External clients such as Apache Spark, Apache Flink and PyIceberg reach Unity Catalog tables through /api/2.1/unity-catalog/iceberg-rest.

Lakeflow Pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime."

FAQs

Which table format should I choose in 2026?

Delta Lake, Iceberg and Hudi are all open projects with public documentation. Decide by the engines and catalog you must support, the features you rely on, and whether other engines need to write. Hudi documents upserts, indexes and incremental queries, Iceberg documents hidden partitioning and partition evolution, and Delta Lake documents time travel and schema enforcement.

What is Spark Declarative Pipelines, and when did it arrive?

It is the Apache Spark framework listed in the Spark 4.1.0 release notes as "Declarative Pipelines, a new component to define and run data pipelines." Spark 4.2.0 adds Auto CDC, a declarative SCD Type 1, to it.