State of data engineering 2026

Summary

• Flink, Airflow, Spark and Delta Lake started new major versions in 2025, and all of them kept shipping in 2026. As of October 2026 the newest feature releases are Apache Spark 4.2.0 (July 14), Delta Lake 4.4.0 (August 20), Apache Iceberg 1.12.0 (September 29), Apache Flink 2.3.0 (June 25), Apache Kafka 4.3.0 (May 22) and Apache Airflow 3.3.0 (July 6).
• Declarative pipelines and change data capture are now part of Apache Spark. Spark Declarative Pipelines arrived in 4.1.0, and Spark 4.2.0 added a SQL CHANGES clause and Auto CDC.
• Streaming widened. Spark 4.1.0 added Real-Time Mode for Scala stateless queries and 4.2.0 added the PySpark trigger. Flink 2.3.0 added FROM_CHANGELOG and TO_CHANGELOG operators.
• Delta Lake and Iceberg both shipped catalog and format work. Delta Lake 4.3.0 lists Unity Catalog Delta REST API integration and atomic, incremental Iceberg conversion for UniForm. Iceberg 1.12.0 lists geometry and geography read and write support.
• Upgrades need version checks. Delta Lake 4.4.0 is built for Spark 4.2.0, 4.1.0 and 4.0.1, and Iceberg 1.12.0 removes Spark 3.4 and Flink 2.0 support.

State of data engineering 2026

This page reads the state of the open-source stack from official release notes, anchored at October 2026. Four core projects began a new major version in 2025: Apache Flink 2.0.0 (March 24), Apache Airflow 3.0.0 (April 22), Apache Spark 4.0.0 (May 23) and Delta Lake 4.0.0 (June 9). Spark 4.2.0 is, in its release notes' words, "the third release of the Spark 4.x line".

Which releases define the stack today?

Project	Newest feature release	What its notes lead with
Apache Spark	4.2.0 (July 14, 2026)	Geospatial types, Change Data Capture, Arrow-optimized Python UDFs on by default
Delta Lake	4.4.0 (August 20, 2026)	Spark 4.2 support, identity columns in SQL DDL, generated columns with Unity Catalog
Apache Iceberg	1.12.0 (September 29, 2026)	Geometry and geography read and write, Flink 2.2 and 2.3 support
Apache Flink	2.3.0 (June 25, 2026)	Changelog conversion operators, materialized table evolution, watermark alignment
Apache Kafka	4.3.0 (May 22, 2026)	Cordoned log directories, headers in Kafka Streams state stores
Apache Airflow	3.3.0 (July 6, 2026)	Asset partitioning, task and asset state store, Java and Go task SDKs

What changed in Apache Spark from 4.0 to 4.2?

• 4.0.0 (May 23, 2025): Scala 2.13 and JDK 17 became the defaults, ANSI SQL mode became the default, and the VARIANT type was added. Structured Streaming gained the Arbitrary State API v2.
• 4.1.0 (December 16, 2025): Spark Declarative Pipelines (SDP), a framework where "you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries". Real-Time Mode arrived for Structured Streaming, and SQL Scripting and VARIANT became GA.
• 4.2.0 (July 14, 2026): a SQL CHANGES clause plus DataFrame, PySpark and Spark Connect APIs to read row-level changes in batch and streaming, Auto CDC in SDP for SCD Type 1 upserts, geospatial types enabled by default, and the Real-Time Mode trigger in PySpark.

What changed in table formats and catalogs?

Delta Lake shipped four feature releases in 2026: 4.1.0 (February 26), 4.2.0 (April 16), 4.3.0 (June 18) and 4.4.0 (August 20). The 4.3.0 notes lead with Unity Catalog Delta REST API integration for Spark, the replaceOn and replaceUsing DataFrame APIs, and a UniForm that "now writes Iceberg metadata atomically with the Delta commit".
Iceberg 1.11.0 (May 19, 2026) introduced a SQL UDF specification and geospatial bounding box types. Iceberg 1.12.0 (September 29, 2026) reads and writes geometry and geography values in Parquet and adds Flink 2.2 and 2.3 support.

What changed in streaming and orchestration?

Flink went from 2.0.0 (March 24, 2025) through 2.2.0 (December 4, 2025) to 2.3.0, which its announcement says "significantly expands SQL capabilities with changelog conversion operators". Kafka 4.3.0 added headers to Kafka Streams state stores (KIP-1271) and deprecated the streams-scala module. Airflow 3.0.0 introduced the Task Execution API and Task SDK, DAG versioning and a React UI rewrite. Airflow 3.3.0 lists asset partitioning, a task and asset state store, pluggable retry policies and a language Task SDK for Java and Go.

What should you check before upgrading?

Check	What the release notes say
Spark baseline	Spark 4.0.0 made Scala 2.13 and JDK 17 the defaults and dropped Scala 2.12 and JDK 8 and 11
Delta Lake and Spark pairing	Delta Lake 4.4.0 is built for Spark 4.2.0, 4.1.0 and 4.0.1
Iceberg and engine versions	Iceberg 1.12.0 removes Spark 3.4 and Flink 2.0 support, and 1.11.0 dropped Java 11
Kafka 5.0 deprecations	Kafka 4.3.0 deprecates the streams-scala module, marked for removal in Kafka 5.0
Airflow major upgrade	Airflow 3.0.0 replaces REST API v1 with v2 and removes SequentialExecutor

How does Databricks fit?

Lakeflow Pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime". For table-format interoperability, the Databricks feature is called Iceberg reads. It configures Delta Lake tables to generate Iceberg metadata, so Iceberg clients can read Delta Lake data without rewriting files. The table must be registered in Unity Catalog and have column mapping enabled. Through the Unity Catalog Iceberg REST catalog, managed Iceberg tables can be read and written, while Delta tables with Iceberg reads enabled can only be read.

FAQs

Is Spark Declarative Pipelines part of Apache Spark?

Yes. It arrived in Spark 4.1.0 as the SPIP "Declarative Pipelines" (SPARK-51727). The Databricks product built on it is Lakeflow Pipelines.

Which Spark release is current?

Spark 4.2.0, released July 14, 2026. The news page also lists 4.1.3 and 4.0.4 (July 15) and 3.5.9 (July 16) as patch releases for older lines.

Can Iceberg clients read Delta tables?

Yes, when Iceberg reads (UniForm) is enabled on a Unity Catalog table with column mapping. The Databricks docs list Databricks Runtime 14.3 LTS or above for writes to such a table.