What's the state of Apache Spark in 2026?

Summary

• As of October 2026, Apache Spark 4.2.0, released July 14, 2026, is the newest feature release. Maintenance releases 4.1.3, 4.0.4 and 3.5.9 followed that week with security and correctness fixes.
• Spark 4.0 made ANSI SQL mode the default and moved to JDK 17 and Scala 2.13. Spark 4.1 added Spark Declarative Pipelines (SDP) and a Structured Streaming real-time mode. Spark 4.2 added GEOMETRY and GEOGRAPHY types, change data capture and metric views.
• Spark Connect, the client-server architecture introduced in Spark 3.4, keeps expanding: Spark ML on Connect went GA for the Python client in 4.1.
• The versioning policy moves Spark to quarterly feature releases starting with 4.3.0, and gives 3.5.x an extended LTS period, security fixes only, until November 2027.
• On Databricks, Databricks Runtime 19 is powered by Apache Spark 4.2.0, and serverless compute is versionless.

What's the state of Apache Spark in 2026?

As of October 2026, Apache Spark is on its 4.x line, and 4.2.0, released July 14, 2026, is the newest feature release. The 4.1, 4.0 and 3.5 lines each received a maintenance release (4.1.3, 4.0.4 and 3.5.9) the same week. The 4.x releases center on stricter SQL defaults, declarative pipelines, a real-time streaming mode, Arrow-based Python execution and a broader Spark Connect, and the project now plans quarterly feature releases.

Which Apache Spark releases are current?

Release	Date	Notable changes (from the release notes)
4.2.0	July 14, 2026	Third release of the 4.x line: GEOMETRY and GEOGRAPHY types with ST_* functions, change data capture through a SQL CHANGES clause, metric views, Arrow-optimized Python UDFs on by default, Java 25 support
4.1.3	July 15, 2026	Maintenance release with security and correctness fixes for the 4.1 line
4.0.4	July 15, 2026	Maintenance release with security and correctness fixes for the 4.0 line
3.5.9	July 16, 2026	Maintenance release with security and correctness fixes for the 3.5 line
4.1.0	December 16, 2025	Spark Declarative Pipelines, Structured Streaming real-time mode, SQL scripting GA, VARIANT GA
4.0.0	May 23, 2025	First 4.x release: ANSI SQL mode by default, JDK 17 and Scala 2.13 as defaults, SQL UDFs, SQL pipe syntax, string collation

What changed across Spark 4.0, 4.1 and 4.2?

SQL got stricter by default. Since Spark 4.0, spark.sql.ansi.enabled is on by default, so Spark throws an exception at runtime instead of returning null when inputs to a SQL operator or function are invalid. Spark 4.0 also dropped JDK 8/11 and Scala 2.12, making JDK 17 and Scala 2.13 the defaults.
Pipelines became part of Spark. Spark 4.1.0 added Spark Declarative Pipelines: you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries. SDP ships a spark-pipelines command line with init, run and dry-run. Spark 4.2.0 added Auto CDC to SDP for declarative SCD Type 1 upserts.
Streaming gained a real-time mode. Spark 4.1.0 shipped real-time mode in Structured Streaming with Scala stateless support, and 4.2.0 adds real-time mode trigger support to PySpark. Earlier, 4.0 introduced the Arbitrary State API v2 for more flexible state management and the State Data Source for debugging.
Python moved closer to Arrow. Spark 4.0 brought a Python Data Source API and a native plotting API. Spark 4.1 added Arrow-native UDF and UDTF decorators, and 4.2.0 turns on Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default.
Spark Connect kept growing. Spark Connect arrived in 3.4 as a decoupled client-server architecture and supports PySpark and Scala applications. Spark 4.0 added a lightweight Python client (pyspark-client) and the spark.api.mode configuration for switching applications onto Spark Connect. In 4.1, Spark ML on Connect became GA for the Python client, and 4.2 adds RDD API compatibility work such as DataFrame.zipWithIndex in PySpark.
Deployment options are unchanged. Spark runs on its own standalone cluster manager, Hadoop YARN or Kubernetes.

How is Spark released and supported now?

Spark versions follow [MAJOR].[FEATURE].[MAINTENANCE]. Under the current versioning policy, feature releases move to a quarterly schedule starting with 4.3.0, the first feature release after 4.2.0, and major releases happen annually. Major and feature releases get a 6-month maintenance window. The last feature release in each major line is the LTS release and is maintained for 18 months; 4.2.x keeps the 18-month window it was already promised. As an exception, 3.5.x has an extended LTS period, limited to security fixes, that ends November 2027.

How Databricks approaches Apache Spark

Databricks Runtime versions track Spark releases. Databricks Runtime 19, now generally available, is powered by Apache Spark 4.2.0; Databricks Runtime 18 LTS runs Spark 4.1.0, and 17.3 LTS runs Spark 4.0.0. Serverless compute is versionless, so Databricks upgrades its runtime automatically, and it runs your code through Spark Connect. Lakeflow Pipelines extend and are interoperable with open-source Spark Declarative Pipelines.

FAQs

What is the latest Apache Spark release?

Apache Spark 4.2.0, released July 14, 2026, is the newest feature release as of October 2026. The most recent maintenance releases are 4.1.3 and 4.0.4 (July 15, 2026) and 3.5.9 (July 16, 2026).

Is Spark 3.5 still maintained?

Yes, in a limited way. The versioning policy gives 3.5.x an extended LTS period that includes only security fixes and ends November 2027.

What should I check before upgrading from Spark 3.5 to 4.x?

Start with ANSI mode. Since Spark 4.0, spark.sql.ansi.enabled is on by default; to restore the previous behavior, set it to false. Then confirm your build runs on JDK 17 and Scala 2.13, and work through the "Upgrading from Spark SQL 3.5 to 4.0" section of the SQL migration guide.

Is Spark Declarative Pipelines part of open-source Spark?

Yes. SDP shipped in Apache Spark 4.1.0 as a declarative framework for building data pipelines on Apache Spark, with the spark-pipelines command line for creating and running pipeline projects.