What's the future of Apache Spark?

Summary

• As of October 2026, the latest Apache Spark release is 4.2.0 (July 14, 2026), the third release of the 4.x line after 4.0.0 (May 23, 2025) and 4.1.0 (December 16, 2025).
• The release notes show where Spark is moving: Spark Declarative Pipelines and Real-time Mode in 4.1, geospatial types and Change Data Capture in 4.2, and wider Spark Connect coverage in every 4.x release.
• Spark Connect supports PySpark and Scala applications, and the docs list stability, upgradability and debuggability as its benefits.
• The versioning policy says that starting with Spark 4.3.0, feature releases are quarterly and major releases are annual.
• Major improvements go through the SPIP process, which needs a vote with at least 3 +1 votes from PMC members and no -1 votes.

What's the future of Apache Spark?

As of October 2026, the latest Apache Spark release is 4.2.0, published on July 14, 2026. It is the third release of the 4.x line, after 4.0.0 (May 23, 2025) and 4.1.0 (December 16, 2025). Spark's direction shows up in three official places: the release notes, the SPIP process that approves major changes, and the versioning policy. This page sticks to what those sources state and to features that have shipped.

What has the 4.x line shipped so far?

Release	Date	Notable changes in the release notes
4.0.0	May 23, 2025	Scala 2.13 and JDK 17 as defaults (JDK 8 and 11 dropped), ANSI SQL mode on by default, VARIANT data type, SQL pipe syntax, Spark Kubernetes Operator, Spark Connect improvements, Python Data Source API
4.1.0	December 16, 2025	Spark Declarative Pipelines, Real-time Mode in Structured Streaming (Scala stateless support), SQL Scripting and VARIANT enabled by default, JDBC driver for Spark Connect, minimum Python raised to 3.10
4.2.0	July 14, 2026	GEOMETRY and GEOGRAPHY types, Change Data Capture (SQL CHANGES clause, Auto CDC in Spark Declarative Pipelines), Arrow-optimized Python UDFs on by default, metric views, Data Source V2 transaction management, Java 25 support

How is Spark Connect changing application architecture?

Spark Connect uses a decoupled client-server architecture. The 3.4.0 release notes list Python and Scala clients for it. The docs name three benefits: stability (an application that uses too much memory only affects its own process), upgradability (the Spark driver can be upgraded independently of applications) and debuggability and observability. Spark Connect supports PySpark and Scala applications and the majority of the Structured Streaming API.
Later releases widened it. Spark 4.0.0 added a lightweight Python client (pyspark-client) and ML on Spark Connect. Spark 4.1.0 made Spark ML on Connect GA for the Python client and added a JDBC driver for Spark Connect. Spark 4.2.0 added a server-side GetStatus API. After starting a server with ./sbin/start-connect-server.sh, a client connects like this:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

What is changing in Structured Streaming?

Spark 4.1.0 added Real-time Mode (Scala stateless support), and Spark 4.2.0 adds Real-time Mode trigger support to PySpark. State management keeps evolving too. The default HDFS-backed state store keeps state in an in-memory map backed by files. Since Spark 3.2 you can set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider, which the docs introduce for stateful queries that keep millions of keys and see long JVM garbage collection pauses. Spark 4.2.0 adds automatic snapshot repair and row checksums for the state store.

Where does Spark run, and what does it need?

The docs list three cluster managers: Standalone, Hadoop YARN and Kubernetes. Spark 4.0.0 dropped Mesos support. Kubernetes work in the release notes includes the Spark Kubernetes Operator (4.0.0) and, in 4.2.0, Deployment API support, built-in executor resize plugins and a Kubernetes resource-manager API promoted to Stable. Spark 4.2.0 runs on Java 17, 21 or 25, Scala 2.13 and Python 3.10 or later. The R API is deprecated.

How are new features decided?

Major changes go through Spark Project Improvement Proposals (SPIPs). Anyone may propose one, and the vote is open for at least 72 hours and passes with at least 3 +1 votes from PMC members and no -1 votes from PMC members. Several 4.x highlights were SPIPs, including Declarative Pipelines, Real-time Mode, the JDBC driver for Spark Connect, geospatial types and the Python Data Source API.
The versioning policy sets the cadence. Starting with Spark 4.3.0, feature releases occur quarterly and major releases occur annually, with maintenance releases as needed. The policy also gives an illustrative calendar for the transition and says exact dates stay subject to the usual release-discussion and voting process.

How does Databricks relate to Apache Spark?

Databricks was founded in 2013 by the team that started the Spark research project at UC Berkeley, and it says it continues to contribute to the Apache Spark project. Databricks Runtime 17.3 LTS includes Apache Spark 4.0.0, Databricks Runtime 18 LTS includes 4.1.0, and Databricks Runtime 19 includes 4.2.0. Photon is a Databricks-native vectorized query engine that is compatible with Apache Spark APIs, so existing code runs without changes.

FAQs

When will Spark 5.0 ship?

No release date has been announced. The versioning policy says major releases occur annually and describes its calendar as an illustration, with exact dates subject to release discussion on the dev list and a vote.

Will Spark 4.x break my existing code?

Check the removals. Spark 4.0.0 dropped JDK 8 and 11, Scala 2.12, Python 3.8 and Mesos support, deprecated SparkR and turned ANSI SQL mode on by default. Spark 4.1.0 raised the minimum Python version to 3.10, and Spark 4.2.0 dropped R 3.x. Test in a staging environment before you upgrade.

Is Spark still actively developed?

Yes. The 4.2.0 release notes say the release addressed over 1,700 Jira tickets with contributions from more than 250 individuals. In July 2026 the project also released 4.1.3, 4.0.4 and 3.5.9.

What should I learn now?

Spark SQL and DataFrames. The Quick Start says RDDs were replaced by Dataset after Spark 2.0 and recommends switching to Dataset. For new applications, try Spark Connect.