Apache Spark in 2026

Summary

• Apache Spark 4.2.0 was released on July 14, 2026. It is the third release in the 4.x line, after 4.0.0 (May 23, 2025) and 4.1.0 (December 16, 2025).
• The 4.2.0 release notes highlight geospatial types, Change Data Capture (CDC), Arrow-optimized Python UDFs on by default, a NEAREST BY join, and metric views.
• 2026 also brought patch releases for the 3.5, 4.0 and 4.1 lines.
• Databricks's runtime table lists Spark 4.2.0 for Databricks Runtime 19 and Spark 4.1.0 for Databricks Runtime 18 LTS.

Apache Spark in 2026

As of October 2026, Spark 4.2.0 is the newest feature release on spark.apache.org, and the "latest" documentation is the 4.2.0 documentation. The release notes call it "the third release of the Spark 4.x line".

Which Spark releases shipped in 2026?

The project's news page lists these stable releases for 2026: 4.1.1 (January 9), 3.5.8 (January 15), 4.0.2 (February 5), 4.1.2 (May 21), 4.0.3 (June 11), 4.2.0 (July 14), 4.1.3 and 4.0.4 (July 15), and 3.5.9 (July 16). Preview builds of 4.2.0 were posted from January 11 to May 1.

Release	Date	What its release notes lead with
4.0.0	May 23, 2025	First 4.x release: Scala 2.13 and JDK 17 as defaults, ANSI SQL mode on by default, a 1.5 MB Python Spark Connect client
4.1.0	December 16, 2025	Spark Declarative Pipelines, Structured Streaming Real-Time Mode (Scala stateless), SQL Scripting GA, VARIANT GA
4.2.0	July 14, 2026	Geospatial types, CDC, Arrow-optimized Python UDFs by default, NEAREST BY, metric views

What is new in Spark 4.2?

Feature	What the 4.2.0 release notes say	JIRA
Geospatial types	GEOMETRY and GEOGRAPHY types with ST_* functions, WKB/WKT and Parquet I/O, and an SRID registry, enabled by default	SPARK-51658
Change Data Capture	A SQL CHANGES clause plus DataFrame, PySpark and Spark Connect APIs for row-level changes in batch and streaming; Auto CDC (SCD Type 1) in Spark Declarative Pipelines	SPARK-55668, SPARK-56249
Arrow by default	Arrow-optimized Python UDFs and Arrow-based PySpark IPC enabled by default	SPARK-54555
NEAREST BY	A top-K ranking join for nearest-neighbor queries	SPARK-56395
Metric views	Declarative semantic modeling with a CREATE VIEW ... WITH METRICS surface	SPARK-54119
Streaming names	DataStreamReader.name() and IDENTIFIED BY name sources so they can be added, removed and reordered without breaking checkpoints	SPARK-54909
Web UI	Dark mode and searchable, zoomable, side-by-side SQL plan visualization	SPARK-55760
Java 25	Build and run Spark on Java 25	SPARK-51167

Real-Time Mode arrived in 4.1.0 for Scala stateless queries, and 4.2.0 adds Real-Time Mode trigger support to PySpark (SPARK-54660).

What limits come with the new 4.2 features?

The 4.2 docs add some limits that the headlines leave out. For CDC, the PySpark reference shows this call:

spark.read.option("startingVersion", "10").option("endingVersion", "20").changes("my_table")

• CDC reads need a connector. DataFrameReader.changes returns row-level changes only for Data Source V2 tables whose catalog implements TableCatalog.loadChangelog().
• NEAREST BY is not index-backed yet. The JOIN reference gives the clause as { APPROX | EXACT } NEAREST [ num_results ] BY { DISTANCE | SIMILARITY } ranking_expression, with INNER and LEFT OUTER joins supported. It says the current implementation evaluates the full cross-product of both sides and that index-backed strategies are planned for a future release, so pre-filter a large right side.
• Geospatial is types plus five functions. The 4.2.0 built-in function list shows five ST functions: st_asbinary, st_geogfromwkb, st_geomfromwkb, st_setsrid and st_srid. SQL columns need an explicit SRID or ANY, such as GEOGRAPHY(4326), and the mixed-SRID ANY forms cannot be persisted in Parquet, Delta or Iceberg.
• SQL PATH is opt-in. SET PATH works only when spark.sql.path.enabled is true; the default is false.

What should I check before upgrading to 4.2?

Change in 4.2	What the migration guides say
PyArrow	Minimum version raised from 15.0.0 to 18.0.0
Python UDFs and UDTFs	Arrow-optimized by default; set spark.sql.execution.pythonUDF.arrow.enabled or spark.sql.execution.pythonUDTF.arrow.enabled to false for the legacy path
pandas UDFs	A nullable integer column with nulls arrives as a pandas nullable integer dtype, not float64
Shuffle checksums	Order-independent checksums are on by default and trigger a rollback and retry on a mismatch
Grouping sets	GROUP BY GROUPING SETS (()) now returns one row over empty input
PyPy	No longer officially supported

How does Databricks relate to Spark 4.2?

Databricks's Runtime release notes list the Apache Spark version for each runtime: Databricks Runtime 19 lists 4.2.0, 18 LTS lists 4.1.0, and 17.3 LTS lists 4.0.0. The Databricks docs describe Lakeflow Pipelines as extending, and interoperable with, Spark Declarative Pipelines while running on Databricks Runtime.

FAQs

Is geospatial support opt-in?

The 4.2.0 release notes say geospatial support is enabled by default (SPARK-56771). You use it by declaring GEOMETRY or GEOGRAPHY columns.

Did Spark 4.1 get skipped?

No. Spark 4.1.0 was released on December 16, 2025, and 4.1.3 shipped on July 15, 2026, the day after 4.2.0.

Where are metric views documented?

The 4.2.0 release notes list them under SPARK-54119, with CREATE and SELECT resolution (SPARK-54405) and YAML serde (SPARK-54403). The CREATE VIEW reference page did not describe a WITH METRICS form when checked, so use the release notes and JIRA for the current detail.