What's new in Apache Spark 4.x?

Summary

• Apache Spark 4.x includes three feature releases: 4.0.0 (May 23, 2025), 4.1.0 (December 16, 2025) and 4.2.0 (July 14, 2026). Maintenance releases 4.0.4 and 4.1.3 followed in July 2026.
• Spark 4.0 changed the defaults (ANSI SQL mode on, JDK 17 and Scala 2.13 as the baseline) and added the VARIANT type, SQL UDFs, SQL pipe syntax, string collation and the Python Data Source API.
• Spark 4.1 added Spark Declarative Pipelines, made SQL scripting and VARIANT generally available, introduced Real-time Mode in Structured Streaming for stateless Scala queries, and raised the minimum Python version to 3.10.
• Spark 4.2 added GEOMETRY and GEOGRAPHY types, a SQL CHANGES clause for change data capture, Auto CDC in SDP, metric views and the NEAREST BY top-K join, and turned on Arrow-optimized Python UDFs by default.

What's new in Apache Spark 4.x?

As of October 2026, Apache Spark 4.x spans three feature releases. Spark 4.0 reset the defaults, with ANSI SQL on and JDK 17 and Scala 2.13 as the baseline, and added a batch of SQL and Python features. Spark 4.1 added a new component, Spark Declarative Pipelines, along with Real-time Mode for Structured Streaming. Spark 4.2 added geospatial types, change data capture in SQL, metric views and Arrow-optimized Python UDFs by default. Upgrading from 3.5, test the 4.0 default changes first.

Spark 4.x releases at a glance

Release	Date	Notable changes (from the release notes)
4.0.0	May 23, 2025	ANSI SQL mode by default; JDK 17 and Scala 2.13 by default (JDK 8/11, Scala 2.12 and Python 3.8 dropped); VARIANT type; SQL UDFs; SQL pipe syntax; string collation; Python Data Source API; Python UDTFs; transformWithState (Arbitrary State API v2); State Data Source reader
4.1.0	December 16, 2025	Spark Declarative Pipelines; SQL scripting GA and on by default; VARIANT GA with shredding; recursive CTEs; Real-time Mode in Structured Streaming (Scala, stateless); JDBC driver for Spark Connect; Python Arrow UDFs and UDTFs; minimum Python 3.10
4.2.0	July 14, 2026	GEOMETRY and GEOGRAPHY types with ST_* functions; CDC CHANGES clause; Auto CDC in SDP; Arrow-optimized Python UDFs on by default; NEAREST BY top-K join; metric views; QUALIFY; PySpark Real-time Mode trigger; named streaming sources and sinks; Java 25

Spark 4.0: new defaults and new SQL

The 4.0 change most likely to break an existing job is ANSI mode. Since Spark 4.0, spark.sql.ansi.enabled is on by default, so Spark throws an exception at runtime instead of returning null when a SQL operator or function gets invalid input. CAST('a' AS INT) now fails with CAST_INVALID_INPUT. You can set the flag back to false, but the cleaner fix is in the query: try_cast returns NULL instead of raising an error. CREATE TABLE without USING or STORED AS now uses spark.sql.sources.default instead of Hive.
SQL pipe syntax, also new in 4.0, writes a query as a chain of operators:

SELECT try_cast('a' AS INT) AS parsed;

FROM orders
|> WHERE status = 'shipped'
|> AGGREGATE COUNT(*) AS shipped_orders GROUP BY region
|> ORDER BY shipped_orders DESC;

Spark Connect matured in the same release, with a lightweight Python client (pyspark-client), a spark.api.mode setting (classic or connect) that runs a Spark Classic application through a local Spark Connect server, and ML on Spark Connect.

Spark 4.1: declarative pipelines and real-time streaming

• Spark Declarative Pipelines (SDP): you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries. The Python API is from pyspark import pipelines as dp, and the spark-pipelines CLI runs pipelines.
• SQL: scripting and VARIANT are GA and on by default, and recursive CTEs are new.
• Real-time Mode in Structured Streaming arrived for stateless Scala queries through Trigger.RealTime(...), an API the 4.1.0 source marks @Experimental.
• Python and Connect: Arrow-native UDF and UDTF decorators that skip Pandas conversion, Python Data Source filter pushdown, a JDBC driver for Spark Connect, and Spark ML on Connect GA for Python.

Spark 4.2: geospatial, CDC and semantic modeling

• Spark SQL: GEOMETRY and GEOGRAPHY types with ST_* functions, enabled by default. A CHANGES clause, with DataFrame, PySpark and Spark Connect APIs, reads row-level changes in batch and streaming. Metric views (CREATE VIEW ... WITH METRICS) and the NEAREST BY top-K ranking join are new too.
• SDP: Auto CDC for declarative SCD Type 1 upserts.
• PySpark: Arrow-optimized Python UDFs and Arrow-based PySpark IPC are on by default.
• Structured Streaming: the PySpark trigger trigger(realTime=...), plus DataStreamReader.name() (marked experimental) and SQL IDENTIFIED BY, which give sources stable names so they can be added, removed and reordered without breaking checkpoints.
• Core: Java 25 support, a modernized Web UI, and order-independent shuffle checksums on by default.

How Databricks approaches Spark 4.x

Databricks Runtime includes Apache Spark and adds its own components. According to the Databricks Runtime release notes, Databricks Runtime 17.3 LTS includes Apache Spark 4.0.0, 18 LTS includes 4.1.0, and 19 includes 4.2.0. Lakeflow Pipelines extend and interoperate with Spark Declarative Pipelines, and Real-time Mode for Structured Streaming is generally available on Databricks classic compute.

FAQs

Will my Spark 3.5 code run on Spark 4.x unchanged?

Not necessarily. The Spark SQL migration guide lists the behavior changes and, where one exists, the setting that restores the old behavior, such as spark.sql.ansi.enabled set to false.

Which Java and Python versions does Spark 4.x need?

Spark 4.0 dropped JDK 8 and 11, made JDK 17 the default and added Java 21 support; Spark 4.2 adds Java 25. Spark 4.0 dropped Python 3.8, and Spark 4.1 raised the minimum to Python 3.10.

Is Spark Declarative Pipelines the same as Lakeflow Pipelines?

No. SDP is the open-source Apache Spark framework. Lakeflow Pipelines is the Databricks product that extends and interoperates with it.

Which 4.x features are marked experimental?

The Spark docs label the State Data Source reader and DataStreamReader.name() experimental, and the 4.1.0 Real-time Mode trigger API carried an @Experimental annotation.