What changed for Spark users in 2026?

Summary

• Apache Spark 4.2.0 was released on July 14, 2026, following 4.1.0 on December 16, 2025.
• Spark 4.1 added Spark Declarative Pipelines, Real-time Mode for Scala stateless queries, and GA SQL Scripting and VARIANT. Spark 4.2 added geospatial types, change data capture, metric views and Arrow-optimized Python UDFs by default.
• Check before upgrading: Python 3.10 or later, PyArrow 18.0.0 or later for PySpark 4.2, and the new Arrow defaults.
• Databricks Runtime 19 is powered by Spark 4.2.0, and Databricks Runtime 18 LTS by Spark 4.1.0.

What changed for Spark users in 2026?

Apache Spark 4.2.0 was released on July 14, 2026, the third release in the 4.x line. It follows Spark 4.1.0 (December 16, 2025), so this page covers both releases. Maintenance releases also shipped in 2026: 4.1.1 through 4.1.3, 4.0.2 through 4.0.4, and 3.5.8 and 3.5.9.

What shipped in Spark 4.1 and 4.2?

Area	Spark 4.1.0 (Dec 2025)	Spark 4.2.0 (Jul 2026)
Pipelines	Spark Declarative Pipelines (SDP), a new component to define and run data pipelines	Auto CDC (declarative SCD Type 1) in SDP
SQL	SQL Scripting and VARIANT are GA and on by default; recursive CTEs	GEOMETRY and GEOGRAPHY types; a CHANGES clause for change data capture; metric views (CREATE VIEW ... WITH METRICS); NEAREST BY join; opt-in SET PATH
Python	Arrow UDFs and UDTFs; minimum Python 3.10; Python 3.14 support	Arrow-optimized Python UDFs and Arrow-based data exchange on by default
Streaming	Real-time Mode (Scala, stateless queries)	Real-time Mode trigger support in PySpark; stream-stream non-outer joins in Update mode
Spark Connect	JDBC driver for Spark Connect; ML on Connect GA for the Python client	GetStatus API for execution status monitoring
Platform	Scala 2.13.17	Java 25 support; a Web UI with dark mode

What should I check before upgrading?

• Python: Spark 4.1 dropped Python 3.9 in PySpark and raised the minimum PyArrow to 15.0.0 and pandas to 2.2.0. It also maps BinaryType to Python bytes by default and enables spark.sql.execution.pandas.convertToArrowArraySafely, so PyArrow raises errors for unsafe conversions such as integer overflows. Spark 4.2 raises the PyArrow minimum to 18.0.0 and no longer officially supports PyPy.
• Arrow defaults: in 4.2, spark.sql.execution.arrow.pyspark.enabled, spark.sql.execution.pythonUDF.arrow.enabled and spark.sql.execution.pythonUDTF.arrow.enabled default to true. Set them to false to restore the legacy behavior. A pandas UDF that receives a nullable integer column with nulls now gets a pandas nullable integer dtype instead of float64.
• SQL: Spark 4.2 enables order-independent shuffle checksums by default, makes NATURAL JOIN honor spark.sql.caseSensitive, and returns one row for GROUP BY GROUPING SETS (()) over empty input. SET PATH is opt-in: spark.sql.path.enabled defaults to false.
• Runtime: the 4.2.0 docs say "Spark runs on Java 17/21/25, Scala 2.13, Python 3.10+, and R 4.0+ (Deprecated)."
• Coming from 3.5: Spark 4.0 had already dropped JDK 8 and 11 and turned ANSI SQL mode on by default.

How do I try Spark Declarative Pipelines?

Install it with pip install pyspark[pipelines]. The command spark-pipelines init --name my_pipeline generates a simple project with a spec file and example transformations. The spec file requires name, libraries and storage, a directory where checkpoints can be stored for streaming tables. spark-pipelines dry-run launches an execution that doesn't write or read any data but catches many kinds of errors, and spark-pipelines run runs the pipeline and monitors its progress.

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.materialized_view
def basic_mv() -> DataFrame:
    return spark.table("samples.nyctaxi.trips")

In Spark 4.1, every pipeline file had to declare spark = SparkSession.active(). Starting in Spark 4.2, the framework injects spark into each pipeline file's module namespace, as the example above relies on.

Which Spark versions still get fixes?

The Spark news page shows July 2026 maintenance releases for three lines: 3.5.9, 4.0.4 and 4.1.3. The versioning policy gives 3.5.x an extended LTS period that ends in November 2027 and includes only security fixes. Starting with 4.3.0, feature releases are quarterly and major releases are annual.

How do these releases map to Databricks Runtime?

Databricks Runtime	Apache Spark	Release date
19	4.2.0	Beta June 15, 2026; GA July 23, 2026
18 LTS	4.1.0	June 10, 2026
17.3 LTS	4.0.0	Oct 22, 2025
16.4 LTS	3.5.2	May 9, 2025

Databricks Runtime 19 "only supports JDK 21", and regular Python UDFs use Arrow-based serialization by default there too. To restore the earlier behavior, set spark.databricks.execution.pythonUDF.arrow.enabled to false.

FAQs

Is Spark Declarative Pipelines the same as Lakeflow Pipelines?

No. Spark Declarative Pipelines is the Apache Spark framework, imported in Python with from pyspark import pipelines as dp. The Databricks docs say Lakeflow pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime."

When did Python 3.9 support end?

In Spark 4.1. The PySpark upgrade guide says that in 4.1 "Python 3.9 support was dropped in PySpark."

Is the SQL PATH feature on by default in Spark 4.2?

No. It is opt-in, and SET PATH is rejected while spark.sql.path.enabled is false.