What performance features are new in Spark 4.x, and is Photon relevant to OSS Spark?

Summary

• As of October 2026, Spark 4.x has three feature releases: 4.0.0 (May 23, 2025), 4.1.0 (December 16, 2025) and 4.2.0 (July 14, 2026). Each one's release notes list optimizer, execution, Python and streaming performance work.
• For PySpark users, the visible change is Arrow: Spark 4.1.0 added Arrow-native UDF and UDTF decorators, and Spark 4.2.0 turned on Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default.
• Spark 4.2.0 also brings planner and execution work such as subplan merging, pre-aggregation before Expand for multiple COUNT(DISTINCT), whole-stage codegen fusion for UnionExec, and vectorized Parquet reader optimizations.
• Photon isn't part of open-source Spark. It's a proprietary, Databricks-native vectorized engine inside the Databricks Runtime, so a self-managed Spark 4.x cluster doesn't have it.
• Photon still matters when you choose where to run Spark code. It accepts the same SQL and DataFrame APIs with no code changes, keeps Catalyst for planning, and falls back to the Spark runtime for operations it doesn't support.

What performance features are new in Spark 4.x, and is Photon relevant to OSS Spark?

Spark 4.x performance work is spread across three releases, and the release notes are the authoritative list. For PySpark users the visible change is Arrow: Spark 4.2.0 made it the default path for Python UDFs and PySpark data exchange, after 4.1.0 added Arrow-native UDF decorators. Planner and execution improvements land in every release. Photon isn't relevant to a self-managed cluster in the sense that you can't run it there: it's a proprietary engine in the Databricks Runtime, not an Apache Spark component.

Performance changes by release

Everything below is quoted or paraphrased from the official Apache Spark release notes, as of October 2026.

Release	Notable performance and resource-use changes
Spark 4.0.0 (May 23, 2025)	Broadcast of multiple filtering keys in dynamic partition pruning (SPARK-46946); relaxed restrictions on multi-layer runtime filters (SPARK-45606); improved Storage Partition Join (SPARK-51938); shuffle cleanup based on ShuffleCleanupMode (SPARK-47764); concurrency level for Arrow-optimized Python UDFs (SPARK-50446)
Spark 4.1.0 (December 16, 2025)	Python Arrow UDF and UDTF (SPARK-52214, SPARK-52979); filter pushdown for Python data sources (SPARK-51271); VARIANT GA, with shredding the notes describe as speeding up reads of semi-structured data; AQE in stateless streaming (SPARK-53941); Unix domain sockets between Python and the JVM (SPARK-51688); RocksDB memory counted in the unified memory manager (SPARK-53001); Real-time Mode in Structured Streaming, Scala stateless support (SPARK-53736)
Spark 4.2.0 (July 14, 2026)	Arrow-optimized Python UDFs and Arrow-based PySpark IPC on by default (SPARK-54555); vectorized Parquet reader optimizations (SPARK-55517); merging subplans with different filter conditions (SPARK-40193); pushing joins through Union when the right side is broadcastable (SPARK-56034); pre-aggregation before Expand for multiple COUNT(DISTINCT) (SPARK-56315); whole-stage codegen fusion for UnionExec (SPARK-56482); BroadcastHashJoin skew handling in OptimizeSkewedJoin (SPARK-44065); optimized Py4J calls in toPandas and createDataFrame (SPARK-54300); Real-time Mode trigger support in PySpark (SPARK-54660)

Two notes. Real-time Mode is a Structured Streaming trigger, not a Spark Declarative Pipelines feature. Arrow by default covers regular Python UDFs and UDTFs plus columnar PySpark-JVM data exchange (spark.sql.execution.arrow.pyspark.enabled), per the PySpark migration guide.
The 4.2.0 notes also list stability work, such as a bounded k-way merge in UnsafeExternalSorter to reduce out-of-memory risk (SPARK-56410).

What is Photon, and does it apply to open-source Spark?

Photon is the Databricks-native vectorized query engine. Databricks documents it as one of the proprietary features the Databricks Runtime adds on top of Apache Spark, so it isn't in the Apache Spark distribution and there's nothing to enable on a self-managed cluster.
How it relates to Spark matters for portability. For supported operations, Photon replaces the JVM-based Spark SQL execution engine with a native C++ runtime that processes columnar batches. Catalyst, Spark's optimizer, still plans the query. When Photon meets an unsupported operation, it falls back to the Spark runtime for the rest of that operation and the query still produces correct results. Photon is compatible with Spark APIs, so the same SQL and DataFrame code runs with or without it.

Question	Open-source Spark 4.x	Databricks with Photon
Execution layer for SQL and DataFrames	Spark's JVM engine with whole-stage codegen	Photon's native C++ engine for supported operators, Spark runtime otherwise
Query planning	Catalyst	Catalyst
Code changes needed	Not applicable	None
UDFs, RDD APIs, Dataset APIs	Run on Spark's engine	Run on the Spark runtime; Photon doesn't support them
Stateful streaming	Supported by Structured Streaming	Runs on the Spark runtime; Photon supports stateless streaming only

How Databricks approaches Spark performance

Databricks Runtime releases track Apache Spark: Databricks Runtime 19, for example, is powered by Apache Spark 4.2.0. Photon is enabled on serverless compute, SQL warehouses and serverless Lakeflow Pipelines, and Databricks enables it by default on classic all-purpose and jobs compute, where a checkbox turns it on or off. Some Databricks features require Photon, including predictive I/O and dynamic file pruning in MERGE, UPDATE and DELETE. To see what ran on Photon, look for Photon operators in the Spark UI query DAG or check the query profile. See What is Photon?.

FAQs

Can I install Photon on open-source Spark?

No. Photon is a proprietary part of the Databricks Runtime and isn't distributed with Apache Spark. Self-managed clusters run Spark's own engine.

Does Spark code need changes to run on Photon?

No. Photon is compatible with Apache Spark APIs and needs no code changes. Operations it doesn't support, such as UDFs, fall back to the Spark runtime.

Which Spark 4.x changes matter for PySpark?

The Arrow work: Arrow-native UDF and UDTF decorators in 4.1.0, Unix domain sockets between Python and the JVM in 4.1.0, and Arrow-optimized Python UDFs on by default in 4.2.0.

Is Real-time Mode an SDP feature?

No. In Apache Spark, it's a Structured Streaming trigger. Spark 4.1.0 shipped it with Scala stateless support, and Spark 4.2.0 added PySpark trigger support.