What SDP features shipped in the latest release?

Summary

• As of October 2026, the latest Apache Spark feature release is 4.2.0 (July 14, 2026). Its release notes list one Spark Declarative Pipelines (SDP) feature: Auto CDC, for declarative SCD Type 1 upserts (SPARK-56249).
• The Auto CDC group has four items: Python APIs for Auto CDC SCD Type 1, Auto CDC Spark Connect APIs, an SCD Type 1 streaming write, and registering Auto CDC flows from PipelinesHandler.
• The releases since then, 4.1.3, 4.0.4 and 3.5.9, are maintenance releases with security and correctness fixes.
• Metric views, the SQL CHANGES clause and streaming source and sink naming also arrived in 4.2.0, but the notes file them under Spark SQL and Structured Streaming, not SDP.

What SDP features shipped in the latest release?

As of October 2026, the latest Apache Spark feature release is 4.2.0, released July 14, 2026. Its release notes list one Spark Declarative Pipelines (SDP) feature: Auto CDC, which brings "declarative SCD Type 1 upserts" to pipelines (SPARK-56249). The releases published after it, 4.1.3, 4.0.4 and 3.5.9, are maintenance releases, so 4.2.0 holds the newest SDP functionality. SDP itself first shipped in Spark 4.1.0.

What do recent releases say about SDP?

Release	Date	SDP in the release notes
4.2.0	July 14, 2026	Auto CDC for declarative SCD Type 1 upserts, with Python and Spark Connect APIs (SPARK-56249)
4.1.3	July 15, 2026	A maintenance release "containing security and correctness fixes"
4.1.0	December 16, 2025	SDP added: "a new component to define and run data pipelines" (SPARK-51727)

The 4.0.4 and 3.5.9 maintenance releases are on the 4.0 and 3.5 branches, which predate SDP.

What is in the 4.2.0 Auto CDC group?

The "Auto CDC in Declarative Pipelines" section of the 4.2.0 notes lists four changes:

• SPARK-56651: "Add Python APIs for Auto CDC SCD Type 1"
• SPARK-56650: "Add Auto CDC Spark Connect APIs"
• SPARK-56957: "Introduce and integrate the SCD Type 1 streaming write (Scd1MergeStreamingWrite)"
• SPARK-57080: "Register Auto CDC flows from PipelinesHandler"

SCD Type 1 means a target keeps only the latest version of each record, with no history for updated rows. The 4.2.0 notes mention only Type 1 for Auto CDC.

How do you write SDP code on 4.2.0?

As of the 4.2.0 docs, neither the SDP programming guide nor the pyspark.pipelines API reference documents an Auto CDC function. The API reference lists append_flow, create_sink, create_streaming_table, materialized_view, table and temporary_view, so check the JIRA tickets above for the exact Auto CDC call. The rest of a pipeline uses the documented API, and the 4.2 guide notes one change for authors: Spark now injects spark into each pipeline file, so the spark = SparkSession.active() line that 4.1 required is optional.

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.table
def basic_st() -> DataFrame:
    return spark.readStream.table("samples.nyctaxi.trips")

pip install pyspark[pipelines]
spark-pipelines init --name my_pipeline
spark-pipelines run --remote sc://my-cluster:15002

@dp.table creates a streaming table from a streaming read, and spark-pipelines run --remote runs the pipeline against a Spark Connect server.

Which 4.2.0 features are not SDP features?

Several 4.2.0 highlights are easy to mistake for pipeline features. The release notes file them elsewhere:

• Metric views (SPARK-54119) sit under SQL Foundation. They're a Spark SQL feature.
• The SQL CHANGES clause and the DataFrame, PySpark and Spark Connect APIs for reading row-level changes (SPARK-55668) are Spark SQL and API features. Auto CDC is the pipeline piece of that CDC work.
• Streaming source and sink naming (SPARK-54909, SPARK-56719) and stream-stream non-outer joins in Update mode (SPARK-56384) are Structured Streaming features.

How Databricks approaches SDP

On Databricks, Lakeflow Pipelines "extend and are interoperable with Spark Declarative Pipelines". For change data capture, Lakeflow Pipelines offers the AUTO CDC and AUTO CDC FROM SNAPSHOT APIs, which compute SCD Type 1 and Type 2 from a CDC feed or from database snapshots. In SQL you write AUTO CDC ... INTO; in Python you call create_auto_cdc_flow(). These APIs replace the older APPLY CHANGES APIs, which have the same syntax. They need serverless Lakeflow pipelines or the Pro or Advanced editions.

FAQs

When did SDP first ship?

In Apache Spark 4.1.0, released December 16, 2025. Its release notes say, "This release adds Spark Declarative Pipelines (SDP)".

Does upstream Auto CDC handle SCD Type 2?

The 4.2.0 release notes mention only SCD Type 1. On Databricks, the Lakeflow Pipelines AUTO CDC APIs handle both Type 1 and Type 2.

Can Auto CDC run over Spark Connect?

The 4.2.0 notes list "Add Auto CDC Spark Connect APIs" (SPARK-56650).

Is a newer Spark feature release out?

No. The Spark news page lists 4.2.0 as the most recent feature release, followed only by maintenance releases.