Is SDP GA on Databricks?

Summary

• Yes. On Databricks, Spark Declarative Pipelines (SDP) is delivered as Lakeflow Pipelines, which is generally available. Databricks release notes record its GA on AWS and Microsoft Azure on April 5, 2022 and on Google Cloud on August 2, 2023, under the product's earlier name.
• In Apache Spark, SDP shipped as a new component in Spark 4.1.0 (December 16, 2025). Spark 4.2.0 (July 14, 2026) added Auto CDC for declarative SCD Type 1 upserts.
• GA describes the product, not every option inside it. Individual pipeline features can carry their own preview label, so check the docs page for the feature you plan to use.
• Code written with the open-source pyspark.pipelines module runs without modification on Databricks. Expectations (@dp.expect), AUTO CDC from snapshot and continuous mode are Databricks additions that a plain Spark cluster won't run.

Is SDP GA on Databricks?

Yes. On Databricks, Spark Declarative Pipelines (SDP) is delivered as Lakeflow Pipelines, and Lakeflow Pipelines is a generally available product. Databricks release notes record general availability on AWS and Microsoft Azure on April 5, 2022 and on Google Cloud on August 2, 2023, under the product's earlier name, and the later renames required no migration. What can still be in preview is a specific feature, so check the label on the capability you depend on before you build on it.

What does SDP mean on Databricks and in Apache Spark?

SDP is the open-source declarative framework in Apache Spark: you define streaming tables, materialized views and the flows that feed them, and the framework works out the dependency graph and execution order. Databricks documents Lakeflow Pipelines as built on SDP and interoperable with it, so transformation code stays portable to other SDP runtimes.
The upstream status is clear from the release notes. Apache Spark 4.1.0 added SDP as "a new component to define and run data pipelines" (SPARK-51727). Spark 4.2.0 added Auto CDC for declarative SCD Type 1 upserts, with Python and Spark Connect APIs (SPARK-56249). The 4.1.0 notes call some features GA, such as SQL Scripting and the VARIANT type, but introduce SDP without a GA or experimental tag, and the Spark 4.2.0 programming guide carries no experimental marker. It ships in official releases: install it with pip install pyspark[pipelines] and run it with the spark-pipelines CLI.
The same pipeline file runs in both places. In Spark 4.1 each file must create its session with SparkSession.active(); from Spark 4.2 the framework injects spark for you, and the explicit line still works.

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()  # required in Spark 4.1, optional from 4.2

@dp.table
def events_raw() -> DataFrame:
    return spark.readStream.table("raw_events")

@dp.materialized_view
def events_by_type() -> DataFrame:
    return spark.table("events_raw").groupBy("event_type").count()

To try it on open-source Spark, generate a project and run it from the project directory:

pip install pyspark[pipelines]
spark-pipelines init --name my_pipeline
cd my_pipeline && spark-pipelines run

How to check the status of what you plan to use

Layer	Status as of October 2026	Where to confirm it
Lakeflow Pipelines on Databricks	Generally available on AWS, Azure and Google Cloud	Databricks platform release notes
Individual pipeline features	Labeled feature by feature, for example GA or Public Preview	The label at the top of each feature's docs page, or next to a parameter
Pipeline runtime	Versionless; the channel setting picks current (default) or preview	Lakeflow Pipelines release notes and the pipeline properties reference
SDP in Apache Spark	In official releases since Spark 4.1.0; Auto CDC for SCD Type 1 added in 4.2.0	spark.apache.org release notes for your version
Portability to open-source Spark	Open-source pyspark.pipelines code runs on Databricks unchanged	Databricks' list of APIs that aren't part of Apache Spark

How Databricks approaches SDP

Lakeflow Pipelines shares SDP's authoring model and adds production features on top: AUTO CDC for SCD Type 2 and from snapshots, data quality expectations, a queryable event log, update flows and foreachBatch sinks, and continuous mode. Those are the parts that won't move to a self-managed Spark cluster.
The runtime is managed for you. Lakeflow Pipelines is versionless: Databricks upgrades the pipeline runtime automatically. The optional channel field takes current, the default and the one Databricks recommends for production, or preview, to test a pipeline against upcoming runtime changes. Per the release notes, features marked generally available or Public Preview are available in the current channel. See Apache Spark Declarative Pipelines and the release process.

FAQs

Is Lakeflow Pipelines the same thing as SDP?

No, but it's built on it. Lakeflow Pipelines uses the same declarative model (streaming tables, materialized views, flows) and adds managed-service features such as expectations, the event log and continuous mode.

Does open-source SDP code run on Databricks?

Yes. Databricks' Python reference states that code written with the open-source pipelines module runs without modification on Databricks. The reverse isn't guaranteed, because dp.create_auto_cdc_from_snapshot_flow and @dp.expect(...) aren't part of Apache Spark.

Which Apache Spark version do I need for SDP?

Spark 4.1.0 or later. Spark 4.2.0 adds Auto CDC for SCD Type 1 and injects the spark session into pipeline files automatically.

Do I have to migrate older pipelines to stay on the GA product?

No. Databricks says no migration is required after the rename and existing code keeps working. It recommends moving to the pyspark.pipelines module names for compatibility with Apache Spark SDP.