SDP versus Lakeflow Declarative Pipelines: what's the naming in 2026?

Summary

• As of October 2026 there are two current names. Spark Declarative Pipelines (SDP) is the open-source framework inside Apache Spark, and Lakeflow Pipelines is the Databricks product built on it.
• "Lakeflow Declarative Pipelines" is an older name for the Databricks product. Databricks used it from June 2025, including in the Lakeflow general availability announcement on June 12, until November 12, 2025, when it was renamed Lakeflow Spark Declarative Pipelines.
• SDP shipped in Apache Spark 4.1.0 on December 16, 2025. Its Python API is from pyspark import pipelines as dp, and its CLI is spark-pipelines.
• Databricks documentation now says the product is called Lakeflow pipelines and the open-source framework it extends is Apache Spark Declarative Pipelines (SDP).
• Old code still runs. Databricks says DLT-era pipelines need no migration, and the dlt Python module still works, though Databricks recommends pyspark.pipelines.

SDP versus Lakeflow Declarative Pipelines: what's the naming in 2026?

It isn't really "SDP versus" anything. In 2026, SDP (Spark Declarative Pipelines) is the open-source Apache Spark framework for declaring streaming tables, materialized views and the flows between them. Lakeflow Pipelines is the current name of the managed Databricks product that runs on top of it. "Lakeflow Declarative Pipelines" was that product's name for about five months in 2025, so if you see it today, you're reading 2025 material. Use SDP for the open-source project and Lakeflow Pipelines for Databricks.

The two current names

Spark Declarative Pipelines (SDP). Apache Spark 4.1.0 added it as a declarative framework where "you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries." You install it with pip install pyspark[pipelines], define datasets in the pyspark.pipelines module, and run projects with spark-pipelines init, spark-pipelines run and spark-pipelines dry-run. Because spark-pipelines is built on spark-submit, it works with every cluster manager spark-submit supports.
Lakeflow Pipelines. The Databricks product. The Databricks Python reference states it plainly: "In Databricks documentation, the Databricks product is called Lakeflow pipelines, and the open-source framework it extends is Apache Spark™ Declarative Pipelines (SDP)." Note the docs' lowercase "pipelines."

How the names changed

Date	Name or change	Dated source
April 5, 2022	Delta Live Tables (DLT) becomes generally available on AWS and Azure	Databricks blog
June 13, 2024	Lakeflow is announced with three components; its pipelines component, Lakeflow Pipelines, is described as built on the Delta Live Tables framework	Introducing Databricks Lakeflow
June 12, 2025	Lakeflow Declarative Pipelines is generally available, "built on the new open Spark Declarative Pipelines standard" and backward-compatible with DLT pipelines	Lakeflow GA announcement
June 12, 2025	Databricks announces it is contributing Spark Declarative Pipelines to Apache Spark	Databricks blog
November 12, 2025	"Lakeflow Declarative Pipelines has been renamed to Lakeflow Spark Declarative Pipelines."	Release notes, November 2025
December 16, 2025	Apache Spark 4.1.0 ships SDP	Spark 4.1.0 release notes
2026	Databricks docs and release notes use Lakeflow pipelines, for example "Run Lakeflow pipelines as a group" (October 5, 2026)	Lakeflow pipelines docs, updated September 11, 2026

What the rename means in code

The Python module changed along with the product name, which is the quickest way to date a snippet. DLT-era code uses import dlt. Current code uses the SDP module, which both Apache Spark and Lakeflow Pipelines read:

from pyspark import pipelines as dp
from pyspark.sql import SparkSession

spark = SparkSession.active()  # required in Spark 4.1, optional from Spark 4.2

@dp.table  # streaming table; DLT-era code used @dlt.table on a streaming read
def raw_orders():
    return spark.readStream.table("orders_source")

@dp.materialized_view  # DLT-era code used @dlt.table on a batch read
def order_counts():
    return spark.read.table("raw_orders").groupBy("order_date").count()

Two more mappings from the Databricks reference: @dlt.view became @dp.temporary_view, and @dlt.append_flow became @dp.append_flow. In SQL, the Databricks reference lists the same CREATE STREAMING TABLE, CREATE MATERIALIZED VIEW and CREATE FLOW syntax for DLT, Lakeflow Pipelines and SDP.

How Databricks approaches SDP and Lakeflow Pipelines

Databricks describes Lakeflow Pipelines as built on SDP and interoperable with it: "Because pipelines build on SDP rather than proprietary APIs, the transformation code you write stays portable to other SDP runtimes." The product runs on Databricks Runtime and adds features SDP doesn't have, including AUTO CDC for SCD Type 2 and from snapshots, data quality expectations, a queryable event log, update flows, foreachBatch sinks and continuous mode. Some old names remain: the classic SKUs for Lakeflow Pipelines still begin with DLT, and event log schemas with dlt in the name haven't changed. See What happened to Delta Live Tables (DLT)?

FAQs

Is Lakeflow Declarative Pipelines still the product name?

No. Databricks renamed it Lakeflow Spark Declarative Pipelines on November 12, 2025, and its documentation now calls the product Lakeflow pipelines.

Are SDP and Lakeflow Pipelines the same thing?

No. SDP is the open-source framework in Apache Spark 4.1 and later. Lakeflow Pipelines is the Databricks product built on SDP; it shares the authoring model and adds managed features such as SCD Type 2 AUTO CDC and expectations.

Do I have to rewrite my DLT pipelines?

No. Databricks says no migration is required and existing DLT code still works. It does recommend moving to from pyspark import pipelines as dp, which also makes the code compatible with Apache Spark Declarative Pipelines.

Why do some 2026 release notes say "Spark Declarative Pipelines on Lakeflow"?

Release notes through July 2026, such as the February 5, 2026 note on pipeline-created datasets, use that phrase for the Databricks product. Entries from August 2026 on, and the current docs, say Lakeflow pipelines.