How does SDP relate to DLT and Lakeflow?

Summary

• SDP, short for Spark Declarative Pipelines, is an Apache Spark component for defining and running data pipelines. Its first release was Apache Spark 4.1.0 on December 16, 2025.
• DLT (Delta Live Tables) is the original Databricks product SDP grew out of. Databricks announced it at Data + AI Summit in May 2021, and it became generally available on AWS and Azure in April 2022.
• Lakeflow Pipelines is the current name of the Databricks product once called DLT. It is built on SDP, runs on Databricks Runtime and adds production features such as AUTO CDC, data quality expectations and a queryable event log.
• The product name changed in steps: DLT, then Lakeflow Declarative Pipelines (June 2025), then Lakeflow Spark Declarative Pipelines (November 2025), then Lakeflow Pipelines, the name in current Databricks docs.
• Existing DLT code still runs without migration. Databricks recommends moving from import dlt to from pyspark import pipelines as dp, which is also the SDP API.

How does SDP relate to DLT and Lakeflow?

They are one lineage under three names. DLT was the declarative pipeline product Databricks launched in 2021. In June 2025, Databricks announced it was donating its declarative pipeline framework to Apache Spark, where it shipped as Spark Declarative Pipelines (SDP) in Spark 4.1. The Databricks product formerly called DLT is now Lakeflow Pipelines, which is built on SDP and extends it. In short: SDP is the open-source framework, Lakeflow Pipelines is the managed Databricks product, and DLT is that product's old name.

Timeline of the names

Date	What happened	Source
May 27, 2021	Databricks announces Delta Live Tables (DLT) at Data + AI Summit, with a gated public preview	Databricks blog
April 5, 2022	DLT becomes generally available on AWS and Azure	Databricks blog
June 12, 2025	Databricks announces it is donating Declarative Pipelines to Apache Spark, and Lakeflow Declarative Pipelines reaches GA, built on SDP and backward-compatible with DLT pipelines	Databricks blog
November 12, 2025	Lakeflow Declarative Pipelines is renamed Lakeflow Spark Declarative Pipelines	Databricks release notes
December 16, 2025	Apache Spark 4.1.0 ships SDP (SPARK-51727)	Apache Spark release notes
July 14, 2026	Apache Spark 4.2.0 adds Auto CDC for SCD Type 1 to SDP	Apache Spark release notes
As of October 2026	Databricks docs: the product "formerly known as Delta Live Tables (DLT) has been updated to Lakeflow pipelines"	Databricks docs

The current product name is Lakeflow Pipelines. DLT, Delta Live Tables and the two 2025 names are history.

What is SDP in open-source Apache Spark?

The Spark docs define SDP as "a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark." You declare datasets (streaming tables, materialized views, temporary views) and the flows that fill them, and SDP works out dependency order, execution, checkpoints and retries. It installs with pip install pyspark[pipelines], and the spark-pipelines CLI provides init, run and dry-run commands that work with a spark-pipeline.yml spec file.

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()

@dp.table
def basic_st() -> DataFrame:
    return spark.readStream.table("samples.nyctaxi.trips")

@dp.materialized_view
def basic_mv() -> DataFrame:
    return spark.table("samples.nyctaxi.trips")

@dp.table creates a streaming table and @dp.materialized_view a materialized view. The explicit SparkSession.active() line was required in Spark 4.1; Spark 4.2 injects spark into pipeline files automatically, and files that keep the line still work.

How DLT code maps to the current API

Databricks documents the rename at the code level too:

• import dlt becomes from pyspark import pipelines as dp, and @dlt decorators become @dp.
• @table now creates streaming tables, and the new @materialized_view creates materialized views.
• @view is now @temporary_view.

No migration is required: DLT code keeps working in Lakeflow Pipelines, and the dlt Python names remain usable, though Databricks recommends the new names. A few traces of the old name remain: classic SKUs still begin with DLT, and event log schemas with dlt in the name haven't changed.

How Databricks approaches SDP with Lakeflow Pipelines

Lakeflow Pipelines are built on SDP, run on Databricks Runtime and stay interoperable with SDP. Because the authoring model is SDP's rather than a proprietary API, transformation code stays portable to other SDP runtimes. Both share streaming tables, materialized views, temporary views, append flows, sinks, and automatic orchestration and dependency resolution.
On top of that shared model, Databricks documents production features that Lakeflow Pipelines add: AUTO CDC (SCD Type 1 and Type 2) and AUTO CDC from snapshot, data quality expectations, a queryable event log, update flows and foreachBatch sinks, and continuous mode. The open-source side keeps moving: the Spark 4.2.0 release notes add Auto CDC for SCD Type 1 upserts to SDP.

FAQs

Is DLT deprecated?

The name is retired; the code isn't. DLT is the old name of the product now called Lakeflow Pipelines, and Databricks says no migration is required: existing DLT code still works and the dlt Python APIs can still be used, though Databricks recommends the pyspark.pipelines names.

Can I run Lakeflow Pipelines code on open-source Spark?

Code that sticks to the shared SDP model, such as streaming tables, materialized views, temporary views and append flows, is portable to SDP runtimes. Features Databricks lists as Lakeflow additions, such as expectations and the queryable event log, need Databricks.

Which Spark version includes SDP?

SDP first shipped in Apache Spark 4.1.0 on December 16, 2025. Apache Spark 4.2.0, released July 14, 2026, added Auto CDC for SCD Type 1 and injects the Spark session into pipeline files.

Is SDP the same as Structured Streaming?

No. SDP is a pipeline framework that sits above Spark's batch and streaming APIs. Streaming tables read with Structured Streaming, and SDP can load from any format that spark.readStream supports.