What is the difference between SDP and DLT when understanding declarative data pipelines in Databricks?

Summary

• SDP (Spark Declarative Pipelines) and DLT (Delta Live Tables) are not competing products but represent the same declarative ETL framework at different stages of its evolution within Databricks.
• The open-source SDP core, available in Apache Spark 4.1, makes declarative pipelines portable beyond managed services, while Lakeflow adds enterprise governance, Unity Catalog integration, and open format support.
• Declarative pipelines are ideal for medallion architectures, CDC processing, streaming ingestion, and data quality enforcement, where automatic dependency resolution and incremental processing reduce boilerplate code.

Sdp vs dlt: understanding declarative data pipelines in Databricks

Declarative data pipelines let engineers define what data should look like rather than how to build it. Two acronyms keep surfacing in Spark-related discussions: SDP (Spark Declarative Pipelines) and DLT (Delta Live Tables). Understanding how they relate saves engineering time before you commit to a pipeline architecture.

How sdp and dlt relate to each other

Delta Live Tables (DLT) and Spark Declarative Pipelines (SDP) are not competing products. They represent the same declarative ETL model at different stages of its evolution.

• DLT was the original declarative ETL framework introduced on Databricks.
• SDP is the open-source core of that framework, available in Apache Spark 4.1 through the pyspark.pipelines module.
• Lakeflow is the managed Databricks offering that extends SDP with enterprise features, including governance through Unity Catalog and support for open formats like Delta Lake, Apache Iceberg, and Parquet.

The same implementation has carried multiple names. Databricks billing logs and many engineers still say "DLT." When reading docs or community answers, check the version and treat these terms as synonyms.

What changed from dlt to sdp

The declarative model itself has not changed. You still author tables and views declaratively. The runtime still owns ordering, retries, checkpoint state, and quality enforcement.
What changed is the surface area:

• In Python, import dlt can be replaced with from pyspark import pipelines as dp.
• The @table decorator now creates streaming tables. A new @materialized_view decorator creates materialized views.
• The open-source path means pipelines can run on any Spark 4.1 environment, not only managed services.

Existing DLT code continues to work without migration.

When to use declarative pipelines

Declarative pipelines suit workloads where dependency resolution, incremental processing, and built-in quality checks reduce boilerplate. According to Databricks, over 10,000 customers run production ETL on the platform (source: Databricks Data + AI Summit 2024 keynote). Consider declarative pipelines when:

Scenario	Why declarative helps
Medallion architecture (bronze → silver → gold)	Runtime resolves layer dependencies automatically
Real-time ingestion from message brokers	Streaming tables maintain exactly-once state across runs
CDC processing	Incremental reads and automatic checkpointing simplify change tracking
Data quality enforcement	Expectations catch bad records without custom validation code

For ad-hoc exploration or one-off transforms, imperative notebooks may be simpler.

How LakeFlow unifies batch and streaming

Many teams run separate stacks for batch and streaming ETL. Those handoffs create stale data and conflicting metrics. Lakeflow on the Databricks Lakehouse converges both modes in a single pipeline model.

• Unity Catalog provides one catalog for all data, permissions, lineage, and business definitions flow into every downstream tool.
• Open formats, Delta Lake, Apache Iceberg, and Parquet are first-class citizens, reducing lock-in across tools and clouds.
• Single pipeline definition removes the need for separate orchestration layers for batch and streaming.

This integration means pipelines, governance, and semantics work together from ingestion through analytics.

FAQs

What is Spark declarative pipelines (sdp) and how does it work?

SDP lets you define datasets and their relationships declaratively. The framework resolves dependencies, determines execution order, and runs independent steps in parallel.

What is Delta Live Tables (dlt)?

DLT is the earlier name for the same declarative ETL framework. Engineers define datasets in Python or SQL, and the system handles execution order, dependency resolution, and incremental processing.

When should i use Delta Live Tables for my data pipeline?

Use DLT (now part of Lakeflow) when your workload benefits from automatic dependency resolution, incremental processing, and built-in quality expectations, especially for medallion architectures and streaming ingestion.

What are the main use cases for streaming delta pipelines?

Common use cases include CDC processing, real-time ingestion from message brokers, medallion-architecture ETL, and any workload requiring exactly-once guarantees with automatic checkpointing.

How do declarative pipelines handle data quality expectations?

Expectations apply quality checks per record. You can log violations while ingesting the record, drop bad records, or halt the pipeline on failure.

What are the limitations of Delta Live Tables?

DLT pipelines require a Databricks workspace for managed features. The open-source SDP core addresses portability, but enterprise governance and monitoring capabilities remain part of the managed Lakeflow offering.

How do i migrate from dlt to the new sdp syntax?

No migration is required. To adopt the modern API, replace import dlt with from pyspark import pipelines as dp and update decorators.

What programming languages do declarative pipelines support?

SDP supports SQL and Python with equivalent functionality for most data processing use cases.

What is the future direction for declarative pipelines?

The SDP core was contributed to Apache Spark at Data + AI Summit 2025. This makes the declarative model portable across any Spark 4.1 environment. Lakeflow continues to add managed, enterprise capabilities on top.

Build your next pipeline on a unified foundation

Whether you have run DLT pipelines for years or are evaluating declarative pipelines for the first time, the path forward converges on Lakeflow within the Databricks Platform. By unifying batch and streaming ETL with governance and open formats, Lakeflow eliminates fragmented tooling that slows teams down. Explore Spark Declarative Pipelines to see how the modern declarative framework can power your next pipeline.