SDP vs Lakeflow Pipelines

Summary

• SDP stands for Spark Declarative Pipelines, an open-source declarative pipeline framework contributed to Apache Spark 4.1; it provides the authoring model for both batch and streaming pipelines in SQL and Python.
• Lakeflow Pipelines is the Databricks managed pipeline service built on top of SDP; it implements the same declarative authoring model and adds production features that are not part of the open-source standard.
• Because Lakeflow Pipelines uses SDP as its foundation, pipeline code is portable across any SDP-compatible runtime.
• Lakeflow Pipelines extends SDP with AUTO CDC (SCD Type 1 and Type 2 change data capture), built-in data quality expectations, a queryable pipeline event log, continuous mode, and additional sink and flow types.
• Teams using Databricks run Lakeflow Pipelines and automatically get the full SDP authoring model plus these managed-service additions; teams building on open Apache Spark can run SDP directly.

SDP vs Lakeflow Pipelines

Spark Declarative Pipelines (SDP) is the open-source declarative pipeline framework now part of Apache Spark 4.1. Lakeflow Pipelines is the Databricks managed pipeline product that implements SDP and extends it with production features. Lakeflow Pipelines is built on SDP, not a separate approach: it uses SDP's declarative authoring model so that pipeline code written for Lakeflow Pipelines remains portable across other SDP runtimes.

What is Spark Declarative Pipelines (SDP)?

Apache Spark Declarative Pipelines is a declarative framework for developing and running batch and streaming data pipelines in SQL and Python. SDP was contributed to Apache Spark and released as part of Apache Spark 4.1. The key design goal is to let authors declare what the pipeline produces (a streaming table, a materialized view) rather than manage how orchestration, retries, and incremental processing work. SDP handles dependency resolution automatically, choosing whether to run a full refresh or an incremental update based on the declared dataset types.
The two core dataset types in SDP are:

• Streaming tables: process each record exactly once from append-only sources; suited for continuous ingestion from event streams and files.
• Materialized views: recomputed as needed to reflect current data state; suited for aggregations and transformations serving multiple downstream consumers.

SDP vs Lakeflow Pipelines: how they compare

Feature	SDP (Apache Spark open source)	Lakeflow Pipelines (Databricks managed)
Authoring model	Declarative SQL/Python; streaming tables and materialized views	Same SDP authoring model; code portable across SDP runtimes
Automatic dependency resolution	Yes	Yes
Incremental processing	Yes	Yes
AUTO CDC (SCD Type 1 and Type 2)	Not in the open-source standard	Yes; handles out-of-order CDC events
Data quality expectations	Not in the open-source standard	Yes; built-in validation framework
Queryable pipeline event log	Not in the open-source standard	Yes; for monitoring and troubleshooting
Continuous mode	Not in the open-source standard	Yes; real-time processing option
Update flows and foreachBatch sinks	Not in the open-source standard	Yes
Managed runtime and infrastructure	Community-managed on self-hosted Spark	Databricks-managed; integrated with Unity Catalog
Unity Catalog governance	Requires separate UC integration	Native; tables registered in UC automatically

How Databricks approaches SDP with Lakeflow Pipelines

Lakeflow Pipelines implements SDP and adds a managed-service layer on top of it. Because the authoring model is the same SDP model, teams that know SDP can work with Lakeflow Pipelines without learning a proprietary API. The managed additions address the gap between an OSS framework and a production pipeline service:

• AUTO CDC: Lakeflow Pipelines handles out-of-order change data capture events and applies SCD Type 1 and Type 2 patterns declaratively. You declare APPLY CHANGES and the pipeline manages merge logic and ordering.
• Data quality expectations: Inline EXPECT clauses validate each row at runtime. Failed rows can be dropped, quarantined, or allowed to pass through with the violation recorded in the event log.
• Queryable event log: All pipeline runs, data quality outcomes, and operational events are written to a queryable event log table, making debugging and SLA monitoring straightforward with standard SQL.
• Unity Catalog integration: Tables created by Lakeflow Pipelines are registered in Unity Catalog automatically, inheriting UC lineage capture, access control, and audit logging without additional configuration.

For teams on Databricks, Lakeflow Pipelines is the recommended way to run SDP: the same declarative authoring model runs on managed infrastructure with production features included.
Docs: https://docs.databricks.com/aws/en/ldp/concepts/spark-declarative-pipelines

FAQs

Is Lakeflow Pipelines a replacement for SDP, or built on it?

Lakeflow Pipelines is built on top of SDP. It uses SDP's declarative authoring model and adds production features (AUTO CDC, data quality expectations, queryable event log, continuous mode) that are not in the open-source standard. Pipeline code written for Lakeflow Pipelines is portable to any SDP-compatible runtime because the foundational authoring model is the same.

What is AUTO CDC in Lakeflow Pipelines?

AUTO CDC is Lakeflow Pipelines' change data capture feature, implemented via the APPLY CHANGES command. It processes out-of-order CDC events from source systems and maintains SCD Type 1 (overwrite) or SCD Type 2 (history-preserving) target tables declaratively, without custom merge logic.

Can I run Spark Declarative Pipelines outside of Databricks?

Yes. SDP is part of open-source Apache Spark 4.1. You can run SDP pipelines on any Spark 4.1 cluster. Lakeflow Pipelines-specific features (AUTO CDC, the queryable event log, continuous mode, Unity Catalog integration) require the Databricks managed runtime.

Are streaming tables and materialized views in SDP the same as in Lakeflow Pipelines?

Yes. The streaming table and materialized view concepts are defined in SDP and shared by Lakeflow Pipelines. Streaming tables process each record exactly once from append-only sources; materialized views recompute their state to reflect current source data. Lakeflow Pipelines adds the AUTO CDC flow type on top of these two base types.