use open source SDP

Summary

• Apache Spark Declarative Pipelines (SDP) is an open-source framework for batch and streaming data pipelines using SQL and Python, removing orchestration boilerplate.
• SDP automates dependency resolution and orchestration between pipeline stages, letting teams focus on business logic instead of workflow mechanics.
• Code written in SDP is portable to any SDP runtime; Databricks Lakeflow Pipelines add production features (AUTO CDC, data quality, continuous mode) while remaining interoperable with SDP.
• Use SDP for cloud ingestion (S3, ADLS, GCS), message bus streaming (Kafka, Kinesis, Pub/Sub, Event Hubs, Pulsar), and incremental transformations.
• Lakeflow Pipelines extend SDP with materialized views, automatic clustering, continuous updates, and integration with Unity Catalog for lineage and governance.

use open source SDP
Apache Spark Declarative Pipelines (SDP) is an open-source framework that lets you build batch and streaming data pipelines in SQL and Python without writing orchestration code. Instead of managing DAGs (directed acyclic graphs) and job dependencies by hand, you declare what data you want and where it goes. SDP handles the rest. Databricks Lakeflow Pipelines are built on SDP, remain interoperable with it, and add production features that enterprise data teams need.

What is Apache Spark Declarative Pipelines?

SDP simplifies pipeline development by separating business logic from orchestration. In a traditional orchestration framework, you define tasks, set dependencies between them, and manage execution state. With SDP, you write SQL or Python transformations, and SDP automatically detects dependencies and orchestrates the pipeline.
Core concepts:

• Flows: Stateless transformations that process data, using Apache Spark DataFrame API or Structured Streaming.
• Streaming tables: Delta tables that support incremental or streaming data processing.
• Materialized views: Cached views that auto-refresh when dependencies change.
• Sinks: External targets (Kafka, Event Hubs, custom Python sinks, or external tables).

SDP orchestrates these components automatically. When a source table changes, dependent streaming tables and materialized views refresh. No manual job triggers required.

Why use SDP instead of traditional orchestration?

Aspect	Traditional Orchestration	SDP
Dependency management	Manual DAG definition	Automatic inference
Code reusability	Framework-specific APIs	Portable SQL and Python
Incremental processing	Requires explicit state tracking	Built-in for streaming tables
Development speed	Verbose job definitions	Concise transformation code
Multi-engine portability	Limited	Full (any SDP runtime)

With SDP, a transformation that runs in your local development environment on DuckDB or Spark works identically on Databricks. You're not locked into Databricks APIs; you're using a standardized Apache Spark model.

Common use cases

Cloud data ingestion: Read files from S3, Azure Data Lake Storage, or Google Cloud Storage incrementally. SDP detects new files and processes them without manual backfill logic.
Real-time streaming: Consume from Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs, or Apache Pulsar. SDP manages consumer groups, checkpoints, and exactly-once semantics.
Incremental transformations: Apply SCD Type 1 or Type 2 slowly changing dimensions, merge streams with reference data, or aggregate metrics. SDP handles state and checkpointing automatically.
Federated pipelines: A data lakehouse might ingest data from multiple sources (databases, APIs, message buses), transform it incrementally, and produce analytics tables. SDP makes this a single declarative definition.

How Lakeflow Pipelines extend SDP

Databricks Lakeflow Pipelines are built on Apache SDP but add enterprise features:

• AUTO CDC: Automatically apply change-data-capture Type 1 and Type 2 slowly changing dimensions without handwritten merge logic.
• Data quality expectations: Assert data freshness, schema validation, and metric thresholds. Pipelines pause on data quality violations.
• Queryable event log: Inspect pipeline execution history, lineage, and data quality results.
• Continuous mode: Pipelines run continuously (not just on triggers), enabling low-latency updates.
• Unity Catalog integration: Lineage, governance, and access control flow end-to-end through your pipeline.

Lakeflow Pipelines remain compatible with open-source SDP. If you write your pipelines using SDP's public APIs, they can run on any SDP runtime, including Databricks.

Open-source contribution

At Data and AI Summit 2025, Databricks contributed its declarative pipeline technology to the Apache Spark project as Spark Declarative Pipelines. This move standardizes the declarative model for the entire Spark community and ensures the framework evolves as an open standard, not a Databricks-proprietary feature.

When to use SDP

Choose SDP for:

• Batch ingestion from cloud object storage with incremental processing.
• Streaming from message buses or external databases.
• Transformations that need automatic state and checkpoint management.
• Portable code that might run across different Spark engines or platforms.
• Teams that want to avoid workflow orchestration boilerplate.

Avoid SDP if:

• Your pipeline requires custom orchestration logic that SDP doesn't express (rare).
• You need non-Spark compute (SDP is Spark-specific).
• You already have an orchestration framework you're satisfied with and can't migrate.

FAQs

Is SDP production-ready?

Yes. Apache Spark Declarative Pipelines was contributed to the Apache Spark project at DAIS 2025 and is production-ready. Databricks Lakeflow Pipelines (the SDP-based implementation) is GA and serves production workloads at scale.

Can I run SDP code outside of Databricks?

Yes. SDP is open-source, and the Apache Spark community maintains the reference implementation. You can run SDP on any Spark cluster, including on-premises or other clouds. Databricks Lakeflow adds extensions, but core SDP features are portable.

How does SDP compare to Apache Airflow?

Airflow is a general-purpose orchestration framework; SDP is a declarative data pipeline framework. Airflow requires you to define tasks and dependencies. SDP infers dependencies from your code. For data pipelines specifically, SDP requires less boilerplate. Some teams use both: Airflow for cross-functional workflows, SDP for data pipelines within the data platform.

Does SDP support backfills?

Yes. You can specify a backfill window, and SDP will replay your transformation logic over historical data. This is useful when you change a transformation and need to recompute results.