SDP versus dbt?

Summary

• Spark Declarative Pipelines (SDP) is the open-source Apache Spark framework for batch and streaming pipelines, added in Spark 4.1. You define streaming tables, materialized views and temporary views in Python or SQL, and Spark works out the execution order.
• dbt compiles SQL select statements into tables and views and runs them on the data platform behind its adapter. It covers the transformation step only, after the data is loaded.
• They overlap on dependency-aware, declarative transformations. SDP has streaming tables built in and runs on Spark. dbt has data tests, generated documentation and adapters for different data platforms.
• On Databricks you can use both: a Lakeflow Jobs pipeline task runs a pipeline, and a dbt task runs a dbt project.

SDP versus dbt?
Spark Declarative Pipelines (SDP) and dbt both let you declare a graph of datasets and leave the run order to the tool. They are built for different jobs. SDP is the open-source Apache Spark framework for batch and streaming pipelines. dbt turns SQL select statements into tables and views on whichever data platform you connect it to.

What is SDP?

The Spark 4.1.0 release notes introduce SDP as "A new declarative framework where you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries." A pipeline project is a set of .py or .sql files that define streaming tables, materialized views, temporary views and flows. You install it with pip install pyspark[pipelines] and run it with the spark-pipelines command, which is built on spark-submit.

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.materialized_view
def daily_orders_by_state() -> DataFrame:
    return (
        spark.table("customer_orders")
        .groupBy("state", "order_date")
        .count()
        .withColumnRenamed("count", "order_count")
    )

Spark 4.2 injects spark into each pipeline file. On Spark 4.1, add spark = SparkSession.active() yourself.

What is dbt?

The Databricks docs describe dbt (data build tool) as "a development environment for transforming data by writing select statements" that "compiles your code into raw SQL and then runs that code on the specified database". dbt "does not extract or load data", so it assumes the data is already in your database. The dbt docs add that a model can be a SQL select statement or a Python DataFrame, and that dbt reaches each data platform through a dedicated adapter plugin. The Databricks docs call the open-source command-line version dbt Core.

-- models/daily_orders_by_state.sql
select state, order_date, count(*) as order_count
from {{ ref('customer_orders') }}
group by state, order_date

ref() links one model to another. The dbt docs say it also builds the dependency graph, so models deploy in the correct order.

How do SDP and dbt compare?

Dimension	SDP	dbt
You write	Python decorators or SQL statements	SQL select statements with Jinja and YAML; Python models also work
Dependencies	The pipeline analyzes how defined datasets reference each other	ref() and source()
Runs on	Apache Spark	The platform behind the adapter
Dataset types	Streaming table, materialized view, temporary view	Table, view, incremental, ephemeral, materialized view
Streaming	Streaming tables fed by streaming flows, for example from Kafka	Near real time by running dbt often over the platform's continuous ingestion
Checks	spark-pipelines dry-run reports syntax, analysis and graph errors	dbt test data tests, plus a generated documentation site

When should you use each?

Pick SDP when the pipeline includes streaming or incremental ingestion, for example from cloud storage or Apache Kafka, and you want one framework to handle ordering, checkpoints and retries. Pick dbt when the data is already loaded into a platform, the work is SQL transformation, and you want tests and generated documentation around your models. Use both when your platform has two layers. The dbt guide to near real-time data describes continuous landing through the platform's streaming ingestion, followed by frequent dbt runs.

How do they fit on Databricks?

Lakeflow Pipelines extend SDP and run on Databricks Runtime. Databricks lists additions such as data quality expectations, a queryable event log and continuous mode. Its Python reference marks @dp.expect(...) as not part of Apache Spark. dbt runs on Databricks through the dbt-databricks adapter, which the adapter's README recommends for projects on Databricks. In Lakeflow Jobs, a dbt task runs a project against a SQL warehouse and a pipeline task runs a pipeline, so one job can contain both. The adapter can also materialize a model as a streaming table or materialized view, which needs Unity Catalog and a serverless SQL warehouse.

FAQs

Does dbt run on Apache Spark?

Yes. The dbt docs include an Apache Spark setup that connects over thrift, http or livy. For Databricks projects, the dbt-databricks README recommends its own adapter.

Are both open source?

Yes. The Apache Spark repository and the dbt repository are both Apache-2.0 licensed. Databricks says SDP code stays portable to other SDP runtimes.

Does dbt handle streaming?

The dbt guide recommends landing data continuously with the platform's streaming ingestion and running dbt frequently. For sub-second results it points to a dedicated streaming system alongside dbt.

Can one Databricks job run both?

Yes. Tasks in a job can depend on each other, and the task types include pipeline and dbt.