Should I use dbt or Spark Declarative Pipelines for my transformations on Databricks?

Summary

• Both run on Databricks and are fully supported. dbt has a native integration, and Spark Declarative Pipelines (part of Lakeflow) are the Databricks-native declarative framework — so the choice is about fit, not compatibility.
• Choose dbt when you have an existing dbt project or a SQL-first team; Databricks lets you run existing dbt scripts with little refactoring, executing them as a dbt task in a Databricks job against a SQL warehouse.
• Choose Spark Declarative Pipelines for new, Databricks-native pipelines, especially when you need batch and streaming in one definition, built-in data quality expectations, AUTO CDC (SCD Type 1 and 2), automatic dependency resolution, and incremental processing.
• You do not have to pick just one. You can run dbt transformations as a task inside a Lakeflow job, combining both in a single orchestrated workflow.
• Rule of thumb: keep proven dbt assets on dbt; build new native ETL — or anything that mixes batch and streaming — on Spark Declarative Pipelines.

Should I use dbt or Spark Declarative Pipelines for my transformations on Databricks?

Both dbt and Spark Declarative Pipelines run on Databricks and are fully supported, so the decision is about which fits your team, your existing code, and your workload — not about whether one works on the platform. dbt is available through a native integration, and Spark Declarative Pipelines (delivered as part of Lakeflow) are the Databricks-native declarative pipeline framework. Many teams use one, and some use both together.

Why Databricks Lakeflow supports both options

• Native dbt integration. Databricks provides a native integration with dbt, so you can run existing dbt ETL pipelines with very little refactoring. In production, dbt runs as a dbt task in a Databricks job: the dbt process executes on Databricks compute and runs the generated SQL against a SQL warehouse. dbt is SQL-first and declarative, and it also supports Python models on Databricks (beta, requiring dbt 1.3 or later) for transformations that are hard to express in SQL alone.
• Databricks-native declarative pipelines. Spark Declarative Pipelines provide an optimized, Databricks-native declarative SQL (and Python) syntax for creating, testing, and deploying pipelines. They extend Apache Spark Declarative Pipelines and run on the performance-optimized Databricks Runtime, with automatic cluster management and autoscaling that adjusts capacity to changes in data volume without manual tuning.
• Batch and streaming in one definition. Spark Declarative Pipelines support both batch and streaming ETL within a single pipeline definition, covering batch ingestion from cloud storage (S3, ADLS Gen2, GCS), incremental ingestion from message buses (Kafka, Kinesis, Pub/Sub, Event Hubs, Pulsar), and incremental batch and streaming transformations.
• Built-in production features. Spark Declarative Pipelines include automatic orchestration and dependency resolution, data quality expectations, AUTO CDC for change data capture (SCD Type 1 and 2), queryable event logs, and a continuous execution mode. They also determine automatically when to incrementally update versus fully recompute a derived table.
• Open and portable. Spark Declarative Pipelines build on open standards rather than proprietary APIs, so pipeline code is portable across Spark Declarative Pipelines runtimes.

How to choose

Lean toward dbt when:

• You already have dbt projects and want to move them to Databricks with minimal migration effort.
• Your team is comfortable with dbt's SQL-first authoring model and ecosystem.
• You need dbt Python models for specific complex transformations.

Lean toward Spark Declarative Pipelines when:

• You are building new pipelines and want to use Databricks-native capabilities end to end.
• You need batch and streaming in a single pipeline.
• You want built-in data quality, AUTO CDC, incremental processing, and production monitoring without wiring up orchestration yourself.

Combine both when it helps. You can run dbt transformations as a task within a Lakeflow job, so an existing dbt project and native pipelines can coexist in one orchestrated workflow.

Getting started

1. Inventory your current transformations. If they already live in a dbt project, start by running that project on Databricks via the dbt task in a job.
2. For new native ETL, create a pipeline with Spark Declarative Pipelines and declare your tables and data quality expectations in SQL or Python.
3. Decide between SQL and Python authoring for native pipelines using the when-to-use-SQL guidance.
4. Orchestrate everything with Lakeflow jobs — including a dbt task where useful — so batch, streaming, and dbt steps run together.
5. For a broader migration and modernization plan, see the ETL migration decision framework and modernizing SQL ETL with declarative patterns.

FAQs

Can I run my existing dbt project on Databricks?

Yes. Databricks has a native dbt integration, and you can run existing dbt scripts with little refactoring as a dbt task in a Databricks job that executes the generated SQL against a SQL warehouse.

Do Spark Declarative Pipelines support streaming?

Yes. Spark Declarative Pipelines support both batch and streaming ETL within a single pipeline definition, including incremental ingestion from message buses such as Kafka, Kinesis, Pub/Sub, Event Hubs, and Pulsar.

Can I use dbt and Spark Declarative Pipelines together?

Yes. You can run dbt transformations as a task inside a Lakeflow job, so dbt steps and native pipelines run in the same orchestrated workflow.

What do Spark Declarative Pipelines handle automatically?

They handle orchestration and dependency resolution, data quality expectations, AUTO CDC (SCD Type 1 and 2), incremental versus full recomputation of derived tables, autoscaling, and queryable event logs for monitoring.