Best ETL tools 2026

Summary

• ETL tooling splits by role: Apache Spark is the compute engine, Spark Declarative Pipelines (SDP) defines datasets and queries, Apache Airflow schedules and monitors workflows, and dbt transforms data with SQL select statements.
• SDP was added in Apache Spark 4.1.0. Expectations and Auto CDC flows are Databricks extensions, not part of Apache Spark.
• dbt Core v1.x and dbt OSS (v2) are Apache 2.0. The dbt Core name no longer applies to any dbt v2 distribution.
• Unity Catalog OSS is an Apache 2.0 catalog for tables, unstructured data, and AI assets, and it implements the Iceberg REST Catalog APIs.
• On Databricks, Lakeflow Pipelines extends SDP and Lakeflow Jobs orchestrates tasks, including pipelines, dbt, SQL, and Python.

Best ETL tools 2026
There is no single best ETL tool in 2026, because the work splits into roles: a compute engine, a pipeline definition layer, an orchestrator, a SQL transformation tool, and a catalog. The open-source tools below each cover one or two of those roles, and you can combine them.

Which ETL tool does which job?

Tool	Role	What its docs say
Apache Spark	Compute engine	"a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters"
Spark Declarative Pipelines (SDP)	Pipeline definition	"a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark"
Apache Airflow	Scheduling and monitoring	"an open-source platform for developing, scheduling, and monitoring workflows"
dbt	SQL transformation	"You write simple SQL select statements"
Delta Lake and Apache Iceberg	Open table formats	Delta Lake: "Serializable isolation levels ensure that readers never see inconsistent data." Iceberg: "schema changes never require rewriting your table."
Unity Catalog OSS	Open catalog	Apache 2.0 licensed, "including an OpenAPI specification, server and clients"

What does batch ETL look like in Spark SQL?

SELECT customer_id, SUM(amount) AS total_spend, COUNT(*) AS orders
FROM events
WHERE event_date >= date_sub(current_date(), 30)
GROUP BY customer_id;

Spark SQL, DataFrames, and Structured Streaming run on the same engine. For streaming ETL, see the Structured Streaming guide.

What are Spark Declarative Pipelines?

The Spark 4.1.0 release notes add SDP as "A new declarative framework where you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries." A pipeline project is a set of .py or .sql source files, plus a spec file conventionally named spark-pipeline.yml. A materialized view always has exactly one batch flow writing to it, and streaming tables support incremental processing.

CREATE MATERIALIZED VIEW customer_spend
AS SELECT customer_id, SUM(amount) AS total_spend
FROM events
GROUP BY customer_id;

name: customer_pipeline
libraries:
  - glob:
      include: transformations/**
storage: file:///absolute/path/to/storage/dir

Create a project with spark-pipelines init --name my_pipeline, check it with spark-pipelines dry-run, and execute it with spark-pipelines run. If no refresh options are specified, a default incremental update is performed.

When do I add Airflow?

Airflow targets "traditional time based or event-triggered batch-oriented data pipelines," with workflows defined in code. The apache-airflow-providers-apache-spark package connects it to Spark. The Airflow docs also say that if you prefer clicking over coding, it might not be the best fit.

Where does dbt fit?

dbt's engine "compiles your project, executes your transformation graph, and produces metadata." It ships four generic data tests (unique, not_null, accepted_values, relationships) that run with dbt test, supports model contracts with contract: enforced: true, and dbt docs generate builds project documentation that includes a DAG. dbt documents Apache Spark support, and its v2 Spark adapter is currently Beta.

What does Unity Catalog OSS cover?

It supports Delta Lake, Apache Iceberg via UniForm, Parquet, CSV, JSON, and other formats, and it implements the Iceberg REST Catalog APIs. The project site describes "built-in governance and security" with "strong authentication, secure credential vending, and asset-level access control." The docs home page lists integrations including Apache Spark and DuckDB.

How does Databricks approach ETL?

Lakeflow Pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime."

• Beyond open-source SDP: Databricks adds expectations (@dp.expect), which "validate data as it flows through the pipeline," and Auto CDC flows. The Databricks docs say these are not part of Apache Spark.
• Unity Catalog: pipelines configured with Unity Catalog "publish all defined materialized views and streaming tables to the specified catalog and schema."
• Execution: a pipeline runs in triggered or continuous mode, runs flows "in the correct order with maximum parallelism," and retries transient failures progressively.
• Lakeflow Jobs: provides orchestration. Task types include Pipeline, Python script, SQL, and dbt, and triggers include scheduled, file arrival, table update, and continuous. With serverless compute, Databricks configures your compute resources.

FAQs

Should I use Spark Declarative Pipelines or Airflow?

They answer different questions. A running SDP pipeline "analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically." Airflow handles schedules and event triggers across workflows. Use SDP for the dataset graph and an orchestrator for when it runs.

Do I need Unity Catalog OSS if I use Spark and Airflow?

Only if you want a shared open catalog. The Unity Catalog OSS docs home page lists Apache Spark among its integrations and does not list Airflow or dbt.

What is dbt Core called now?

dbt's licensing FAQ says dbt Core v1.x is still Apache 2.0, dbt OSS is the Apache 2.0 subset of dbt v2, and "The dbt Core name no longer applies to any dbt v2 distribution."