Best open-source ETL tools 2026

Summary

• No single open-source tool does ETL end to end. A pipeline combines a processing engine (Apache Spark or Apache Flink), data movement (Apache Kafka or Apache NiFi), an orchestrator (Apache Airflow), SQL transformation (dbt Core) and data validation (Great Expectations, GX Core).
• Spark Declarative Pipelines, introduced in Spark 4.1.0, define datasets and flows in Python or SQL on Apache Spark. As of October 2026 the newest Spark minor version is 4.2.0 (July 14, 2026), which added Auto CDC to Spark Declarative Pipelines.
• Flink is built for stateful computations over unbounded and bounded streams, and Kafka Streams is a Java and Scala client library that needs no separate processing cluster.
• Airflow targets batch-oriented workflows defined in Python and is not a streaming solution, and dbt transforms data that is already in your platform.

Best open-source ETL tools 2026

No single open-source tool covers ETL end to end. A pipeline is assembled from tools that each do one job: processing, data movement, orchestration, SQL transformation and validation. The table maps each job to the open-source projects behind it, and the sections after it show how they fit together.

Which open-source ETL tool does which job?

Tool	Job in a pipeline	What its docs say
Apache Spark	Distributed processing	A multi-language engine for data engineering, data science and machine learning; Python, SQL, Scala and Java (R is deprecated)
Spark Declarative Pipelines	Pipeline definition on Spark	A declarative framework for building data pipelines on Apache Spark, in Python or SQL
Apache Flink	Stream processing	Stateful computations over unbounded and bounded data streams; Java, Python and SQL
Apache Kafka	Event streaming	Kafka Connect imports and exports data; Kafka Streams is a client library for Java and Scala applications with no separate processing cluster
Apache NiFi	Data flows	A system to process and distribute data, with data provenance tracking and a browser-based interface
Apache Airflow	Orchestration	A platform to author, schedule and monitor workflows, defined entirely in Python
dbt Core	SQL transformation	You write SQL select statements, and dbt builds the models, runs data tests and generates documentation
Great Expectations (GX Core)	Data validation	A Python library for building and running data validation workflows; an Expectation is a verifiable assertion about data

How do these tools fit together in a pipeline?

Kafka Connect moves data between Kafka and systems such as relational databases. Spark reads from and writes to Kafka through the Structured Streaming Kafka integration, and Spark Declarative Pipelines lists Apache Kafka among its supported message bus sources. dbt transforms data directly in your cloud data platform, GX Core connects to pandas and Spark dataframes, and Airflow schedules and monitors the workflow.

How do I choose?

• Batch or streaming: Structured Streaming uses a micro-batch engine by default, and Spark 4.1.0 added Real-Time Mode for low-latency queries. Flink supports event-time and processing-time semantics. Airflow targets batch-oriented pipelines.
• Language: Spark, Flink and Spark Declarative Pipelines accept SQL, dbt models are SQL select statements, Airflow and GX Core are Python, and Kafka Streams is Java and Scala.
• How pipelines are run: A Spark Declarative Pipelines project has a YAML spec file, conventionally named spark-pipeline.yml, and runs with spark-pipelines run. The spark-pipelines dry-run command checks a pipeline without reading or writing data.
• Failure handling in streams: Flink checkpoints state for failure recovery and uses savepoints for planned operations such as upgrades.

What changed for ETL in Spark 4.1 and 4.2?

Spark 4.1.0 introduced Declarative Pipelines and Real-Time Mode in Structured Streaming. Spark 4.2.0 added a SQL CHANGES clause with DataFrame, PySpark and Connect APIs to read row-level changes in batch and streaming, and Auto CDC (declarative SCD Type 1) in Spark Declarative Pipelines. It also added Real-Time Mode trigger support to PySpark.

How does Databricks approach open-source ETL?

Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines while running on the Databricks Runtime. Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage, and Databricks recommends it in Lakeflow Pipelines for incremental ingestion. Lakeflow Jobs coordinates multiple tasks as one workflow with time-based or event-based triggers, and it can run dbt Core projects as job tasks. Photon accelerates SQL workloads, DataFrame API calls and ETL pipelines, and works with existing Apache Spark code with no changes required.

FAQs

Should I use Spark or Flink?

Spark's docs describe an engine that unifies batch and real-time streaming processing. Flink's docs describe stateful computations over unbounded and bounded streams. Start with Spark when you need batch, SQL and machine learning alongside streaming, and look at Flink first when stateful, event-driven stream processing is the core workload.

Do Airflow and Spark Declarative Pipelines overlap?

They work at different levels. Airflow schedules and monitors workflows, and its Spark provider includes operators such as SparkSubmitOperator. Spark Declarative Pipelines lets you define what tables should exist and what their contents should be, and it handles orchestration, compute management and error handling within the pipeline.

Do I need both Kafka and Spark?

Not always. Kafka Streams processes event streams in Java and Scala applications without a separate processing cluster. When you do combine them, the Structured Streaming Kafka integration reads data from and writes data to Kafka.

Is dbt enough for ETL?

No. dbt covers the transform step and works alongside your ingestion tools, so you still need a tool that extracts and loads the data.