Spark versus DataFusion?

Summary

• Apache Spark and Apache DataFusion work at different layers. Spark is a distributed analytics engine with its own cluster runtime and libraries; DataFusion is a Rust query engine library that runs inside a single process and is designed to be embedded.
• Spark offers APIs in Java, Scala, Python and R (R is deprecated), plus Spark SQL, pandas API on Spark, MLlib, GraphX and Structured Streaming.
• DataFusion uses Apache Arrow as its in-memory format, offers SQL and DataFrame APIs in Rust and Python, and lets developers add data sources, functions and operators.
• Multi-node DataFusion is a separate subproject, Ballista. DataFusion Comet runs supported Spark operators on the DataFusion engine and falls back to Spark for the rest.

Spark versus DataFusion?

Apache Spark and Apache DataFusion solve different problems, so the choice follows from what you're building. Spark is "a unified analytics engine for large-scale data processing" that runs on a cluster or on one machine and ships libraries for SQL, streaming, machine learning and graph processing. DataFusion is "an extensible query engine written in Rust that uses Apache Arrow as its in-memory format," and its FAQ says it is "designed to run within a single process, using threads for parallel query execution." If the job is running data pipelines across a cluster, Spark is built for it. If the job is building a database, query service or data tool on an embeddable engine, DataFusion is built for that.

What is Apache DataFusion?

DataFusion is a query engine "for building high-quality data-centric systems in Rust." It was donated to the Apache Software Foundation in 2019 as part of Apache Arrow and became a top-level ASF project on April 16, 2024. Its FAQ describes it as "targeted at developers, rather than end users / data scientists." Two subprojects widen its reach: Ballista, "a distributed compute platform primarily implemented in Rust, and powered by Apache DataFusion," and Comet, an Apache Spark plugin that "intercepts Spark physical plans" and hands supported operators to the DataFusion native engine.

Apache Spark vs Apache DataFusion: how they compare

Capability	Apache Spark	Apache DataFusion
Runtime	Java 17/21/25 and Scala 2.13; Python 3.10+ for PySpark	Rust library; Python bindings install with pip install datafusion
Execution scope	Spark Standalone, Hadoop YARN or Kubernetes clusters, or locally on one machine	One process, with threads for parallel query execution
Multi-node execution	Built into the engine	Ballista subproject (scheduler and executors)
APIs	Java, Scala, Python and R (R is deprecated); Spark Connect for remote clients	SQL and DataFrame APIs in Rust and Python; Substrait query plans
Included libraries	Spark SQL, pandas API on Spark, MLlib, GraphX, Structured Streaming	Extension points for functions, data sources, SQL and custom nodes
File formats	Parquet, ORC, JSON, CSV, text, XML, Avro, Protobuf and binary files; Hive tables; JDBC	Parquet, CSV, JSON and Avro natively; async IO from S3, Azure and GCS
Open table formats	Delta Lake and Apache Iceberg document Spark integrations	delta-rs documents a DataFusion integration; Apache iceberg-rust publishes an iceberg-datafusion crate
Governance and license	Apache Software Foundation; Apache License 2.0	ASF top-level project; Apache License 2.0

How to choose

• Where the work runs. Jobs that span many machines fit Spark's built-in cluster managers. A query engine you link into your own Rust or Python service fits DataFusion's in-process design.
• Which libraries you need. Structured Streaming, MLlib and GraphX ship with Spark. DataFusion supplies the SQL and DataFrame engine plus extension points for data sources, query languages, functions and custom operators.
• Which runtime your team operates. Spark runs on the JVM. DataFusion is native Rust with Arrow in memory.
• Whether you want both. Comet keeps your Spark applications and APIs, executes supported operators on DataFusion, and falls back to Spark for unsupported features.

The same query in both engines

Both engines evaluate lazily. Spark computes transformations only "when an action requires a result," and DataFusion won't run a query "until you call a method such as show() or collect()."

# Apache Spark (PySpark)
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
spark.read.parquet("events.parquet").createOrReplaceTempView("events")
spark.sql("SELECT event_type, count(*) AS n FROM events GROUP BY event_type").show()

# Apache DataFusion (pip install datafusion)
from datafusion import SessionContext

ctx = SessionContext()
ctx.register_parquet("events", "events.parquet")
ctx.sql("SELECT event_type, count(*) AS n FROM events GROUP BY event_type").show()

How Databricks approaches Spark and native execution

On Databricks, Apache Spark powers compute clusters and SQL warehouses. Photon, the Databricks-native vectorized query engine, "replaces the JVM-based Spark SQL execution engine with a native C++ runtime" for supported operations. It's compatible with Apache Spark APIs, so existing code runs without changes, and an unsupported operation switches transparently to the Spark runtime. Photon is enabled on serverless compute, SQL warehouses and serverless Lakeflow pipelines.
Databricks Connect, built on open-source Spark Connect, connects IDEs such as Visual Studio Code and PyCharm to Databricks compute. For engines outside Databricks, Unity Catalog serves Delta Lake clients through the Unity REST API and Apache Iceberg clients through the Iceberg REST catalog, with credential vending so those clients inherit governed privileges.

FAQs

Is DataFusion a replacement for Spark?

They target different jobs. DataFusion is a library for building query engines and data systems inside one process; Spark is a cluster engine with libraries for SQL, streaming, machine learning and graphs. Comet runs DataFusion inside Spark.

Can DataFusion run on a cluster?

Core DataFusion runs in a single process and uses threads for parallelism. Ballista, a DataFusion subproject, runs scheduler and executor processes to execute work across multiple nodes.

Can both engines read Delta Lake and Apache Iceberg tables?

Yes, through integrations. Delta Lake and Iceberg document Spark support. On the DataFusion side, delta-rs documents registering Delta tables with DataFusion, and Apache iceberg-rust publishes the iceberg-datafusion crate of table providers.