When should I use DuckDB instead of Spark?

Summary

• Use DuckDB when one machine can run the job, one process writes the data, and you want analytical SQL inside a Python, R, Java or other application with no database server to operate.
• Use Spark when the work has to spread across a cluster, several pipelines write the same tables, or batch jobs, Structured Streaming and MLlib need to run in one engine.
• DuckDB handles larger-than-memory queries by spilling to disk, within documented limits, so data that exceeds RAM doesn't by itself rule DuckDB out.
• The two engines can share tables. DuckDB's delta and iceberg extensions read tables that Spark writes, and its unity_catalog extension attaches to Unity Catalog on Databricks or Unity Catalog OSS.

When should I use DuckDB instead of Spark?

Use DuckDB when a single machine can handle the job, one process does the writing, and you want analytical SQL inside your application, notebook or script without running a server. Use Spark when the job needs a cluster, when several jobs write the same tables concurrently, or when batch processing, streaming and machine learning have to share one engine. The choice doesn't have to be exclusive: DuckDB can read the Delta Lake and Apache Iceberg tables that Spark jobs write.

What each engine is built for

DuckDB describes itself as an analytical in-process SQL database management system, and it's released under the MIT License. It targets OLAP workloads, meaning complex, relatively long-running queries over significant portions of a dataset. It provides ACID transactions through multi-version concurrency control, and it parallelizes a query across threads by row group. By default there's no server process: the database runs inside whatever program opens it.
Apache Spark is a unified analytics engine for large-scale data processing, released under the Apache License 2.0. A driver program schedules tasks on executor processes, which a cluster manager allocates across machines. The same engine runs Spark SQL, Structured Streaming, MLlib and the pandas API on Spark.

DuckDB vs Apache Spark: how they compare

Capability	DuckDB	Apache Spark
Architecture	In-process database embedded in the host application	Driver program coordinating executor processes
Parallelism	Threads on one machine, split by row group	Tasks on executors across the nodes of a cluster
Cluster management	No server or cluster to run	Spark standalone, YARN or Kubernetes cluster managers
Concurrent writers	Single writer process per file; Quack (beta) or DuckLake when several processes write	Set by the table format; Delta Lake, for example, lets writers on several clusters commit in a serial order on supported storage systems
Larger-than-memory data	Spilling to disk, with documented limitations (see the FAQs)	Operators spill to disk; cached data spills or is recomputed
Interfaces	SQL, a Python relational API and an experimental API that implements PySpark's	PySpark, Scala and Java DataFrame APIs, Spark SQL, and pandas-style code through the pandas API on Spark
Beyond core queries	Extensions such as delta, iceberg and httpfs	Libraries for Structured Streaming, MLlib and GraphX
License	MIT	Apache License 2.0

Signs DuckDB is the right tool

• One analyst or one service queries the data. DuckDB queries Parquet, CSV and JSON files directly, from local disk or object storage.
• SQL belongs inside an application. Embedding means there's no separate database server to install, update or maintain.
• A single process writes. A DuckDB file accepts one read-write process at a time, which fits a scheduled job or a local tool.
• Interactive exploration. A notebook can query files or tables without starting a cluster.

Signs Spark is the right tool

• Data or compute exceeds one machine. Executors spread tasks across a cluster.
• Many pipelines write shared tables. Table formats such as Delta Lake coordinate concurrent commits from several clusters.
• Streaming or machine learning runs next to batch work. Structured Streaming and MLlib share the engine and its DataFrame API.
• The team already writes PySpark. Existing jobs and skills carry over.

Reading Spark-written tables from DuckDB

The delta extension, built on Delta Kernel, scans a Delta table in place:

INSTALL delta;
LOAD delta;
SELECT * FROM delta_scan('s3://some/delta/table');

It also appends rows when you attach the table and run INSERT INTO. For Iceberg, iceberg_scan reads a table straight from storage without a catalog, read-only, and attaching an Iceberg REST catalog adds writes. DuckDB lists S3, S3 Tables and Google Cloud Storage as the storage backends it supports for REST catalogs.

How Databricks approaches DuckDB and Spark together

Unity Catalog can sit between both engines. DuckDB's unity_catalog extension attaches to a Unity Catalog on Databricks or Unity Catalog OSS, lists tables, runs SELECT with time travel and supports inserts, but not UPDATE, DELETE or table creation. Unity Catalog's Iceberg REST catalog is a second route, open to supported Iceberg clients. Spark jobs on Databricks can use serverless compute, so no cluster has to be created in your cloud account first.

FAQs

Can DuckDB process data larger than memory?

Yes, by spilling to a temporary directory on disk that you can move with temp_directory. Its docs list limitations, including that a query that stacks several blocking operators may still hit an out-of-memory error and that aggregates such as list() and string_agg() can't be offloaded to disk.

Can several processes write to one DuckDB database?

In the default in-process mode, either one process reads and writes the database file, or several processes open it read-only. For multi-process writes, DuckDB's docs point to the Quack remote protocol, which is in beta, or to DuckLake with a PostgreSQL catalog.

Does DuckDB support transactions?

Yes. DuckDB provides ACID guarantees through multi-version concurrency control. When two threads update or delete the same row at the same time, optimistic concurrency control makes the second one fail with a conflict error.