Spark versus DuckDB for analytics?

Summary

• DuckDB's FAQ calls it "a single-node database system" that scales vertically with more CPU, memory and disk. Apache Spark runs on one machine or distributes work across a cluster.
• DuckDB runs embedded in the process that calls it, with no database server to install. A Spark application runs a driver program that sends tasks to executor processes.
• Both process data larger than memory on one machine: DuckDB spills to disk for joins, grouping, sorting and window functions, and Spark's operators spill to disk too.
• Decide on growth path, the number of concurrent writers, and whether batch and streaming should share one engine.
• They combine well: DuckDB's delta and iceberg extensions read the open tables Spark pipelines write, and Unity Catalog's open APIs let DuckDB access Databricks managed tables.

Spark versus DuckDB for analytics?

Use DuckDB when the data and the workload fit on one machine. Use Apache Spark when the work has to spread across a cluster, or when analytics shares an engine with streaming and machine learning pipelines. That split comes from the projects' own descriptions: DuckDB is "a single-node database system", and Spark is "a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters." Both read the same open table formats, so you can decide per workload instead of once for the whole stack.

How each engine executes a query

DuckDB has no server to install or maintain. Its docs say it "does not run as a separate process, but completely embedded within a host process." It targets analytical (OLAP) work such as aggregations over entire tables and joins between large tables, using a columnar, vectorized execution engine.
Spark splits an application into a driver program and executors. On a cluster, the driver connects to a cluster manager, acquires executors on worker nodes and sends them tasks. Spark's homepage notes that you can train machine learning algorithms on a laptop and use the same code to scale to fault-tolerant clusters.

Spark vs DuckDB: how they compare

Spark cells cite spark.apache.org, and DuckDB cells cite duckdb.org.

Capability	Apache Spark	DuckDB
Design focus	Data engineering, data science and machine learning	Analytical (OLAP) SQL queries
Scaling model	One machine or a cluster (standalone, YARN or Kubernetes)	Single node, scaled vertically with more CPU, memory and disk
How it runs	Driver program plus executor processes	Embedded in the host process; Quack (beta) adds a server mode
Interfaces	Python, SQL, Scala, Java and R (R is deprecated)	SQL, plus client APIs for C, C++, Go, Python, R, Rust, Java and Node.js
Larger-than-memory data	Operators spill data to disk	Spills to disk for grouping, joins, sorting and windowing
Sharing with other users and jobs	Each application gets its own executors; cluster managers allocate resources across applications	In-process: one read-write process or many read-only ones; DuckLake or Quack (beta) for more writers
Documented data sources	Parquet, ORC, JSON, CSV, text, XML, Avro and JDBC	CSV, JSON and Parquet, plus lakehouse formats through extensions
License and stewardship	Apache License 2.0; an Apache Software Foundation project	MIT License; IP held by the non-profit DuckDB Foundation

How to choose between them

• Growth path. DuckDB scales up: a larger machine gives it more CPU, memory and disk. Spark scales out by adding executors on more nodes. If a workload will outgrow the largest machine you're willing to run, plan for Spark.
• Memory-heavy queries. DuckDB documents its limits. A query with several blocking operators can still raise an out-of-memory error, and aggregates such as list() and string_agg() don't offload to disk. Test your heaviest query before you commit.
• Concurrent writers. In in-process mode, one process reads and writes a DuckDB database file, and several processes can share it read-only. For more writers, DuckDB's docs point to the DuckLake format with a PostgreSQL catalog, or to Quack.
• Batch and streaming in one engine. Structured Streaming lets you express a streaming computation "the same way you would express a batch computation on static data." If dashboards depend on data that arrives continuously, one engine can run both the pipeline and the analytics.

Querying tables that Spark writes

The decision isn't exclusive. Spark pipelines can write Delta Lake or Apache Iceberg tables, and DuckDB can query them: its delta extension, built on Delta Kernel, reads and writes Delta tables, and its iceberg extension reads a table from its metadata or attaches an Iceberg REST catalog, which adds writes.

INSTALL delta;
LOAD delta;

SELECT region, count(*) AS orders
FROM delta_scan('s3://some/delta/table')
GROUP BY region;

How Databricks approaches Spark and DuckDB analytics

Databricks runs Spark workloads on serverless compute, where Databricks allocates and manages the resources, and can execute them with Photon, a native vectorized query engine that's compatible with Apache Spark APIs, so existing code runs without changes. Tables stay reachable from outside the platform: through open APIs and credential vending, Unity Catalog lets external engines, DuckDB among them, access its managed tables in Delta Lake or Apache Iceberg format. DuckDB's unity_catalog extension attaches a catalog from Databricks or Unity Catalog OSS and supports reads and standard inserts. Analysts keep a local SQL engine while governance stays in one catalog.

FAQs

Is DuckDB a distributed database?

No. DuckDB's FAQ describes a single-node system that scales vertically. Its Quack extension, in beta, turns one DuckDB instance into a server that other DuckDB clients connect to over HTTP.

Can Spark run on a single machine?

Yes. Spark's homepage describes an engine for "single-node machines or clusters", and its docs say it's easy to run locally on one machine.

Which engine suits embedded analytics inside an application?

DuckDB's in-process design targets that case: there's no server to run, and each release compiles into two files, a header and an implementation file. A Spark application runs as a driver program with executors.