What are the single-node alternatives to Spark?

Summary

• The main single-node alternatives to Apache Spark are pandas, Polars and DuckDB, and Spark itself can also run on one machine in local mode.
• pandas works on in-memory data structures. For datasets close to or above RAM, the pandas docs recommend loading less data, using efficient data types, chunking, or another library.
• Polars and DuckDB both document ways to process data that doesn't fit in memory: Polars through its streaming engine, DuckDB by spilling to disk (out-of-core processing).
• Choose by the interface you work in (DataFrames or SQL), how close the data is to memory limits, and whether the job may outgrow one machine.
• On Databricks, single node compute runs Spark locally with no worker nodes, and %pip adds other Python libraries to a notebook's environment.

What are the single-node alternatives to Spark?

The common single-node alternatives to Apache Spark are pandas, Polars and DuckDB. pandas provides in-memory DataFrame structures for Python, Polars is a DataFrame library with a Rust core and a query optimizer, and DuckDB is an in-process SQL database designed for analytical queries. Spark also runs on one machine: the local[*] master URL starts it with as many worker threads as your machine has logical cores. Which one fits depends on the interface you want, how close your data sits to memory limits, and whether the job will need a cluster later.

pandas vs Polars vs DuckDB vs Spark local mode: how they compare

Each cell below comes from the project's own documentation.

Capability	pandas	Polars	DuckDB	Spark in local mode
What it is	Python package with DataFrame and Series data structures	DataFrame library for structured data, core written in Rust	Relational database for analytical (OLAP) queries	The Spark engine running locally
Main interface	Python DataFrame API	Python, R and NodeJS APIs, with eager and lazy modes	SQL, plus APIs for Python, R, Java, Go, Rust, Node.js, C and C++	DataFrame API, Spark SQL and the pandas API on Spark
How it runs	Inside your Python process, on in-memory data	Inside your process, dividing work among available CPU cores	Embedded in the host process rather than run as a separate process; no external dependencies	Driver plus worker threads; local[*] uses one per logical core
Data larger than memory	Docs advise loading less data, efficient data types, chunking or other libraries	Streaming engine runs queries in batches	Out-of-core processing by spilling to disk	Operators spill to disk when data doesn't fit in memory
Reading Parquet	pandas.read_parquet	pl.read_parquet, or pl.scan_parquet for a lazy query	SELECT * FROM 'file.parquet' or read_parquet	spark.read.parquet
License	BSD 3-Clause	MIT	MIT	Apache License 2.0

How to choose a single-node engine

• Interface. DuckDB is a SQL database. pandas and Polars are DataFrame APIs. Spark offers DataFrames, SQL and pandas-style syntax through pyspark.pandas.
• Memory headroom. The pandas docs note that some operations make intermediate copies, so even datasets that are a sizable fraction of memory become unwieldy. Polars' streaming engine (collect(engine="streaming")) and DuckDB's spilling to disk both let you process data larger than memory.
• Query planning. Polars' lazy API applies predicate and projection pushdown while reading. DuckDB reads only the Parquet columns a query needs and can skip parts of a file using zonemaps, when the file has them.
• Concurrency. DuckDB's native database file allows one read-write process, or many processes in read-only mode. Plan for that if several services must write.
• Growth path. If the job may outgrow one machine, PySpark code moves with it: spark-submit launches applications on every supported cluster manager (Standalone, Hadoop YARN, Kubernetes) through a uniform interface.

The tools also combine well. DuckDB can query a Polars or pandas DataFrame by its variable name and hand results back through Apache Arrow (the pyarrow library must be installed):

import duckdb
import polars as pl

events = pl.read_parquet("events.parquet")
counts = duckdb.sql("SELECT user_id, count(*) AS n FROM events GROUP BY user_id").pl()

How Databricks approaches single-node work

On Databricks, single node compute runs Spark locally: the driver acts as both master and worker, with no worker nodes. Databricks intends it for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries, and recommends multi-node compute for larger jobs with distributed workloads. A single node resource can't be converted to multi-node later, so create a new compute resource when a job grows.
Notebook-scoped libraries installed with %pip add Python packages to one notebook's environment without affecting other notebooks on the same cluster. For existing pandas code that has to scale, the pandas API on Spark (import pyspark.pandas as ps) provides pandas-equivalent APIs that run on Apache Spark.

FAQs

Can Apache Spark run on a single machine?

Yes. Setting the master URL to local[*] runs Spark locally with as many worker threads as logical cores. The same application can later be submitted to a Standalone, Hadoop YARN or Kubernetes cluster with spark-submit.

Does pandas handle data larger than memory?

The pandas docs describe its data structures as built for in-memory analytics. For data near or above memory size they suggest loading less data, using efficient data types, processing in chunks, or using another library.

Can DuckDB query a pandas or Polars DataFrame directly?

Yes. DuckDB resolves a DataFrame's variable name inside SQL through replacement scans, and the .pl() method returns results as a Polars DataFrame using its Apache Arrow integration.

Is single node compute on Databricks still Apache Spark?

Yes. It runs Spark locally on the driver, with no worker nodes. Databricks intends it for small data and for non-distributed workloads such as single-node machine learning libraries.