Is Polars replacing Spark?

Summary

• Polars and Apache Spark document different execution scopes. Polars' docs say it has always been optimized for performance on a single machine, with a separate distributed engine for datasets too large for one node, and Spark runs the same code on one machine or across a cluster.
• Polars' lazy API lets its optimizer push filters and column selection into the scan, and its streaming engine processes a query in batches so the data doesn't have to fit in memory at once.
• "Streaming" means different things in the two projects: batched, out-of-core execution of a single query in Polars, and continuous processing of arriving data in Spark's Structured Streaming.
• For scale beyond one machine, Polars documents a distributed engine available with Polars Cloud, a managed offering built on the open-source project, and with Polars On-Prem, which runs Polars clusters in your own infrastructure.
• The two interoperate through Apache Arrow, and on Databricks Polars installs as a notebook-scoped library with %pip.

Is Polars replacing Spark?

Polars covers DataFrame work that fits on one machine, and Spark covers work that needs a cluster or continuous stream processing. That split follows from how each project describes itself. Polars is a DataFrame library with a Rust core, available for Python, R and NodeJS, and its stated goals include using "all available cores on your machine" and handling "datasets much larger than your available RAM." Spark is a multi-language engine for single-node machines or clusters, with Structured Streaming built in. The practical question is which jobs belong on which engine, and a single pipeline can use both.

How Polars runs a query

Polars offers eager and lazy APIs. In lazy mode nothing runs until you call collect, so the query planner can apply predicate pushdown (filtering rows while reading) and projection pushdown (reading only the columns a query needs). The streaming engine then executes the plan in batches, which lets a query process datasets that don't fit in memory. Operations without a streaming implementation fall back to the in-memory engine. In the current API reference, collect() defaults to the streaming engine unless you configure another one.

import polars as pl

q = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("status") == "error")
    .group_by("service")
    .agg(pl.len())
)
df = q.collect(engine="streaming")

Polars consumes and produces Apache Arrow data, often without copying, but it runs its own compute and buffer implementations instead of building on PyArrow. For SQL, SQLContext translates queries into Polars expressions; the docs treat the DataFrame interface as primary.

Polars vs Spark: how they compare

Polars cells cite docs.pola.rs (and docs.cloud.pola.rs for the distributed engine); Spark cells cite spark.apache.org, plus the Delta Lake and Iceberg docs for table formats.

Capability	Polars (open source)	Apache Spark
Execution scope	Work divided among the available CPU cores of one machine	Driver plus executors on a cluster, or local mode
Scale beyond one machine	Distributed engine available with Polars Cloud and Polars On-Prem	Built in, through the standalone manager, YARN or Kubernetes
APIs	Expression-based DataFrame API (eager and lazy) for Python, Rust, R and NodeJS; SQL through SQLContext	DataFrame API and Spark SQL for Python, Scala, Java and R (R is deprecated); pandas API on Spark
Query optimization	Lazy API with predicate and projection pushdown	Adaptive Query Execution adapts the plan at runtime
Data larger than memory	Streaming engine runs the query in batches	Operators spill data to disk
What "streaming" means	Batched, out-of-core execution of one query	Structured Streaming: incremental processing of continuously arriving data
Delta Lake and Iceberg	scan_delta, write_delta, scan_iceberg and write_iceberg; write_iceberg and some parameters of the others are marked unstable	Delta Lake and Iceberg both ship Spark integrations
License	MIT	Apache License 2.0

Which jobs fit where

• Polars fits single-machine ETL, feature preparation and notebook analysis where inputs and intermediate results stay within one machine's CPU, memory and disk, including larger-than-memory scans through the streaming engine.
• Spark fits joins and aggregations over data too large for one machine, continuous pipelines over event streams, and shared jobs where a cluster manager allocates resources across applications.
• Both fit together when a pipeline reduces data in Spark and hands a small result to Polars. Arrow makes the hand-off direct:

import polars as pl

summary = spark.table("orders").groupBy("region").count()
regions = pl.from_arrow(summary.toArrow())  # collects to the driver

spark_regions = spark.createDataFrame(regions.to_arrow())

DataFrame.toArrow() is new in PySpark 4.0.0, is documented as a developer API, and loads all data into the driver's memory, so keep it for small results. createDataFrame accepts a pyarrow.Table from Spark 4.0.0 on.

How Databricks approaches Polars and Spark

On Databricks, Polars installs like any other Python package. Notebook-scoped libraries added with %pip install polars are available to the current notebook and its jobs, and other notebooks attached to the same cluster aren't affected. For jobs on serverless compute, Databricks recommends environments to install and manage libraries. Because Spark and Polars share the notebook, one job can aggregate a large table with Spark and finish the analysis in Polars.

FAQs

Can Polars run on a cluster?

The open-source library divides work among the available CPU cores of the machine it runs on. Its docs describe a distributed engine, available with Polars Cloud, that "lets you scale your Polars queries beyond a single machine," and Polars On-Prem runs Polars clusters inside your own infrastructure.

Is Polars' streaming engine the same as Spark Structured Streaming?

No. Polars' streaming engine executes one query in batches to limit memory use. Structured Streaming is "a scalable and fault-tolerant stream processing engine built on the Spark SQL engine," made for data that keeps arriving.

Can Polars read Delta Lake and Iceberg tables?

Yes. scan_delta and scan_iceberg read them lazily, and write_delta and write_iceberg write them. Polars' API reference marks write_iceberg as unstable, along with some parameters of the other three functions, so pin library versions in production.