Has the industry moved past Spark?

Summary

• There's no sourced, industry-wide answer, so check two things you can verify: what each engine cast as Spark's successor is designed to do, and whether Spark covers the workload in front of you.
• The open-source engines usually named in this debate document narrower scopes. DuckDB is a database for analytical queries embedded in a host process, Polars is a DataFrame library that splits work across one machine's cores, Ray is a framework for scaling Python and AI workloads, and Apache Flink is an engine for stateful computations over data streams.
• Apache Spark documents itself as "a unified analytics engine for large-scale data processing", with SQL, DataFrames, the pandas API, MLlib and Structured Streaming on one engine, locally or on Standalone, YARN or Kubernetes clusters.
• Choose per workload. Single-machine analytics, Python-native distributed ML, event-driven stream processing and large multi-table ETL map to different engines, and Spark, DuckDB and Polars can all work on the same Parquet files.
• Databricks runs Apache Spark under its compute clusters and SQL warehouses, and the same platform runs Ray clusters and single-node Python libraries next to it.

Has the industry moved past Spark?

It depends on the workload, because the engines usually cast as Spark's successors don't claim the same job. DuckDB, Polars, Ray and Apache Flink each document a narrower scope: in-process analytical SQL, single-machine DataFrames, Python-native distributed computing and stateful stream processing. Spark documents a general engine for large-scale data processing and keeps shipping releases, most recently 4.2.0 on July 14, 2026. The question worth answering is which engine fits each workload, and the answer can include more than one.

How the engines compare

Each cell paraphrases that project's own documentation.

Capability	Apache Spark	DuckDB	Polars	Ray	Apache Flink
Stated scope	Unified analytics engine for large-scale data processing	Database for analytical (OLAP) query workloads	DataFrame library for structured data	Framework for scaling AI and Python applications	Engine for stateful computations over unbounded and bounded data streams
How it runs	Driver and executors, locally or on Standalone, YARN or Kubernetes	Embedded in a host process, with no server to install	Inside your program, across the machine's CPU cores	A head node plus worker nodes, fixed-size or autoscaling	Designed to run in common cluster environments
Processing model	Batch, plus incremental and continuous stream processing with Structured Streaming	Complex queries over large parts of a stored dataset	Eager or lazy queries, plus a streaming engine for data larger than memory	Tasks, actors, and libraries for training, tuning, serving and batch inference	Batch queries on bounded data and continuous queries on unbounded streams
Interfaces	DataFrame API in Python, Scala and Java; Spark SQL; pandas API on Spark	SQL, with client APIs for Python, Java, R, Rust, Go and Node.js	Python, R and NodeJS; SQL through SQLContext	Python tasks and actors; Data, Train, Tune, Serve and RLlib	SQL on stream and batch data, the DataStream API and ProcessFunction
License	Apache License 2.0	MIT	MIT	Apache 2.0	Apache License 2.0

Matching workloads to engines

• Analytical SQL or DataFrames on data one machine holds. DuckDB and Polars run inside your process with no cluster. Spark's local mode (local[*]) is the option when the same code has to move to a cluster later.
• Distributed Python and ML built from tasks, actors or GPU steps. Ray's libraries cover training, tuning, serving and batch inference. Spark's MLlib offers a DataFrame-based machine learning API, and TorchDistributor supports distributed PyTorch training with PySpark.
• Stateful, event-driven stream processing. Flink documents exactly-once state consistency and event-time processing. Spark's Structured Streaming expresses streaming queries with the batch API, and Real-time Mode (Scala stateless queries in 4.1.0, a PySpark trigger in 4.2.0) adds continuous, low-latency execution.
• Large-scale ETL across many tables, in SQL and Python, batch and streaming. Spark's DataFrame and SQL engine fits this shape, and Spark Declarative Pipelines, added in 4.1.0, lets you define datasets and queries while Spark handles dependency ordering, checkpoints and retries.

Engines can share the same data

Choosing an engine per workload doesn't require separate copies of the data. Spark SQL reads and writes Parquet files, DuckDB queries a Parquet file directly with SELECT * FROM 'file.parquet', and Polars reads one with pl.read_parquet. A dataset that Spark writes as Parquet can be explored with either single-node tool, and a Spark job can pick up files the others produce.

How Databricks approaches this

Databricks documents Apache Spark as the technology powering its compute clusters and SQL warehouses. Photon, the Databricks-native vectorized query engine, is compatible with Apache Spark APIs, so Spark SQL and DataFrame code runs on it without changes. The platform isn't limited to Spark, though. Ray 2.3.0 and later can run as Ray clusters on top of Apache Spark clusters in Databricks, and notebook-scoped libraries installed with %pip bring Python tools such as Polars or DuckDB into the same notebooks. See What is Ray on Databricks? and What is Photon?.

FAQs

Are DuckDB and Polars replacements for Spark?

They target a different scope. DuckDB runs embedded in a host process and Polars splits work across one machine's CPU cores, so they suit jobs that fit on one machine, and Spark distributes work across executors on a cluster.

Is Apache Flink an alternative to Spark Structured Streaming?

For stream processing the two overlap. Flink documents stateful computations over bounded and unbounded streams with exactly-once state consistency, and Structured Streaming runs streaming queries on the Spark SQL engine with the batch API. Compare them on state handling, latency targets and whether batch and streaming code should share one API.

Is Spark still under active development?

Yes. Spark 4.2.0 shipped on July 14, 2026, and the project's published versioning policy schedules quarterly feature releases.