Why do companies move off Spark?

Summary

• A move off Apache Spark makes sense when a workload doesn't fit Spark's distributed design. Spark's own documentation describes the trade-offs involved: shuffles, JVM memory management, cluster operations and micro-batch streaming.
• Data that fits on one machine is the clearest case. In-process engines such as DuckDB and Polars run without a cluster, and DuckDB spills to disk for larger-than-memory joins, aggregations and sorts.
• Other reasons include self-managed cluster work, Python and JVM tuning, and per-event streaming. Several have answers inside Spark: adaptive query execution, Arrow-optimized Python UDFs, native execution plugins and Real-time Mode.
• Moving one workload doesn't require moving the data. With Delta Lake or Apache Iceberg tables, Spark, DuckDB, Polars, Flink and other engines can work on the same tables.
• On Databricks, serverless compute takes over cluster management, Photon runs supported operations in a native C++ runtime, and code keeps using Spark's APIs.

Why do companies move off Spark?

A workload is a candidate to move off Apache Spark when it doesn't need what Spark is built for: spreading data and computation across a cluster. Spark's documentation describes the costs of that design. Shuffles involve disk I/O, data serialization and network I/O, JVM memory needs tuning, and someone has to run the clusters. For large, parallel jobs those costs buy scale. For small datasets or per-event processing they can be overhead, and another engine may fit the job. Treat it as a per-workload decision, because several of the reasons have fixes that don't require leaving Spark.

The reasons, and what's behind them

Reason	What the documentation says	Check before moving
The data fits on one machine	DuckDB runs "completely embedded within a host process" and handles larger-than-memory workloads by spilling to disk; Polars' core is written in Rust	Whether the data or job will outgrow one machine; Spark also runs on single-node machines
Cluster operations	Spark runs on Standalone, YARN or Kubernetes, and dynamic allocation needs an external shuffle service, shuffle tracking or decommissioning	Whether a managed or serverless Spark service removes that work
Shuffle-heavy jobs	"The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O."	Adaptive query execution, on by default since Spark 3.2.0, coalesces post-shuffle partitions and splits skewed join tasks
JVM memory and garbage collection	The tuning guide notes that "the cost of garbage collection is proportional to the number of Java objects"	Serialized caching and data structures with fewer objects, per the tuning guide; native plugins such as Apache Gluten or Apache DataFusion Comet; or Photon on Databricks
Python overhead	Python UDFs move data between JVM and Python processes	Arrow optimization, on by default for regular Python UDFs since Spark 4.2
Per-event streaming	Structured Streaming runs micro-batches by default; continuous processing is experimental, with at-least-once guarantees	Real-time Mode, added in Spark 4.1.0 with Scala stateless support, or a stream processor such as Apache Flink

Each row is a trade-off rather than a defect, and the right engine depends on the workload's shape. Apache Flink, for example, describes itself as "a framework and distributed processing engine for stateful computations over unbounded and bounded data streams," so a team weighing it for streaming should compare its state and delivery guarantees with Structured Streaming's on the same pipeline.

What moving off Spark costs

• Code. DataFrame, Spark SQL and Structured Streaming code has to be rewritten for the new engine's API or SQL dialect.
• Coverage. Spark offers Spark SQL, pandas API on Spark, MLlib, GraphX and Structured Streaming in one engine. A replacement may cover only part of that, so a full move can mean running several systems.
• Data, which doesn't have to move. With Delta Lake or Apache Iceberg tables, other engines read the same files. DuckDB has delta and iceberg extensions, the delta-rs deltalake package integrates with Polars and DuckDB, and Iceberg's site names Spark, Trino, Flink, Presto, Hive and Impala as engines that can work with the same tables at the same time.

That makes a split possible: keep Spark for distributed jobs and point a single-node engine at the same table for small ones. DuckDB can read a Delta table that Spark wrote:

INSTALL delta;
LOAD delta;
SELECT * FROM delta_scan('s3://<bucket>/<path-to-delta-table>') LIMIT 10;

How Databricks approaches these trade-offs

Databricks keeps Spark's APIs and takes on the operational and engine work. With serverless compute, Databricks automatically allocates and manages compute for notebooks, jobs and Lakeflow pipelines, and because it's versionless, Databricks upgrades the runtime. Photon runs supported SQL and DataFrame operations in a native C++ runtime and falls back to Spark for the rest. SQL warehouses serve SQL analytics, and Databricks Connect, built on open-source Spark Connect, connects IDEs, notebooks and custom applications to Databricks compute. Through Unity Catalog's external access, other engines can read the same Delta Lake and Iceberg tables with credential vending.

FAQs

Is Apache Spark still actively developed?

Yes. Spark 4.2.0 was released on July 14, 2026, and maintenance releases 4.1.3 and 4.0.4 followed on July 15, 2026.

When does a workload belong on Spark?

When its data or joins need more than one machine, when one engine should cover batch, streaming and machine learning, or when the team already has working Spark code and skills.

Can I use DuckDB or Polars on data that Spark writes?

Yes, if Spark writes an open table format. DuckDB's delta extension reads Delta tables, its iceberg extension reads Iceberg tables, and Polars works with Delta tables through delta-rs.

Is moving to managed Spark the same as moving off Spark?

No. A managed service changes who operates the clusters, while DataFrame, SQL and streaming code keeps using Spark's APIs.