What are the biggest downsides or limitations of Spark?

Summary

• Apache Spark's biggest costs come from its distributed design. Its own documentation says shuffles are expensive because they involve disk I/O, data serialization and network I/O, that memory and garbage collection need tuning, and that data skew can severely degrade join performance.
• Structured Streaming processes micro-batches by default. Continuous processing is still marked experimental with at-least-once guarantees, and Real-time Mode arrived only in Spark 4.1 for Scala stateless queries, with a PySpark trigger added in Spark 4.2.
• Long-running stateful streams need planning: the HDFS-backed state store keeps state in executor JVM memory, and changes to stateful operations aren't allowed between restarts from the same checkpoint.
• Spark is a processing engine, not a table store. ACID tables on object storage come from a format such as Delta Lake or Apache Iceberg, and self-managed clusters add cluster-manager and shuffle-service work.
• Databricks addresses some of these limits with serverless compute, Photon, Predictive Optimization and generally available real-time mode, but partitioning, join design and state design remain your responsibility.

What are the biggest downsides or limitations of Spark?

Spark's main limitations come from moving and holding data across a cluster. Shuffles write intermediate files and send data over the network, memory pressure shows up as garbage-collection pauses or OutOfMemoryError, and skewed keys create oversized tasks that drag down joins. Its streaming engine runs micro-batches by default, which gives exactly-once guarantees but not per-event processing. The Apache Spark 4.2 documentation describes each trade-off along with the settings that reduce it, so you can plan for them before they show up in production.

Spark's documented limitations and trade-offs

Limitation	What the Spark docs say	Open-source mitigation
Shuffle cost	"The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O."	Adaptive Query Execution, on by default since Spark 3.2.0, coalesces post-shuffle partitions at runtime
Memory and garbage collection	Programs "can be bottlenecked by any resource in the cluster: CPU, network bandwidth, or memory"; collect() can run the driver out of memory	Raise the level of parallelism, use Kryo serialization, and use take() for samples
Data skew	Skew "can severely downgrade the performance of join queries"	AQE splits skewed sort-merge join tasks (spark.sql.adaptive.skewJoin.enabled, default true)
Python UDFs	Data moves between JVM and Python processes	Arrow optimization, on by default for regular Python UDFs since Spark 4.2
Streaming latency	Micro-batch by default; continuous processing is experimental, at-least-once and limited to map-like operations	Real-time Mode: Scala stateless queries in Spark 4.1, PySpark trigger support in Spark 4.2
Large streaming state	The HDFS-backed store keeps state "in the JVM memory of the executors," which can cause high GC pauses	The RocksDB state store provider keeps state in native memory and on local disk
Changing stateful queries	Changes to stateful operations "are not allowed between restarts" from the same checkpoint	For arbitrary stateful operations, encode state with a scheme that supports schema migration, such as Avro
Self-managed clusters	Dynamic allocation needs an external shuffle service, shuffle tracking or another way to keep shuffle files when executors are removed	Set spark.dynamicAllocation.shuffleTracking.enabled; dynamic allocation works on standalone, YARN and Kubernetes

When these limitations matter most

These trade-offs weigh most heavily in a few workload shapes:

• Join- and aggregation-heavy jobs on large tables, where most of the work is shuffling.
• Python pipelines built on many UDFs, since each one exchanges data between the JVM and Python workers.
• Event-driven applications that must react to each event individually rather than once per micro-batch.
• Streams with large, long-lived state, or with stateful logic that changes often.
• Teams that run their own clusters and own upgrades, shuffle services and capacity.

How Databricks addresses some of these limitations

Databricks runs Apache Spark as a managed platform, and several features target the costs above. None of them removes the need to plan partitioning, join strategy and state design.

• Cluster operations. With serverless compute, Databricks automatically allocates and manages the compute for notebooks, jobs and Lakeflow pipelines.
• Execution engine. Photon, the Databricks-native vectorized query engine, runs SQL, DataFrame, ETL and stateless streaming workloads with no code changes. UDFs, RDD and Dataset APIs, and stateful streaming fall back to the Spark runtime.
• Table maintenance. Predictive Optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables in Delta Lake and Iceberg formats.
• Streaming. Real-time mode has been generally available on Databricks since March 2026 on classic compute, and its reference lists aggregations and transformWithState among supported operators. In Databricks Runtime 17.3 and above, RocksDB is the default state store, with changelog checkpointing on by default.

FAQs

Is Spark suitable for low-latency streaming?

By default, Structured Streaming processes data as a series of small batch jobs with exactly-once guarantees. Spark 4.1 added Real-time Mode for Scala stateless queries and Spark 4.2 added the trigger to PySpark, while continuous processing remains experimental and at-least-once.

Why do Spark jobs run out of memory?

The tuning guide names a common cause: one task's working set, such as a reduce task in groupByKey, grows too large. Calling collect() on a large dataset can also exhaust driver memory. Increase parallelism so each task's input shrinks, and use take() when you only need a sample.

Does Apache Spark provide ACID transactions on its own?

Spark relies on a table format for ACID guarantees on object storage. The Delta Lake protocol brings ACID properties to large collections of files, Apache Iceberg makes table changes atomic with serializable isolation, and Spark reads and writes both.

Does Photon remove the need to tune Spark?

No. Photon runs supported SQL and DataFrame operations on a vectorized engine without code changes, but unsupported operations run on the standard Spark runtime, and data layout and join design still shape every query.