Why is GC eating most of my task time?

Summary

• GC cost is proportional to the number of Java objects, so the Spark tuning guide's first fix is serialized caching with Kryo.
• Confirm GC is the bottleneck with the GC time in the Spark web UI and with JVM GC logging in spark.executor.extraJavaOptions.
• When the Old generation is close to full, lower spark.memory.fraction, because caching fewer objects is better than slowing down task execution.
• Size the Young generation only after measuring, since the JDK 17 docs advise against setting it for G1, the default collector on Spark 4.x.
• On Databricks, Photon replaces the JVM engine for supported operations but does not cover UDFs, RDD APIs or Dataset APIs.

Why is GC eating most of my task time?

GC dominates a task when the JVM spends its effort tracing and freeing objects instead of running your code. The Spark tuning guide gives the core reason: the cost of garbage collection is proportional to the number of Java objects. GC can also get worse when a task's working memory competes with the RDDs cached on the same node. The guide's order is to measure GC first, try serialized caching before other techniques, and only then tune heap regions.

How do I confirm GC is the bottleneck?

In the Spark web UI, the stage detail page reports GC time, the total JVM garbage collection time, in its task summary metrics, and the Executors tab has a GC time column. The REST API exposes jvmGCTime per task and totalGCTime per executor. For per-collection detail, add GC logging to the executor JVM options:

./bin/spark-submit \
  --conf "spark.executor.extraJavaOptions=-Xlog:gc*:stdout" \
  myApp.jar

The tuning guide shows -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps. Spark 4.x uses JDK 17 by default, and the JDK 17 java documentation maps PrintGCDetails to -Xlog:gc* and says the framework logs time stamps itself. The guide says the messages appear in the stdout files in the workers' work directories, not on the driver. If a full GC runs several times before a task completes, the guide says there isn't enough memory available for executing tasks.

Which fix matches which symptom?

Symptom in the GC stats	What to try
Many minor GCs, few major GCs	Give Eden more memory: size it above what a task needs and set the Young generation with -Xmn
Old generation close to full	Lower spark.memory.fraction to cache fewer objects, or shrink the Young generation
Large cache of many small objects	Use serialized caching with Kryo, and data structures with fewer objects
Large executor heaps on G1	Raise -XX:G1HeapRegionSize

How do I reduce the number of objects?

Start with the guide's first suggestion, serialized caching. MEMORY_ONLY_SER stores each RDD partition as one large byte array, which replaces many objects with one. The downside is slower access, because objects are deserialized on the fly, and the guide highly recommends Kryo for serialized caching.

import org.apache.spark.storage.StorageLevel
rdd.persist(StorageLevel.MEMORY_ONLY_SER)

Enable Kryo with spark.serializer=org.apache.spark.serializer.KryoSerializer; the default is org.apache.spark.serializer.JavaSerializer. In Python, stored objects are always serialized with Pickle, so choosing a serialized level makes no difference. For DataFrames, Spark SQL's in-memory columnar cache tunes compression to minimize memory usage and GC pressure.
Data structures matter too. Prefer arrays of objects and primitive types over standard collections such as HashMap, avoid nested structures with many small objects and pointers, and use numeric IDs or enumeration objects instead of strings as keys.

How do I size the Young generation and memory regions?

The JVM heap has a Young and an Old region, and Young splits into Eden and two Survivor regions. The guide aims to keep only long-lived RDDs in Old and to size Young for short-lived task objects. Its example: 128 MiB HDFS blocks that decompress to 2 or 3 times their size, with working space for 3 or 4 tasks, give an Eden of 4*3*128MiB, which is 1536 MiB. Young is 4/3 of Eden, because the extra covers the survivor regions, which is 2048 MiB:

--conf "spark.executor.extraJavaOptions=-Xlog:gc*:stdout -Xmn2048m"

Since Spark 4.0.0 the default JDK is 17, which makes G1GC the default collector, and the JDK 17 java documentation recommends not setting the young generation size for G1. Measure before and after. Setting -Xmx in this option is illegal; use spark.executor.memory.
spark.memory.fraction (default 0.6) sizes the unified execution and storage region as a fraction of heap space minus 300MiB. The rest holds user data structures, internal metadata and a safeguard against OOM errors. spark.memory.storageFraction (default 0.5) is the part of that region immune to eviction. The guide says typical users should not need to change either, but when the Old generation is close to full, caching fewer objects is better than slowing down task execution.

How does Databricks handle GC?

On classic compute, open the compute configuration, choose Advanced, open the Spark tab and enter one Spark config property per line. Serverless compute does not support setting most Spark properties for notebooks or jobs. Photon is the Databricks-native vectorized query engine. For supported operations it replaces the JVM-based Spark SQL execution engine with a native C++ runtime, which the docs say eliminates garbage collection pauses. Photon doesn't support UDFs, RDD APIs or Dataset APIs, and it falls back to the Spark runtime for unsupported operations, so the tuning above still applies to that work.

FAQs

Should I tune the driver JVM?

Start with executors, because tasks run there. The driver can matter when results are large: spark.driver.maxResultSize limits the total size of serialized results for actions such as collect, and the docs warn that a high limit may cause out-of-memory errors in the driver.

What is spark.memory.storageFraction?

It sets the size of the storage region R as a fraction of the unified region M, with a default of 0.5. Cached blocks inside R are immune to eviction by execution. A higher value leaves less working memory for tasks, which may then spill to disk more often.