How do I diagnose GC pauses in Spark?

Summary

• Read GC time in the Spark UI first. The stage detail page and the Executors tab both report it.
• Turn on JVM GC logging through spark.executor.extraJavaOptions, because the UI reports a total and the log shows individual pauses.
• Look for the patterns the Spark tuning guide names: many minor collections, repeated full GCs within a task, or an Old generation close to full.
• Match the fix to the pattern: fewer objects, a larger Eden, a lower spark.memory.fraction, smaller tasks, or larger G1 regions.

How do I diagnose GC pauses in Spark?

Measure first, then tune. Spark reports garbage collection (GC) time for each task and executor, so the Spark UI shows where the time goes, and the JVM's GC log shows the individual pauses that a total cannot. Once one stage or executor stands out, the tuning guide's checklist says what to change.

What does a GC pause look like in Spark?

The JVM heap has a Young and an Old generation. Young holds short-lived objects and is split into Eden and two Survivor regions. A minor GC runs when Eden is full, and a full GC is invoked when Old is close to full. Spark 4.0.0 and later use JDK 17 by default, which makes G1 the default collector, and Oracle's G1 reference says G1 does its collection and space reclamation in stop-the-world pauses.

Where do I see GC time?

Signal	Where	What it tells you
GC time	Stage detail page, summary metrics	Total JVM garbage collection time for the stage's tasks
GC time	Executors tab, one row per executor (driver included)	Which executor loses time to GC
jvmGCTime	REST API task metrics	Milliseconds in GC while running one task
totalGCTime, peakMemoryMetrics	REST API, /applications/[app-id]/executors	GC time summed per executor, plus minor and major GC counts and times
jstat -gcutil	A shell on the node, against one JVM	YGC, YGCT, FGC, FGCT, GCT; the JDK marks jstat experimental and unsupported

These are totals: a large figure can be one long pause or many short ones. Each Executors tab row also links to a thread dump, heap histogram, flame graph and the executor's logs. To see the UI after the application ends, set spark.eventLog.enabled to true before it starts.

How do I turn on GC logging?

The tuning guide says to add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to the Java options. The messages land in the stdout files in the work directories on the worker nodes, not on the driver. The JDK 17 reference maps -XX:+PrintGCDetails to -Xlog:gc* and logs time stamps in the framework, so on JDK 17 use unified logging:

./bin/spark-submit \
  --conf "spark.executor.extraJavaOptions=-Xlog:gc*" \
  --driver-java-options "-Xlog:gc*" \
  myApp.jar

The time decorator (-Xlog:gc*::time) adds ISO-8601 stamps you can line up with task times in the UI. The configuration guide adds two rules: never put -Xmx or Spark properties in these options (use spark.executor.memory), and in client mode set the driver's options with --driver-java-options or the properties file, not in the application's SparkConf.

Which GC log pattern needs which fix?

• Many minor collections, few major ones. Allocating more memory for Eden would help. Size Eden as an over-estimate of what each task needs, then set the Young generation to 4/3 of that to cover the Survivor regions.
• A full GC invoked repeatedly before a task completes. There is not enough memory available for executing tasks. Shuffle operations build a hash table within each task, so the guide's simplest fix is a higher level of parallelism. It recommends 2 to 3 tasks per CPU core.
• Old generation close to full. Reduce the memory used for caching by lowering spark.memory.fraction (default 0.6). The configuration guide recommends the default and warns that lower values mean more frequent spills and cached data eviction, so change it only when the GC statistics point there.
• Too many objects. GC cost is proportional to the number of Java objects. Prefer arrays and primitive types to collection classes such as HashMap. For cached data, try serialized caching such as MEMORY_ONLY_SER (Java and Scala) first, with Kryo.
• Large executor heaps on G1. Raise the region size with -XX:G1HeapRegionSize, a power of 2 from 1 to 32 MB in Oracle's reference.

What changes on Databricks?

Set JVM options in the compute's Spark config (Advanced > Spark), one key-value pair per line. On Databricks Runtime 19 and above in standard access mode, spark.executor.extraJavaOptions, spark.driver.extraJavaOptions and their defaultJavaOptions counterparts are restricted, and creating or editing compute that sets them fails. Standard access mode also has no executor logs and limits driver logs to workspace admins. Dedicated access mode gives the dedicated user or group, and workspace admins, access to both. Databricks' migration guide warns that custom executor or JVM settings can reduce performance, so remove them after diagnosis.
A Databricks knowledge base article on driver restarts describes frequent GC events in the driver's log4j output and Full GC messages in its stdout, and recommends using less memory at once when adding memory does not help.
For supported operations, Photon replaces the JVM-based Spark SQL execution engine with a native C++ runtime, which the docs say eliminates garbage collection pauses. It does not support UDFs, RDD APIs or Dataset APIs, and unsupported operations fall back to the Spark runtime, where this diagnosis still applies.

FAQs

What is the difference between a minor GC and a full GC?

A minor GC runs on Eden when it is full, copying live objects to a Survivor region. A full GC is invoked when Old is close to full. In G1, Oracle describes it as an in-place, stop-the-world full heap compaction used as a backup.

How much pause time is acceptable?

Set the threshold from your own latency goal. G1 works toward a pause-time goal, -XX:MaxGCPauseMillis (default 200), as a target, not a guarantee.