How do I reduce GC pauses?

Summary

• Measure before you tune. The Spark tuning guide says the first step is to collect GC statistics. Add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to the executor Java options and read the worker logs. The Spark UI also reports GC time.
• Try serialized caching before other techniques. MEMORY_ONLY_SER keeps one byte array per RDD partition, and the guide recommends Kryo for it. In Python, objects are always pickled, so the serialized levels make no difference.
• Use data structures with fewer objects. The cost of garbage collection is proportional to the number of Java objects, so prefer arrays and primitive types over collections such as HashMap.
• Then tune the heap. Size Eden from the task working set, and lower spark.memory.fraction only if the old generation is nearly full. Spark 4.0.0 uses JDK 17 by default, which makes G1GC the default.
• On Databricks, Databricks Runtime 19 and above does not let you set the executor or driver JVM option properties in standard access mode.

How do I reduce GC pauses?

According to the Spark tuning guide, garbage collection can be a problem when your program has large churn in the RDDs it stores. It is usually not a problem in programs that read an RDD once and then run many operations on it. The guide's first step is to collect GC statistics, and the first technique it says to try is serialized caching.

What drives GC cost in Spark?

The guide's main point is that the cost of garbage collection is proportional to the number of Java objects. Data structures with fewer objects, such as an array of Ints instead of a LinkedList, lower it. GC can also be a problem when tasks' working memory interferes with the RDDs cached on the same nodes.

Which techniques reduce GC pauses?

Technique	What the Spark docs say
Serialized RDD storage	Use a serialized level such as MEMORY_ONLY_SER (Java and Scala). Spark stores each partition as one large byte array. The downside is slower access, because objects are deserialized on the fly.
Kryo	Set spark.serializer to org.apache.spark.serializer.KryoSerializer. Kryo serializes significantly faster and produces smaller output than Java serialization does, but it does not support all Serializable types and wants classes registered for best performance.
Leaner data structures	Prefer arrays of objects and primitive types over standard collections. The fastutil library has collections for primitive types. Avoid nested structures with many small objects.
Compressed pointers	With less than 32 GiB of RAM, set -XX:+UseCompressedOops so pointers take four bytes instead of eight.
Eden sizing	If there are many minor collections but few major GCs, give Eden more memory.
spark.memory.fraction	If the old generation is close to full, lower it so less memory goes to caching.

How do I measure GC first?

spark-submit \
  --conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
  my_app.py

The messages appear in the workers' logs (the stdout files in their work directories), not in the driver program. In the Spark UI, GC time on the stage detail page is the total JVM garbage collection time. If a full GC runs several times before a task completes, the guide says there isn't enough memory available for executing tasks.

How do I turn on serialized caching and Kryo?

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.storage.StorageLevel

val conf = new SparkConf()
  .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.registerKryoClasses(Array(classOf[MyClass1], classOf[MyClass2]))
val sc = new SparkContext(conf)

val rdd = sc.textFile("/path/to/data").persist(StorageLevel.MEMORY_ONLY_SER)

Without registration, Kryo still works but stores the full class name with each object, which the guide calls wasteful. In Python, stored objects are always serialized with Pickle, so the choice of serialized level does not matter there.

How do I size the Young generation?

The guide's example: a task reading from HDFS uses roughly the size of the block it reads, and a decompressed block is often 2 or 3 times the block size. For 3 or 4 tasks' worth of working space and 128 MiB blocks, Eden is estimated as 4*3*128MiB, which is 1,536 MiB. The guide sets the Young generation to -Xmn=4/3*E, about 2,048 MiB here, because the 4/3 accounts for the survivor regions.
Keep the Old generation large enough that its share of the heap exceeds spark.memory.fraction. The configuration reference recommends the default of 0.6 and warns that the lower it is, the more often spills and cached data eviction occur. The guide's view: it is better to cache fewer objects than to slow down task execution.
Spark 4.0.0 uses JDK 17 by default, which makes G1GC the default. With large executor heaps, the guide says it may be important to increase the G1 region size with -XX:G1HeapRegionSize.

What should I know on Databricks?

• Databricks runs one executor per worker node. Enter Spark properties in the compute's Spark config field, one key-value pair per line.
• On Databricks Runtime 19 and above, you can't set spark.executor.extraJavaOptions, spark.executor.defaultJavaOptions, spark.driver.extraJavaOptions or spark.driver.defaultJavaOptions in standard access mode. Creating or editing compute that sets them fails with an error.
• For suspected memory pressure, the Databricks Spark memory issues guide suggests doubling the memory per core to see if it changes the problem.

FAQs

Can I set the heap size with extraJavaOptions?

No. The docs say it is illegal to set maximum heap size (-Xmx) with this option. Use spark.executor.memory for executors. For the driver, use spark.driver.memory in cluster mode or --driver-memory in client mode.

What is the difference between extraJavaOptions and defaultJavaOptions?

spark.executor.defaultJavaOptions is prepended to spark.executor.extraJavaOptions. The docs say the default option is intended for administrators and the extra option for users.

How do I set driver JVM options in client mode?

In client mode, do not set spark.driver.extraJavaOptions through SparkConf in your application, because the driver JVM has already started. Use the --driver-java-options command line option or your default properties file.