How do I fix "Container killed by YARN for exceeding memory limits"?

Summary

• YARN kills a container whose process exceeds its physical memory limit. For a Spark executor, that limit is the sum of spark.executor.memory, spark.executor.memoryOverhead, spark.memory.offHeap.size and spark.executor.pyspark.memory.
• Raise spark.executor.memoryOverhead, as Spark's own log message suggests. It defaults to executor memory times 0.10, with a 384 MiB minimum.
• Python workers draw on the overhead unless you set spark.executor.pyspark.memory. In cluster mode, size the driver's container with spark.driver.memoryOverhead.
• Set these at launch with spark-submit --conf or spark-defaults.conf, then confirm the cause with yarn logs -applicationId <app ID>.

How do I fix "Container killed by YARN for exceeding memory limits"?

Raise spark.executor.memoryOverhead, the memory Spark reserves outside the JVM heap, rather than the heap itself. YARN's NodeManager enforces a physical memory limit on every container (yarn.nodemanager.pmem-check-enabled, default true). Spark requests each executor container as the sum of spark.executor.memory, spark.executor.memoryOverhead, spark.memory.offHeap.size and spark.executor.pyspark.memory. When the executor's process goes past that total, YARN terminates the container, and Spark's log message says "Consider boosting spark.executor.memoryOverhead."
Adding heap doesn't address the cause. spark.executor.memory sets the executor's maximum heap size (-Xmx), so the heap can't grow past its own share; when it fills, the JVM throws OutOfMemoryError instead. A YARN kill means memory outside the heap went over budget: VM overheads, interned strings, other native memory, or Python worker processes.

What does the error message say?

For memory kills, Spark's YARN allocator turns the exit status YARN reports into one of two messages:

YARN exit status	Spark message (current source)	What the message suggests
KILLED_EXCEEDED_PMEM	"Container killed by YARN for exceeding physical memory limits."	Boost spark.executor.memoryOverhead
KILLED_EXCEEDED_VMEM	"Container killed by YARN for exceeding virtual memory limits."	Boost spark.executor.memoryOverhead or yarn.nodemanager.vmem-pmem-ratio, or disable yarn.nodemanager.vmem-check-enabled (citing YARN-4714)

Both messages include YARN's usage figure, in the form " of physical memory used" (or "virtual memory used"). Spark 2.4 printed the older wording in this page's title and suggested spark.yarn.executor.memoryOverhead. That key has been an alternate name for spark.executor.memoryOverhead since Spark 2.3, so set the current key.

How does Spark size an executor container?

spark.executor.memory defaults to 1g. spark.executor.memoryOverhead defaults to executor memory times spark.executor.memoryOverheadFactor (0.10), with a floor of spark.executor.minMemoryOverhead (384m, a property added in Spark 4.0.0). If you set the overhead directly, Spark ignores the factor. spark.memory.offHeap.size (default 0) adds to the request when you enable off-heap memory. spark.executor.pyspark.memory is not set by default, and then Spark doesn't limit Python's memory: Python workers share the overhead space with other non-JVM processes.
So --executor-memory 4g with nothing else set gets a tenth of 4 GiB as overhead, which clears the 384 MiB floor, and Spark requests about 4.4 GiB per executor. If Python workers and native memory together need more than that tenth, YARN kills the container.

Which setting should you change?

Symptom	Likely cause	Fix
Executors killed for exceeding physical memory	Native and VM memory outgrew the overhead	Raise spark.executor.memoryOverhead, or spark.executor.memoryOverheadFactor so it scales with the heap
Same error in a PySpark job with Python UDFs	Python workers draw on the overhead	Raise the overhead, or set spark.executor.pyspark.memory so Python gets its own capped share of the request
Killed for exceeding virtual memory	YARN's virtual memory check (on by default, ratio 2.1)	Raise the overhead, or ask the cluster admin about the two YARN settings the message names
Driver's container killed in cluster mode	The driver runs inside the YARN application master	Raise spark.driver.memoryOverhead; that container is spark.driver.memory plus this overhead
Application master killed in client mode	The AM's own container (spark.yarn.am.memory, default 512m)	Raise spark.yarn.am.memory or spark.yarn.am.memoryOverhead

How do I set the overhead?

Set memory at launch. The configuration guide warns that deploy-related properties may not take effect when set through SparkConf at runtime, so pass them to spark-submit or put them in conf/spark-defaults.conf:

./bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 4g \
  --executor-memory 4g \
  --conf spark.executor.memoryOverhead=1g \
  --conf spark.driver.memoryOverhead=1g \
  --conf spark.executor.pyspark.memory=1g \
  my_job.py

Each executor now requests 6 GiB (4 GiB heap, 1 GiB overhead, 1 GiB for Python), and the driver's container 5 GiB. Overhead values are in MiB unless you add a unit.

How do I confirm the cause?

Run yarn logs -applicationId <app ID> to print the logs from all containers of the application. That needs log aggregation (yarn.log-aggregation-enable, default false); without it, logs stay on each node under YARN_APP_LOGS_DIR. Search for "Container killed by YARN" and read the usage figure beside it before you pick a new overhead.

How Databricks approaches this

On Databricks classic compute, you pick driver and worker instance types, and the docs note that different instance families "fit different use cases, such as memory-intensive or compute-intensive workloads." Spark properties go in the compute's Spark config field, one key-value pair per line. On serverless compute, "Databricks automatically allocates and manages the necessary compute resources." Executor memory isn't among the few Spark properties serverless supports, and user code such as UDFs, map and mapPartitions can't exceed 1 GB of memory.

FAQs

Should I increase the heap or the overhead?

The overhead. Raising spark.executor.memory lifts the default overhead by only a tenth of the added heap, so set spark.executor.memoryOverhead explicitly.

Does tuning spark.memory.fraction help?

No. spark.memory.fraction (default 0.6) is the fraction of heap space minus 300MB used for execution and storage. It doesn't touch memory outside the heap, and the docs recommend leaving it at the default.

What if YARN rejects the bigger request?

Each container request must fit yarn.scheduler.maximum-allocation-mb (default 8192). Larger requests throw an InvalidResourceRequestException, so the cluster admin raises that limit or you shrink the heap to make room for overhead.

Can I set overhead per task?

No. It's allocated per executor process, and an executor runs its tasks in multiple threads, so concurrent tasks share it.