How do I size executors, cores, and memory?

Summary

• Executor sizing in Apache Spark comes down to memory per executor (spark.executor.memory, default 1g), cores per executor (spark.executor.cores) and executor count (--num-executors or spark.executor.instances). The docs publish no memory-per-core ratio, so derive sizes from your nodes and measure.
• The docs give anchors: allocate at most three quarters of a machine's memory to Spark, plan for 2-3 tasks per CPU core, likely provision at least 8-16 cores per machine, and run several executors per node on machines with more than 200 GiB of RAM.
• Task slots per executor come from spark.executor.cores and spark.task.cpus (default 1). On YARN and Kubernetes, spark.executor.memoryOverhead is added on top of executor memory.
• Read GC time, shuffle spill and peak execution memory in the Spark UI, then change one setting at a time.
• On Databricks, one executor runs per worker node, so you size by worker type and worker count.

How do I size executors, cores, and memory?

Sizing means choosing the executor count, cores per executor and memory per executor. The Apache Spark docs give no single formula, so this page turns their defaults, hardware guidance and tuning rules into a repeatable loop.

What do the settings control?

Setting	Property or flag	What the docs say
Executor memory	spark.executor.memory, --executor-memory	Default 1g, minimum 450m
Executor cores	spark.executor.cores, --executor-cores	Default 1 in YARN mode, all available cores on the worker in standalone mode
Executor count	spark.executor.instances, --num-executors	Default 2 on YARN with static allocation
Cores per task	spark.task.cpus	Default 1
Non-heap memory	spark.executor.memoryOverhead	Executor memory times spark.executor.memoryOverheadFactor (0.10, or 0.40 for non-JVM jobs on Kubernetes), minimum 384m; YARN and Kubernetes

Each application gets its own executors, which run tasks in multiple threads. Task slots come from spark.executor.cores and spark.task.cpus, so with one CPU per task an 8-core executor runs up to 8 tasks at once.

How many tasks should a stage have?

The tuning guide says: "In general, we recommend 2-3 tasks per CPU core in your cluster." For 40 total cores that is about 80 to 120 tasks per stage. Control it with spark.default.parallelism for RDD shuffles, spark.sql.shuffle.partitions (default 200) for SQL and DataFrame shuffles, or repartition(n). coalesce(n) can only lower the count: if you request more partitions, "it will stay at the current number of partitions."

How much memory should each executor get?

• Per machine. Spark "can run well with anywhere from 8 GiB to hundreds of gigabytes of memory per machine." The docs recommend allocating at most three quarters of it to Spark, leaving the rest for the operating system and buffer cache. The Java VM "does not always behave well with more than 200 GiB of RAM," so you can launch multiple executors per node on bigger machines.
• Inside an executor. Execution and storage share a region sized by spark.memory.fraction (default 0.6 of heap minus 300 MiB). spark.memory.storageFraction (default 0.5) protects cached blocks from eviction.
• Containers. On YARN and Kubernetes the container holds spark.executor.memory plus spark.executor.memoryOverhead, spark.memory.offHeap.size and spark.executor.pyspark.memory.

What does a worked example look like?

This is arithmetic from the rules above, not a Spark default. Take workers with 16 cores and 64 GiB of RAM. Spark gets at most three quarters of the memory, or 48 GiB per node.

• One executor per node. --executor-memory 40g plus the default overhead of 0.10 times 40 GiB totals 44 GiB. --executor-cores 16 gives 16 task slots per node.
• Two executors per node. --executor-memory 20g --executor-cores 8 totals 22 GiB per executor, 44 GiB per node, with the same 16 slots.

./bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 4g \
  --executor-memory 20g \
  --executor-cores 8 \
  --num-executors 10 \
  my_app.py

The Spark Streaming guide's memory tips list "more executors with smaller heap sizes" to reduce GC pressure within each JVM heap, so try the second layout when GC time is high.

How do I tell whether the sizing is right?

Open the Spark UI, typically on port 4040 of the driver. The Executors tab lists each executor's memory, disk, cores, task counts, shuffle totals and GC time. The stage detail page adds peak execution memory and shuffle spill to memory and disk, and the Storage tab shows how much memory a cached dataset occupies. The tuning guide says that if a full GC runs several times before a task completes, "there isn't enough memory available for executing tasks." For out-of-memory errors in reduce tasks, the guide's simplest fix is to raise the level of parallelism so each task's input set is smaller.

What about streaming?

For Structured Streaming, memory depends on the state store provider: the default HDFS provider keeps state in the executor JVM, and the RocksDB provider uses native memory and local disk. spark.streaming.backpressure.enabled belongs to the legacy Spark Streaming (DStreams) settings. Structured Streaming limits input with source options such as maxOffsetsPerTrigger.

How does Databricks handle executor sizing?

• Databricks runs one executor per worker node, so you size by choosing the worker instance type and the number of workers, or an autoscaling range.
• Databricks generally recommends against configuring most Spark properties, and the spark.dynamicAllocation.* properties are unsupported on classic compute. Use Databricks autoscaling instead, except for Structured Streaming jobs, where Databricks advises against compute autoscaling.

FAQs

What if I don't set the number of executors?

On YARN, spark.executor.instances defaults to 2. In standalone mode each executor takes all the cores on a worker unless you set spark.executor.cores, which lets several executors share a worker.

Can I change executor settings while the app runs?

Deploy-related properties such as spark.executor.instances "may not be affected when setting programmatically through SparkConf in runtime," so set them with spark-submit or the defaults file. With dynamic allocation (spark.dynamicAllocation.enabled, default false) the executor count scales with the workload.