How should I size Spark executors and cores?

Summary

• Size an executor with spark.executor.cores and spark.executor.memory (default 1g, minimum 450m). Cores default to 1 on YARN and to all worker cores in standalone mode.
• On YARN and Kubernetes the container limit is the sum of the heap, overhead, off-heap and PySpark memory settings.
• The Spark tuning guide recommends 2-3 tasks per CPU core in the cluster.
• The hardware guide suggests at most three quarters of a machine's memory for Spark, at least 8-16 cores per machine, and several executors per node above 200 GiB of RAM.
• On Databricks each worker node runs one executor, so you size by choosing the worker type and count.

How should I size Spark executors and cores?
The Apache Spark docs give defaults, limits and a few rules of thumb rather than one executor shape. This page applies them in the order you meet them: settings, cores, memory, partitions, then the Spark UI.

Which settings control executor sizing?

Setting	Default	What it controls
spark.executor.cores	1 in YARN mode, all available cores on the worker in standalone mode	Cores per executor
spark.task.cpus	1	Cores allocated to each task
spark.executor.memory	1g, minimum 450m	Memory per executor process
spark.executor.memoryOverhead	executorMemory * spark.executor.memoryOverheadFactor (0.10, or 0.40 for Kubernetes non-JVM jobs), minimum spark.executor.minMemoryOverhead (384m)	Extra non-heap memory per executor for VM overheads, interned strings and native overheads, supported on YARN and Kubernetes
spark.sql.shuffle.partitions	200	Partitions used when shuffling data for joins or aggregations
spark.default.parallelism	Depends on the operation and cluster manager	Default partitions for RDDs returned by join, reduceByKey and parallelize

How many cores should each executor get?

An executor runs tasks in multiple threads, and each task takes spark.task.cpus cores, so an executor with 4 cores and the default of 1 runs up to four tasks at once. In standalone mode an executor takes every core on its worker unless you set spark.executor.cores; with it set, several executors from one application can share a worker.
The hardware guide says Spark scales well to tens of CPU cores per machine because it performs minimal sharing between threads, and suggests at least 8-16 cores per machine. It names no ideal core count per executor, so treat any fixed number you read elsewhere as a starting point to test.

How much memory should each executor get?

The hardware guide says Spark runs well with anywhere from 8 GiB to hundreds of gigabytes of memory per machine, and recommends allocating at most three quarters of it to Spark so the operating system and buffer cache keep the rest. It notes that the Java VM does not always behave well with more than 200 GiB of RAM, so larger machines can launch several executors on one node.
Account for overhead. On YARN and Kubernetes the container for one executor is sized from the sum of spark.executor.memoryOverhead, spark.executor.memory, spark.memory.offHeap.size and spark.executor.pyspark.memory. Overhead includes PySpark memory when spark.executor.pyspark.memory is not set, and the docs say non-JVM tasks commonly fail with "Memory Overhead Exceeded" errors.
To estimate what your data needs, load part of it into an RDD and read its size on the Storage tab of the Spark UI. On garbage collection, the tuning guide says that if a full GC runs several times before a task completes, there is not enough memory for executing tasks. Spark 4.0.0 and later default to JDK 17 with G1GC, and with large executor heaps the guide says raising -XX:G1HeapRegionSize may matter.

How many partitions should a stage have?

The tuning guide says clusters are not fully utilized unless parallelism is high enough, and recommends 2-3 tasks per CPU core. For example, 6 executors with 4 cores each give 24 cores, so a starting point is 48 to 72 partitions per stage. Spark can run tasks as short as two hundred milliseconds efficiently because it reuses one executor JVM across many tasks, so going above the core count is safe.

How do I check the result in the Spark UI?

Open http://<driver-node>:4040. The Executors tab lists every executor, including the driver, with memory, disk, cores, task counts, shuffle totals and GC time. A stage page adds summary metrics such as GC time, peak execution memory and shuffle spill. Change one setting at a time and compare these numbers.

./bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 8g \
  --executor-cores 4 \
  --conf spark.executor.memoryOverhead=1g \
  myApp.jar

These values are examples, not recommendations. Deploy-related properties such as spark.executor.instances may not take effect when set from SparkConf at runtime, so set them through spark-submit options or the defaults file.

How does this change on Databricks?

A Databricks compute resource has one driver node and zero or more worker nodes, and Databricks runs one executor per worker node, so sizing means choosing the worker instance type and count. The best-practices guide notes that two workers with 16 cores and 128 GB of RAM each match the compute and memory of eight workers with 4 cores and 32 GB each. Databricks recommends serverless compute for most new workloads because it needs no configuration, and recommends against hardcoding Spark configs such as spark.executor.memory in compute or job definitions. On classic compute, use Databricks autoscaling instead of Spark dynamic allocation.

FAQs

Why do executors get killed even though I set enough memory?

On YARN and Kubernetes the limit applies to the whole container, which is the sum of the overhead, heap, off-heap and PySpark memory settings. For PySpark jobs, check spark.executor.memoryOverhead first, because the overhead includes PySpark memory when spark.executor.pyspark.memory is not set.

Can I change executor cores while the application runs?

You set spark.executor.cores when you submit the application. With dynamic allocation enabled, stage-level scheduling lets a stage request executors with different task and executor resources through a ResourceProfile, on standalone, YARN and Kubernetes.