What are the executor-sizing rules of thumb?

Summary

• The Spark tuning guide's main rule of thumb is 2-3 tasks per CPU core in your cluster. It sets how many partitions a stage should have, not how many tasks run at once: by default a task uses one core.
• The hardware provisioning guide says to give Spark at most three quarters of a machine's memory, to provision at least 8-16 cores per machine, and to launch multiple executors on nodes with more than 200 GiB of RAM.
• Leave spark.memory.fraction (0.6) and spark.memory.storageFraction (0.5) at their defaults, because the configuration docs say leaving them is recommended.
• Measure garbage collection before tuning it. Since Spark 4.0.0, G1GC is the default, and the guide sizes the Young generation at 4/3 of the Eden estimate.
• On Databricks, executors map one-to-one to worker nodes, so you size by worker instance type and by minimum and maximum workers.

What are the executor-sizing rules of thumb?

Spark's documentation gives few hard numbers: 2-3 tasks per CPU core, at most three quarters of a machine's memory for Spark, and defaults for the memory fractions that it recommends leaving alone. Everything else, including heap size per executor, comes from measuring your workload.

How many tasks per core should I aim for?

The tuning guide says: "In general, we recommend 2-3 tasks per CPU core in your cluster." Spark reuses one executor JVM across many tasks and has a low task launching cost, so you can safely set parallelism above the number of cores.
An executor runs tasks at the same time according to spark.executor.cores and spark.task.cpus (default 1). A 4-core executor runs 4 tasks at once, and the rule means about 8-12 tasks queue through it per stage. With eight 4-core executors (32 cores), aim for roughly 64 to 96 partitions. Set partitions with spark.default.parallelism for RDDs or spark.sql.shuffle.partitions (default 200) for SQL shuffles.

Which settings size an executor?

Setting	Default	What it does
spark.executor.memory	1g (minimum 450m)	Memory per executor process
spark.executor.cores	1 in YARN mode, all available cores on the worker in standalone mode	Cores per executor
spark.executor.memoryOverhead	executor memory times spark.executor.memoryOverheadFactor (0.10)	Extra non-heap memory (YARN and Kubernetes) for "VM overheads, interned strings, other native overheads"
spark.memory.fraction	0.6	Fraction of (heap space - 300MB) for execution and storage. Leaving it at the default is recommended
spark.memory.storageFraction	0.5	Storage memory immune to eviction, as a fraction of the region above. Leaving it at the default is recommended
spark.task.cpus	1	Cores per task

Execution and storage share a unified region (M). Execution may evict storage only until storage falls under a threshold (R), and storage may not evict execution.

How much memory and how many cores per machine?

The hardware provisioning guide says Spark "can run well with anywhere from 8 GiB to hundreds of gigabytes of memory per machine," recommends allocating at most three quarters of it to Spark, and says to "provision at least 8-16 cores per machine." The Java VM "does not always behave well with more than 200 GiB of RAM," so on larger machines you can launch multiple executors in a single node.

How do I tune garbage collection?

Measure first. Add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to spark.executor.defaultJavaOptions or spark.executor.extraJavaOptions. The docs say it is illegal to set maximum heap size (-Xmx) there. Use spark.executor.memory.
If there are too many minor collections but not many major GCs, "allocating more memory for Eden would help." The guide's example: with 3 or 4 tasks' worth of working space, 128 MiB HDFS blocks, and decompressed blocks often 2 or 3 times the block size, Eden is estimated as 4*3*128MiB, which is 1536 MiB. Set the Young generation to 4/3 of Eden with -Xmn, about 2048 MiB here, which accounts for the survivor regions.
Since Spark 4.0.0, Spark uses JDK 17 by default, which makes G1GC the default collector. With large executor heaps, the guide says it may be important to increase the G1 region size with -XX:G1HeapRegionSize.

What if tasks run out of memory?

The tuning guide says a task's working set can be too large even when your data fits in memory, because shuffle operations build a hash table within each task. "The simplest fix here is to increase the level of parallelism, so that each task's input set is smaller."

How do I size executors on Databricks?

• One executor per worker: "Databricks runs one executor per worker node." Sizing an executor means choosing the worker instance type.
• Capacity: the Databricks docs say total executor cores "determines the maximum parallelism of a compute," and total executor memory "determines how much data can be stored in memory before spilling it to disk."
• Shuffle-heavy ETL: for a complex ETL job, Databricks recommends fewer workers to reduce the amount of data shuffled.
• Lakeflow Pipelines: Databricks recommends serverless for new pipelines. On classic compute, use enhanced autoscaling ("mode": "ENHANCED") with realistic min_workers and max_workers bounds, rather than a fixed worker count sized for peak load.

FAQs

When should I increase executor memory?

In the Spark web UI, the Stages tab reports task duration, GC time, and shuffle spill (memory and disk). If you see spills or OutOfMemoryError failures, the tuning guide calls increasing parallelism the simplest fix. If more partitions do not help, spark.executor.memory sets the heap per executor.

Should I set spark.default.parallelism?

It applies to RDD operations. For shuffles like join and reduceByKey, the default is the largest number of partitions in a parent RDD. With no parent RDD, it is the total number of cores on all executor nodes or 2, whichever is larger (in local mode, the number of cores on the local machine). For SQL and DataFrame shuffles, use spark.sql.shuffle.partitions.

What does spark.memory.fraction control?

It sets the fraction of (heap space - 300MB) used for execution and storage. The lower it is, the more often spills and cached data eviction occur. The docs recommend the default.