What is a Spark executor versus a driver?

Summary

• The driver is the process that runs your application's main() function and creates the SparkContext. Executors are processes on worker nodes that run tasks and keep data in memory or on disk.
• An application has one driver and its own executors. Executors stay up for the whole application and run tasks in multiple threads.
• The driver sends your code and tasks to the executors, so it must accept connections from them and be network addressable from the worker nodes.
• Deploy mode decides where the driver runs: inside the cluster in cluster mode, or in the submitting process in client mode.
• On Databricks, a compute resource has one driver node and worker nodes that each run one executor. Databricks Connect keeps only a thin client on your machine, and the driver stays on the cluster.

What is a Spark executor versus a driver?
A Spark application is a driver program plus executors on the cluster. The driver is the process running your main() function and creating the SparkContext. An executor is a process launched for the application on a worker node, which runs tasks and keeps data in memory or disk storage across them.

How do the driver and executors compare?

	Driver	Executor
What it is	The process running main() and creating the SparkContext	A process launched for an application on a worker node
How many	One per application	Several, and each application has its own
Job	Connects to the cluster manager, sends application code to executors, schedules tasks and sends them out	Runs tasks in multiple threads and keeps data in memory or on disk
Lifetime	The application's lifetime; it must accept connections from its executors throughout	Stay up for the whole application, unless dynamic allocation (off by default) releases idle ones
Main settings	spark.driver.memory (1g), spark.driver.cores (1, cluster mode only)	spark.executor.memory (1g), spark.executor.cores
Where to see it	Web UI, typically on port 4040	Executors tab of that UI

What does the driver do?

Spark applications run as independent sets of processes coordinated by the SparkContext in your main program. The SparkContext connects to a cluster manager (Spark standalone, YARN or Kubernetes), acquires executors, sends them your application code and then sends tasks to run.
Because the driver schedules tasks, the docs say it should run close to the worker nodes, preferably on the same local area network. Results of actions such as collect come back to the driver, and spark.driver.maxResultSize (default 1g) aborts a job whose total serialized results exceed the limit. Likewise, DataFrame.toPandas() collects all records to the driver, so use it on a small subset of the data.

What do executors do?

Executors run the tasks the driver sends them. A task is a unit of work sent to one executor, and a job is divided into stages, which are sets of tasks that depend on each other. Executors send heartbeats to the driver (spark.executor.heartbeatInterval, 10s by default) so the driver knows they are alive. Tasks from different applications run in different JVMs, so applications cannot share data without an external storage system.
A failed task is retried. spark.task.maxFailures (default 4) is the number of continuous failures of one task before Spark gives up on the job.

Where does the driver run?

spark-submit --deploy-mode chooses between cluster, which deploys the driver on the worker nodes, and client, which runs it locally as an external client (the default). In client mode the driver launches inside the spark-submit process, which suits the Spark shell. The docs say cluster mode is common when you submit from a machine far from the workers, such as a laptop, to reduce network latency between driver and executors.
Local mode is different: --master local[K] runs Spark locally with K worker threads, and local[*] uses as many threads as logical cores. A spark://HOST:PORT master URL points at a standalone cluster instead.

./bin/spark-submit \
  --master "local[4]" \
  --driver-memory 2g \
  myApp.jar

In client mode, set spark.driver.memory with --driver-memory or the defaults file, not from SparkConf inside the application, because the driver JVM has already started.

What happens when the driver fails?

The driver hosts the SparkContext, so recovery depends on the cluster manager. Standalone cluster mode can restart an application that exited with a non-zero exit code if you launch it with --supervise. On YARN, spark.yarn.maxAppAttempts caps the number of submission attempts. A Structured Streaming query set up with a checkpoint location can recover its progress and state after a failure.

How does this work on Databricks?

A Databricks compute resource has one driver node and zero or more worker nodes, and Databricks runs one executor per worker node. The driver node maintains the SparkContext, interprets the commands you run from a notebook or library, and runs the Apache Spark master that coordinates with the executors. By default the driver uses the same instance type as the workers, and you can choose a different type. Single node compute runs Spark locally, with the driver acting as both master and worker.
Databricks Connect is built on open-source Spark Connect, where the client does not run in the same process as the Spark driver. Your Python or Scala code runs on the client side, while the Spark code runs on the remote cluster.

FAQs

Can one machine be both driver and executor?

Yes, in local mode. local[K] runs Spark locally with K worker threads, and the RDD guide notes that in local mode a foreach function can execute within the same JVM as the driver.

Can executors run without a driver?

No. The SparkContext in the driver acquires executors and sends them tasks, and executors belong to one application.