Is Spark too slow to start up for interactive work?

Summary

• Spark's startup cost comes from launching an application: a driver starts, connects to a cluster manager and acquires executors. Those executors stay up for the whole application, so the cost is paid once per session, not once per query.
• For interactive work, open a session once and keep sending it queries. A pyspark shell or a notebook kernel keeps executors in place between commands, and a Spark Connect server stays running between client sessions.
• Spark Connect separates the client from the server. A thin client such as pyspark-client sends unresolved logical plans over gRPC to a Spark server that's already running and ready to accept sessions.
• Dynamic allocation hands idle executors back and requests new ones when tasks queue up. For bursty notebook use, spark.dynamicAllocation.minExecutors keeps a floor of capacity in place.
• On Databricks, serverless compute for notebooks runs code without you provisioning a cluster, pools keep idle instances ready for classic compute, and Databricks Connect links IDEs to Databricks compute through Spark Connect.

Is Spark too slow to start up for interactive work?

It depends on how often you pay for startup. Before the first task runs, a Spark application launches a driver, connects to a cluster manager and acquires executors. Spark keeps those executors for the whole application, so the setup happens once per session and every later query reuses it. Interactive work fits Spark when you open one session and keep it, through a shell, a notebook, a Spark Connect server, or managed compute that handles provisioning for you.

Where Spark's startup time goes

The driver program runs your main() function and creates the SparkContext. The SparkContext connects to a cluster manager (Spark's standalone manager, Hadoop YARN or Kubernetes), which allocates resources. Spark then acquires executors: processes on worker nodes that run tasks and keep data in memory or on disk. Each application gets its own executors, and they stay up for the duration of the whole application.
Classic PySpark adds one piece. On the driver side, PySpark communicates with the driver JVM through Py4J, so a fresh pyspark process starts a JVM-based driver and then requests executors. A notebook or shell that stays open pays that cost once.

Startup options compared

Capability	Classic session (./bin/pyspark, notebook kernel)	Spark Connect client	Local mode (local[*])
What starts with a new session	A driver that creates the SparkContext and acquires executors from the cluster manager	A client session against a Spark server that's already running	Spark on one machine, with as many worker threads as logical cores
Client footprint	Full PySpark; Python talks to the driver JVM through Py4J	pyspark-client; the client doesn't use Py4J	Full PySpark on that machine
Where the client runs	It is the driver program, the process that runs main()	Outside the driver's process: application servers, IDEs, notebooks and other languages	On the same machine as the work
Who holds executors	Your application, for its whole lifetime	The server, which accepts sessions from client applications	No cluster manager; tasks run on local worker threads
API coverage	Full, including RDDs	DataFrame API and SQL; the protocol doesn't support RDDs	Full

Run one Spark Connect server and connect to it

Start the server from a Spark 4.x distribution, then point clients at it. The client encodes unresolved logical plans with protocol buffers and sends them over gRPC, and results return as Apache Arrow-encoded row batches.

# On the server host, from the extracted Spark distribution
./sbin/start-connect-server.sh

# pip install pyspark-client==4.2.0
from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
spark.sql("SELECT current_date() AS today").show()

A shell can use the same server with ./bin/pyspark --remote "sc://localhost". Spark 4.0.0 also added spark.api.mode: when it's set to connect, a Spark Classic application uses Spark Connect automatically by running a local Spark Connect server.

Keep capacity between queries with dynamic allocation

With spark.dynamicAllocation.enabled, Spark scales the executors registered with an application up and down with the workload. It requests executors in rounds once tasks have been pending for spark.dynamicAllocation.schedulerBacklogTimeout, and it removes an executor that stays idle for more than spark.dynamicAllocation.executorIdleTimeout. Interactive sessions pause, so a session can give executors back while you read results and request new ones for the next query. spark.dynamicAllocation.minExecutors sets a lower bound. Dynamic allocation also needs shuffle tracking, an external shuffle service or another documented shuffle option.

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .config("spark.dynamicAllocation.enabled", "true")
    .config("spark.dynamicAllocation.shuffleTracking.enabled", "true")
    .config("spark.dynamicAllocation.minExecutors", "2")  # floor kept between queries
    .getOrCreate()
)

How Databricks approaches interactive startup

• Serverless compute for notebooks. Pick Serverless in a notebook's compute drop-down, and new notebooks default to it on execution when no other compute is selected. You run workloads without provisioning compute in your cloud account: Databricks allocates and manages the resources. Serverless compute uses Spark Connect.
• Pools for classic compute. A pool is a set of idle, ready-to-use instances. When cluster nodes come from those instances, the docs say cluster start and auto-scaling times are reduced. Databricks doesn't charge DBUs for idle pool instances, though instance-provider billing applies.
• Databricks Connect. This client library connects IDEs, notebooks and custom applications to Databricks compute, and it's built on open-source Spark Connect.

See Serverless compute for notebooks, pools and Databricks Connect.

FAQs

Does Spark Connect remove startup time?

It moves it to the server. The server starts once and stays ready to accept Spark Connect sessions, so a client only opens a session. The client library runs outside the driver's process.

Why do executors disappear between my queries?

Dynamic allocation removes executors that stay idle past spark.dynamicAllocation.executorIdleTimeout. Raise spark.dynamicAllocation.minExecutors to keep a floor, knowing those executors hold cluster resources while the session sits idle.

How do I skip cluster setup in Databricks notebooks?

Attach the notebook to serverless compute from the compute drop-down. Databricks allocates and manages the compute, so you don't create or configure a cluster first.