Why does my Spark Connect session expire?

Summary

• A Spark Connect session is state held by the Spark Connect server and cached under your user ID and session ID. The client keeps only the ID, so the session ends whenever the server loses or releases that entry.
• The error class tells you which happened: INVALID_HANDLE.SESSION_CHANGED means the server driver restarted, SESSION_CLOSED means the session was released (for example by spark.stop()), and SESSION_NOT_FOUND means the server has no session under that ID.
• INVALID_HANDLE.OPERATION_ABANDONED concerns one query, not the session: the server dropped an execution it considered abandoned because of inactivity.
• To recover, open a new session, replay its setup (SQL configs, temporary views, functions, artifacts) and rebuild your DataFrames, because each DataFrame stays tied to the session that created it.
• On Databricks, serverless sessions no longer expire after 10 minutes of inactivity as of Databricks Connect 17.3.0, and version 16.4 and above restores idle serverless sessions on a best-effort basis.

Why does my Spark Connect session expire?

Because the session lives on the server, not in your client. Spark Connect gives each client a server-side session that holds its SQL configurations, temporary views, registered functions and uploaded artifacts, and the server caches it under your user ID and session ID. The client sends that ID with every request. When the server no longer has a usable session for it (the driver restarted, the session was released, or the server can't find it), your next call fails with an INVALID_HANDLE error. Read the subclass before changing anything, because each one points to a different cause.

Match the error to the cause

Error	Message in the Spark docs	What happened	What to do
INVALID_HANDLE.SESSION_CHANGED	"The existing Spark server driver instance has restarted. Please reconnect."	The server process restarted, and every in-memory session went with it	Open a new session and replay its setup
INVALID_HANDLE.SESSION_CLOSED	"Session was closed."	The session was released, often by spark.stop() elsewhere in the program	Stop shared sessions in one place only, then reconnect
INVALID_HANDLE.SESSION_NOT_FOUND	"Session not found."	The server holds no session under that user ID and session ID	Reconnect with a new session
INVALID_HANDLE.OPERATION_ABANDONED	"Operation was considered abandoned because of inactivity and removed."	One query's execution was dropped; the session may be fine	Rerun the query

Two protocol details explain the first two rows. Responses carry a server-side session ID that the client sends back, and the protocol says the server "can use this to validate that the server side session has not changed", which is how a restart surfaces as SESSION_CHANGED. And the ReleaseSession call releases every execution in the session, after which "any further requests for the session with that session_id for the given user_id will fail."

Recover cleanly in PySpark

Put session setup in one function so you can rebuild everything in a single call. builder.create() (PySpark 3.5.0 and above) always opens a new session, and on a Spark Connect session spark.stop() releases the session on the server and closes the gRPC connection:

from pyspark.sql import SparkSession

REMOTE = "sc://connect-host:15002"

def open_session():
    spark = SparkSession.builder.remote(REMOTE).create()
    spark.conf.set("spark.sql.session.timeZone", "UTC")
    spark.read.parquet("/data/orders").createOrReplaceTempView("orders")
    return spark

spark = open_session()

After an INVALID_HANDLE session error, call spark.stop(), which is best-effort and idempotent and won't raise on a dead session, then run spark = open_session() again. Recreate your DataFrames as well. DataFrame.sparkSession returns the session that created a DataFrame, so old DataFrames still point at the expired one.

Share one session across clients

To let several processes use the same server-side session, pass the same session_id (a UUID) and the same user_id in the connection string. The server's session cache keys on both:

url = "sc://connect-host:15002/;user_id=etl;session_id=550e8400-e29b-41d4-a716-446655440000"
spark = SparkSession.builder.remote(url).getOrCreate()

A shared session has a shared lifetime. If any client calls spark.stop(), further requests from the others fail too.

How Databricks approaches session lifetime

Databricks Connect is built on open-source Spark Connect, so the same errors apply. Databricks documents its own session policies on top:

• Serverless expiry. The Databricks Connect 15.3.1 for Python release notes (July 18, 2024) say serverless sessions expired after 10 minutes of inactivity and that the client closes expired sessions, so DatabricksSession.builder.getOrCreate() returns a new one. The 17.3.0 release notes for Python (October 28, 2025) state: "Serverless sessions no longer expire after 10 minutes of inactivity."
• Long-lived sessions. With Databricks Connect 16.4 and above on serverless compute, Databricks makes a best effort to preserve a session after the default idle timeout and restore its configurations, temporary views, UDFs, temporary variables and uploaded files when you reconnect. Preserved state expires after two days of inactivity, recovery isn't guaranteed, and streaming query state isn't preserved. See long-lived sessions.
• Classic compute. Standard and dedicated compute don't support long-lived sessions, so keep the reconnect-and-replay pattern there.

FAQs

Can I make a Spark Connect session last longer?

The Spark Connect server settings in the Apache Spark configuration docs don't include a session lifetime setting. Build the client to reconnect and replay setup, which also covers driver restarts.

Does a network blip end my session?

Not by itself. The server finds your session by its ID rather than by the connection, and the protocol lets a client reattach to a running query's result stream with ReattachExecute. If the client never comes back, the server removes the execution as abandoned (OPERATION_ABANDONED).

Why do my temporary views disappear after reconnecting?

They belonged to the old session. Temporary views, SQL configurations and registered functions are session state, so a new session starts without them until you recreate them.