How do I fix a client/server version mismatch in Spark Connect?

Summary

• Fix a Spark Connect version mismatch by installing the client at the server's Spark release: pyspark-client (or pyspark[connect]) for Python and spark-connect-client-jvm for Scala, pinned to the same version.
• Apache Spark guarantees that older Spark Connect clients work with newer servers. A client newer than its server is outside that guarantee and can send plan messages the server can't decode.
• Check both sides before you change anything: the client package version in your environment, and spark.version on the session, which Spark Connect answers from the server.
• Python UDFs add a second rule: the client and the server's Python workers need the same Python minor version, or the worker raises PYTHON_VERSION_MISMATCH.
• On Databricks, Databricks Connect versions follow Databricks Runtime versions, and the runtime must be the same as or newer than the package.

How do I fix a client/server version mismatch in Spark Connect?

Make the client match the server. Find the server's Spark version, then install the Spark Connect client at exactly that release, for example pyspark-client==4.2.0. If you can't match exactly, keep the client older than the server, never newer. Apache Spark guarantees that older clients are compatible with newer Spark Connect servers, and a newer client against an older server is outside that guarantee.

Why the versions have to line up

Spark Connect splits an application in two. The client translates DataFrame operations into unresolved logical plans, encodes them with protocol buffers and sends them to the server over gRPC. The server turns them into Spark logical plans, runs them and streams results back as Apache Arrow batches. New releases add plan messages, expressions and error conditions, so a client that's newer than its server can send a plan node the server doesn't know. The Spark Connect overview describes the same contract: applications can be forward-compatible as long as the server-side RPC definitions are designed to be backwards compatible.
The guarantee runs one way. Spark's issue tracker states it directly in SPARK-51817: "In Spark Connect, we guarantee that older clients are compatible with newer versions of the Spark Connect service." That ticket fixed a break in error handling for older clients against newer servers, and the fix shipped in Spark 4.0.0.

Symptoms, causes and fixes

Symptom	Likely cause	Fix
CONNECT_INVALID_PLAN.INVALID_ONE_OF_FIELD_NOT_SET from a 4.2 server (older servers word it differently)	The client is newer than the server and sent a plan element the server can't decode	Install the client at the server's version, or upgrade the server
UNSUPPORTED_PACKAGE_VERSION when the client starts	pandas, PyArrow, grpcio or another Connect dependency is below the client's minimum	Reinstall with pip install "pyspark[connect]==<server version>" or upgrade the named package
PYTHON_VERSION_MISMATCH when a UDF runs	Your local Python minor version differs from the server workers' Python	Use the same Python minor version on both sides; check PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
spark.version and your installed client report different releases	Client and server were upgraded separately	Pin the client in your requirements and upgrade both together

Find both versions, then pin the client

Start by reading the two versions from the same environment your application runs in:

from importlib.metadata import version
from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://spark-host:15002").getOrCreate()
print("client:", version("pyspark-client"))  # or "pyspark" for a full install
print("server:", spark.version)              # answered by the Spark Connect server

Port 15002 is the default for spark.connect.grpc.binding.port. Then pin the client to the server's release and keep the pin in your build file so the two can't drift apart:

pip install "pyspark-client==4.2.0"

libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "4.2.0"

On the server side, start Spark Connect from the same Spark distribution you're matching (./sbin/start-connect-server.sh). The quickstart pairs Spark 4.2.0 with Scala 2.13, and the sbt %% operator picks the Scala binary suffix for you. For Python UDFs, run the same Python minor version locally and on the server: PySpark records the client's Python version with each UDF, and the server's worker refuses to run it when the versions differ.

How Databricks approaches version matching

Databricks Connect is built on open-source Spark Connect, and for Databricks Runtime 13.3 LTS and above it extends Spark Connect for Databricks compute and Unity Catalog. Its version numbers correspond to Databricks Runtime versions, and its rule matches the open-source guarantee: the runtime version of your compute must be greater than or equal to the Databricks Connect package version. Databricks recommends the most recent package that matches your runtime, installed with pip install --upgrade "databricks-connect==17.3.*" or the X.Y.* for your cluster. Uninstall pyspark first, because the two packages conflict and having both can surface as "stream corrupted" or "class not found" errors. For serverless compute, Databricks Connect versions lower than or equal to the serverless Databricks Runtime release are fully compatible. Databricks Connect doesn't support connecting to open-source Apache Spark servers.

FAQs

Can a newer PySpark client connect to an older Spark Connect server?

It may connect, but nothing guarantees the calls will work. The compatibility guarantee covers older clients against newer servers, so pin the client at or below the server's release.

Which Python package should I install for Spark Connect?

pyspark-client is a pure Python client that only supports spark.remote connection URIs such as sc://localhost. pyspark[connect] installs full PySpark with the Connect dependencies, and pyspark-connect makes Spark Connect the default. Pin whichever you choose to the server's version.

How do I check the server's Spark version from the client?

Read spark.version on a Spark Connect session. PySpark asks the server for it, so the value reflects the server, not your local package.