Spark Connect versus the Thrift server?

Summary

• Spark Connect, introduced in Apache Spark 3.4, sends unresolved logical plans from a thin client to the Spark driver over gRPC and streams results back as Apache Arrow-encoded row batches. It supports PySpark and Scala applications.
• The Thrift JDBC/ODBC server corresponds to HiveServer2 in built-in Hive: JDBC and ODBC clients such as Beeline send it SQL, by default at localhost:10000.
• Pick Spark Connect when the client is code that needs the DataFrame API remotely; pick the Thrift server when the client is a tool that speaks JDBC or ODBC and SQL is enough.
• Neither is stateless. Spark Connect provides a session-based client, and Thrift server connections don't share temporary views or SQL configuration unless spark.sql.hive.thriftServer.singleSession is true.

Spark Connect versus the Thrift server?

Use Spark Connect when the client is code. It's a client-server protocol for the DataFrame API, so a Python or Scala application can run Spark remotely without living inside the driver process. Use the Thrift JDBC/ODBC server when the client is a tool that speaks JDBC or ODBC and only needs SQL. Both put a network endpoint in front of a Spark cluster; the difference is what crosses the wire: logical plans built by a Spark client library, or SQL statements from JDBC and ODBC clients.

How do they compare?

Aspect	Spark Connect	Thrift JDBC/ODBC server
Origin	Introduced in Apache Spark 3.4	Corresponds to HiveServer2 in built-in Hive
What clients send	Unresolved logical plans from the DataFrame API (and spark.sql)	SQL statements
Wire protocol	Protocol buffers over gRPC; results as Arrow row batches	Thrift RPC, optionally over HTTP transport
Clients	PySpark and Scala; Apache also hosts an experimental Go client and a Swift subproject	JDBC and ODBC clients such as Beeline
Start the server	./sbin/start-connect-server.sh	./sbin/start-thriftserver.sh
Default endpoint	Port 15002 (sc://localhost)	localhost:10000; in HTTP mode, port 10001 and endpoint cliservice
Sessions	Session-based client; the server accepts sessions from client applications	Per connection, unless spark.sql.hive.thriftServer.singleSession is true
Not available	RDDs and SparkContext	DataFrame and RDD APIs
Authentication	None built in; put an authenticating proxy in front of gRPC	Non-secure mode, or secure mode as the Beeline docs describe

How does Spark Connect work?

The client translates DataFrame operations into unresolved logical plans encoded as protocol buffers and sends them to the server over gRPC. The server turns them into Spark's logical plan operators, the standard Spark execution process runs, and results stream back as Apache Arrow-encoded row batches. Because the client never runs in the driver process, PySpark doesn't use Py4J (so df._jdf isn't available), and the client can't reach the static Spark configuration or the SparkContext.

./sbin/start-connect-server.sh
./bin/pyspark --remote "sc://localhost"

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
df = spark.read.text("README.md")
print(df.filter(df.value.contains("a")).count())

Standalone Python applications install pyspark-client (for example pip install pyspark-client==4.2.0), and Scala projects add spark-connect-client-jvm. Setting SPARK_REMOTE="sc://localhost" turns unchanged PySpark code into a Spark Connect session. Since Spark 4.0, spark.api.mode (default classic) can be set to connect so a classic application runs on a local Spark Connect server. The operational gains, per the Spark docs: an application that uses too much memory only affects its own process, client dependencies don't conflict with the driver, the driver can be upgraded independently of applications, and you can debug from an IDE.

How does the Thrift server work?

It lets end users and applications run SQL on Spark "without the need to write any code". The start script accepts all spark-submit options plus --hiveconf for Hive properties, and reads Hive configuration from hive-site.xml, core-site.xml and hdfs-site.xml in conf/.

./sbin/start-thriftserver.sh --master <master-uri>
./bin/beeline
beeline> !connect jdbc:hive2://localhost:10000

For HTTP transport, set hive.server2.transport.mode to http. The Spark SQL CLI (./bin/spark-sql) is a separate tool and can't talk to the Thrift JDBC server.

Which one should I choose?

Choose Spark Connect when:

• the client is a Python or Scala application, notebook or IDE session that needs DataFrame transformations, UDFs or Structured Streaming remotely (the majority of the Streaming API is supported);
• you want client failures and dependency conflicts kept away from the driver;
• you can secure the gRPC endpoint with an authenticating proxy.

Choose the Thrift server when:

• the clients are SQL editors, BI tools or other JDBC and ODBC software;
• SQL is all anyone needs;
• you already manage Hive-style configuration and Beeline.

Spark 4.1.0's release notes also list a "JDBC Driver for Spark Connect" (SPARK-53484), an upstream direction worth tracking if you want JDBC clients on Spark Connect.

How Databricks approaches remote Spark access

Databricks Connect, for Databricks Runtime 13.3 LTS and above, is built on open-source Spark Connect and extends it to support Databricks compute modes and Unity Catalog. It's available for Python, R and Scala: your local code runs in the IDE, while DataFrame operations and UDFs run on Databricks compute. Serverless compute uses Spark Connect too, so only Spark Connect APIs work there and RDD APIs don't. For JDBC and ODBC clients, Databricks provides its own JDBC and ODBC drivers, which connect to a cluster or SQL warehouse with the server hostname, port and HTTP path from its connection details.

FAQs

Can Spark Connect run SQL?

Yes. SparkSession.sql has supported Spark Connect since Spark 3.4.0, so spark.sql("SELECT ...") works from a Spark Connect client and returns a DataFrame.

Is the Thrift server stateless?

No. By default (spark.sql.hive.thriftServer.singleSession is false), connections don't share temporary views, function registries, SQL configuration or the current database. Setting it to true makes every JDBC/ODBC connection share them.

Does Spark Connect have authentication?

Not built in. Its gRPC HTTP/2 interface works with authenticating proxies, and a connection string can carry a token, as in sc://myhost.com:443/;token=ABCDEFG.