Thin client versus fat client: the trade-offs?

Summary

• The Apache Spark docs call the Spark Connect client "a thin API" that sends unresolved logical plans to a remote Spark driver over gRPC. The "fat" side, a classic Spark application, is coupled to the driver.
• Thin client gains: the client runs in its own process, the driver can be upgraded independently of applications, and you can debug from your IDE.
• Thin client costs: no SparkContext or RDD API, deferred analysis (schema access triggers an RPC), and no built-in authentication.
• Pick the thin client for DataFrame and SQL applications that embed Spark or run remotely. Keep a classic application when code depends on RDDs, SparkContext or driver JVM internals.

Thin client versus fat client: the trade-offs?

The Apache Spark docs do not use the terms "thin client" or "fat client". They describe the Spark Connect client library as "a thin API that can be embedded everywhere" and contrast it with "classic Spark applications". This page uses "thin client" for the first and "fat client" for the second, where the application and the driver are coupled. The short answer: the thin client trades API coverage and eager analysis for isolation, independent upgrades and easier embedding.

How do the two architectures work?

With Spark Connect, the client "translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers" and sends them to the server over gRPC. The server runs the standard Spark execution, and "results are streamed back to the client through gRPC as Apache Arrow-encoded row batches". The client "does not run in the same process as the Spark driver", so it "cannot directly access and interact with the driver JVM".
A classic PySpark program, in the docs' words, "communicates with the driver on JVM by using Py4J", which is what makes private fields such as df._jdf reachable. The Spark Connect client does not use Py4J.

What are the trade-offs?

Trade-off	Thin client (Spark Connect)	Fat client (classic application)
Process model and isolation	Own process; an application that uses too much memory "will now only impact their own environment"	Application and driver are coupled
Versions and dependencies	The driver, including server-side dependencies, can be updated without updating the client	Client and driver must run identical software versions
API coverage	Most DataFrame, function, Column and SQL APIs; no SparkContext or RDD	Full API, including RDD and SparkContext
Analysis timing	Lazy; schema access triggers an analysis RPC and the schema is cached	Eager; schema access is local
Temporary views and UDFs	A DataFrame references a view by name; UDFs are serialized at execution	The view's plan is embedded; UDFs are serialized at creation

When does the thin client fit?

• Embedding: the docs say Spark Connect "can be embedded in modern data applications, in IDEs, Notebooks and programming languages".
• Multi-tenant clusters: the docs say it "mitigates several multi-tenant operational issues". Users define their own client dependencies without conflicts with the driver, the driver can be upgraded independently of applications, and you can debug interactively from your IDE.
• Other languages: because the client and driver are decoupled, a Spark Connect client can be written in any language. The Spark Connect page shows a Go client that needs neither Java nor Spark installed.

When does the fat client fit?

Use a classic application when the code needs what the Spark Connect protocol does not carry. The docs name RDDs as the most important gap, and add that the client has no access to the static Spark configuration or the SparkContext. The Spark Connect page adds that codebases using the RDD API or private Spark JVM functions are harder to migrate.

What does the thin client cost?

• Remote analysis. Spark Connect defers analysis and name resolution to execution time. Calling df.columns or df.schema triggers an analysis request to the server, and doing it on many new DataFrames "will result in a large number of analysis requests". The docs suggest tracking column names in a set instead.
• Session state by name. Replacing a temporary view changes DataFrames that already reference it. The docs recommend unique view names, for example with a UUID.
• Security. Spark Connect "does not have built-in authentication". Its gRPC HTTP/2 interface lets you secure it with an authenticating proxy.
• Scala user code. UDFs, filter and map need a ClassFinder, and JAR dependencies must be uploaded with SparkSession#AddArtifact.

How do I switch between them?

from pyspark.sql import SparkSession

# Thin client: a Spark Connect session
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

# Fat client: a classic session
spark = SparkSession.builder.appName("MyApp").getOrCreate()

Setting the SPARK_REMOTE environment variable, for example export SPARK_REMOTE="sc://localhost", makes a new session a Spark Connect session with no code change. By default, the REPL attempts to connect to a local Spark server on port 15002.

What about Databricks?

Databricks Connect is "a client library for the Databricks Runtime", built on open-source Spark Connect, that connects IDEs, notebooks and custom applications to Databricks compute. Databricks lists Spark Connect as the architecture for Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on 14.3 and above (both on standard compute), serverless compute, Databricks Connect and Lakeflow pipelines with environment versions configured.

FAQs

Does Spark Connect support Structured Streaming?

Mostly. The Scala section of the docs says the "majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener".

Will my PySpark code run unchanged?

It depends on the APIs it uses. The API reference labels supported APIs "Supports Spark Connect", so you can check before migrating. Code that uses RDDs, SparkContext or df._jdf will not run on a Spark Connect client.