Spark Connect versus Apache Livy?

Summary

• Spark Connect and Apache Livy both let a remote client use a Spark cluster, but they accept different things. Spark Connect takes DataFrame plans built by a client library, sent over gRPC. Livy takes code snippets and jars over a REST interface.
• Spark Connect is part of Apache Spark (since 3.4). Livy is a separate Apache project with its own releases: 0.9.0-incubating shipped on 2026-02-11, and the Apache Incubator lists its graduation on 2026-05-20.
• Spark Connect does not support RDDs or SparkContext. Livy sessions run Scala, Python, R or SQL statements, and the Livy examples use sc.parallelize.
• Spark Connect has no built-in authentication. Livy's configuration template documents built-in SPNEGO authentication for HTTP requests.
• Confirm that your Livy release supports your Spark version. Livy's README says it requires Spark 3.0+, and the 0.9.0-incubating notes list Spark 3.5.6 support.

Spark Connect versus Apache Livy?

Spark Connect versus Apache Livy?

Spark's cluster docs advise that when you send requests to a cluster remotely, "it's better to open an RPC to the driver and have it submit operations from nearby than to run a driver far away from the worker nodes." Spark Connect and Apache Livy both address that situation, in different ways. Spark Connect embeds an endpoint on the Spark server and speaks gRPC with protocol buffers. Livy is a separate service that exposes a REST interface for submitting Spark jobs or snippets of Spark code.

How do they compare?

Aspect	Spark Connect	Apache Livy
Project	Part of Apache Spark since 3.4	Separate Apache project with its own releases (0.9.0-incubating shipped 2026-02-11)
Interface	gRPC with protocol buffers, Arrow-encoded results	REST interface, or an RPC client library
What the client sends	Unresolved logical plans built by a client library	Code snippets (Scala, Python, R, SQL) or precompiled jars
Client requirement	A client library such as pyspark-client	"no Spark client needed"; calls go over REST
RDDs and SparkContext	Not supported	Available in sessions (the examples use sc.parallelize)
Authentication	None built in; use authenticating proxies	Built-in SPNEGO for HTTP requests (livy.server.auth.type = kerberos)
Default port	15002	8998
Where Spark runs	On the Spark server that hosts the endpoint	Spark contexts run on the cluster, not in the Livy server

What does each look like in code?

Spark Connect, from the Spark docs:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate()
logData = spark.read.text("YOUR_SPARK_HOME/README.md").cache()
numAs = logData.filter(logData.value.contains('a')).count()

Apache Livy, from the Livy examples page:

import json, requests

host = 'http://localhost:8998'
headers = {'Content-Type': 'application/json'}

r = requests.post(host + '/sessions', data=json.dumps({'kind': 'spark'}), headers=headers)
session_url = host + r.headers['location']
statements_url = session_url + '/statements'
r = requests.post(statements_url, data=json.dumps({'code': '1 + 1'}), headers=headers)

If a Livy statement takes longer than a few milliseconds, Livy returns early with a statement URL that you poll until the state is available.

When does each one fit?

Spark Connect fits when:

• Your clients can embed a library (PySpark or the Scala client) and use the DataFrame API.
• You want to upgrade the Spark driver independently of applications, or step through code with your IDE's debugger.
• Your code does not depend on RDDs, SparkContext or private JVM methods.

Livy fits when:

• Clients such as web or mobile apps need to submit jobs without a Spark client.
• You submit code snippets or precompiled jars, interactively or as batches (/sessions and /batches).
• Your workloads use RDDs, or share cached RDDs and DataFrames across jobs and clients.
• You want built-in Kerberos (SPNEGO) authentication and user impersonation.

Is Livy compatible with my Spark version?

Check before you commit. Livy's README says it requires Spark 3.0+ and that its default build targets Apache Spark 4.1.2 with Scala 2.13. The 0.9.0-incubating release notes list "Add support for Spark 3.5.6" (LIVY-1010), Java 17 support (LIVY-1017) and Kubernetes submission (LIVY-702). Livy is versioned separately from Spark, so confirm the Livy release you deploy against the Spark version you run. The Apache Incubator page records Livy's graduation to a top-level project on 2026-05-20, and the Livy site still lists 0.9.0-incubating as the current release.

How does Databricks fit in?

Databricks Connect is built on open-source Spark Connect and can be embedded in applications, much like a JDBC driver. For REST access, Databricks documents the Jobs API (POST /api/2.2/jobs/runs/submit submits a one-time run without creating a job) and the Statement Execution API for running SQL statements on SQL warehouses.

FAQs

Is Spark Connect a replacement for Livy?

Neither project's documentation presents it that way. Spark Connect sends DataFrame plans from a client library, and Livy runs submitted code and jars in sessions it manages. Choose by the interface your clients need.

Which one supports RDD code?

Livy's own examples run RDD code with sc.parallelize inside a session. Spark Connect does not support RDDs, and the Spark docs point to DataFrame alternatives.

How do the security models differ?

The Spark docs say Spark Connect has no built-in authentication and works with authenticating proxies through its gRPC HTTP/2 interface. Livy's configuration template says it has built-in SPNEGO authentication for HTTP requests, and its README lists impersonation support.

Where do the sessions run?

A Livy session starts a Spark context on the cluster. Livy's get-started guide recommends YARN cluster mode so the Livy server host does not become overloaded. Spark Connect sessions are served by the endpoint on the Spark server.