Spark versus Ray for distributed Python?

Summary

• Apache Spark and Ray parallelize Python in different shapes. Spark distributes DataFrame and SQL operations over partitioned data, and Ray distributes your own Python functions (tasks) and classes (actors).
• Spark's model is declarative: you describe transformations, and Spark plans, optimizes and runs them lazily across executors, with Spark SQL, Structured Streaming and MLlib in the same engine.
• Ray's model is imperative: you mark functions or classes with @ray.remote, call them with .remote(), and collect results with ray.get(). Its libraries cover data processing for ML (Ray Data), training (Ray Train), tuning (Ray Tune) and serving (Ray Serve).
• Choose by workload shape: table-shaped ETL, joins and aggregations map directly onto Spark's model, and fine-grained, stateful or mixed CPU and GPU tasks map directly onto Ray's.
• They also run together. Ray's ray.util.spark.setup_ray_cluster starts Ray worker nodes inside a Spark application, and Databricks documents the two as complementary frameworks.

Spark versus Ray for distributed Python?

Pick by the shape of the work. Apache Spark distributes operations on data: you write DataFrame or SQL transformations, and Spark plans them and runs them across executors. Ray distributes Python code itself: functions become tasks, classes become stateful actors, and each declares the CPUs, GPUs or custom resources it needs. Neither covers everything the other does, and the two can run in the same pipeline.

What each framework is

Apache Spark is "a unified analytics engine for large-scale data processing," and PySpark is its Python API, covering Spark SQL, DataFrames, Structured Streaming and MLlib. Transformations are lazy, so Spark sees the whole plan before anything runs.
Ray describes itself as "an open-source unified framework for scaling AI and Python applications like machine learning." Ray Core has three primitives. Tasks are functions that Ray runs "asynchronously on separate worker processes." Actors are stateful workers whose methods can read and change that state. Objects live in a distributed shared-memory object store, one per node. On top sit libraries such as Ray Data, "a scalable data processing library for AI workloads built on Ray," which uses a streaming execution engine.

Decision criteria

Criterion	Apache Spark	Ray
Unit of parallel work	Partitions of a DataFrame, processed by tasks on executors	Tasks (functions) and actors (classes) you define
Programming model	Declarative DataFrame and SQL; lazy plans optimized before they run	Imperative: @ray.remote, .remote(), ray.get()
State	Structured Streaming keeps streaming state in a state store	Actors keep state between method calls
Tabular data	Spark SQL, joins, aggregations, Parquet, Delta Lake and Iceberg tables	Ray Data for loading, preprocessing and batch inference
GPUs and resources	Per-executor and per-task resource settings, such as spark.task.resource.{resourceName}.amount	Tasks and actors request CPUs, GPUs and custom resources
Model training	MLlib; TorchDistributor for distributed PyTorch training	Ray Train and Ray Tune
Model serving	Outside Spark's built-in libraries	Ray Serve
Failure handling	Rebuilds lost data by recomputing it	Retries failed code; recovers from some object store failures

When Spark fits the work

• Inputs are tables or files, and the work is filtering, joining, aggregating and writing them.
• You want SQL and Python against the same data, with an optimizer choosing the plan.
• Batch and streaming should share one API.
• Your custom Python fits a per-batch or per-group pattern, such as pandas UDFs, mapInPandas or applyInPandas.

When Ray fits the work

• The work is many independent Python function calls, or dynamic, compute-intensive tasks such as machine learning or reinforcement learning.
• You need long-lived stateful workers, for example a model loaded once in an actor and called many times.
• One pipeline mixes CPU preprocessing with GPU inference or training.
• You want hyperparameter search or model serving from the same framework, through Ray Tune or Ray Serve.

Running Ray inside a Spark application

Ray's documentation includes a Spark integration. setup_ray_cluster launches a background Spark job in which each Spark task starts one Ray worker node, with the Ray head node on the Spark driver:

import ray
from pyspark.sql import SparkSession
from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster, MAX_NUM_WORKER_NODES

spark = SparkSession.builder.appName("Ray on Spark example").getOrCreate()  # the Spark application that hosts Ray
setup_ray_cluster(max_worker_nodes=MAX_NUM_WORKER_NODES)
ray.init()  # connects to the Ray cluster started above

@ray.remote
def square(x):
    return x * x

print(ray.get([square.remote(i) for i in range(4)]))
shutdown_ray_cluster()

Each Ray worker node gets the CPU, GPU and memory allocated to its Spark task. That supports a split where Spark reads and prepares the data and Ray runs the steps that need tasks or actors.

How Databricks approaches Spark and Ray

Databricks calls Ray and Apache Spark "complementary frameworks." Its docs describe Ray as strong at logical parallelism, such as dynamic, compute-intensive machine learning tasks, and Spark as specialized in data parallelism over large datasets. With Ray 2.3.0 and above, Ray on Databricks creates Ray clusters on Apache Spark clusters, so one workspace can prepare data with Spark and Delta Lake, train or tune with Ray, govern data and models in Unity Catalog, track experiments in MLflow and schedule the whole pipeline with Lakeflow Jobs. The Ray-on-Spark integration supports dedicated access mode, no isolation shared access mode and jobs clusters, and it can't start on serverless-based runtimes.

FAQs

Can I use Spark and Ray in the same job?

Yes. ray.util.spark.setup_ray_cluster starts a Ray cluster inside a running Spark application, and shutdown_ray_cluster ends it. If you don't call it, the Ray cluster stops when the Spark application ends.

Which one should I use for distributed training?

Both have an option. PySpark's TorchDistributor supports distributed training on PyTorch using PySpark and has been available since Spark 3.4.0. Ray offers Ray Train and Ray Tune. Weigh where your data preparation already runs and whether the training code needs Ray's actors.

Do both support GPUs?

Yes. Spark schedules GPUs as resources per executor and per task, and Ray tasks and actors declare GPU requirements that its scheduler uses to place them.