PySpark versus pandas: when should I use each?

Summary

• Use pandas when the data fits comfortably in one machine's memory and you want the full pandas API. The pandas scaling guide warns that data larger than memory is "somewhat tricky" for pandas, and that even a sizable fraction of memory can become unwieldy.
• Use PySpark, the Python API for Apache Spark, when the data or the computation outgrows one machine, or when one program needs distributed file reads and writes, Spark SQL, Structured Streaming or MLlib.
• pandas API on Spark (import pyspark.pandas as ps) runs pandas-style code on Spark, but it doesn't aim for full compatibility with pandas, so test a port before you commit to it.
• toPandas() loads the entire result into the driver's memory. Filter and aggregate in Spark first, then convert only the small result.
• You can use both in one pipeline: Spark for the large reads, joins and aggregations, and pandas for the reduced output or for per-group logic inside applyInPandas.

PySpark versus pandas: when should I use each?

Use pandas when your data fits comfortably in one machine's memory and you want pandas' full API. Use PySpark when the data, the computation or the job's reliability needs outgrow a single machine. You don't have to pick one for good: Spark can cut a large dataset down to size, and pandas can take over once the result fits in memory.
Getting it wrong is costly in both directions. pandas on data that doesn't fit ends in memory errors, and PySpark on a small file means running a distributed engine where one Python process would do.

How pandas and PySpark differ

pandas "provides data structures for in-memory analytics," in the words of its own scaling guide. PySpark lets you "perform real-time, large-scale data processing in a distributed environment using Python," and it covers Spark SQL, DataFrames, Structured Streaming and MLlib. Spark is also lazy: transformations such as filter only build a plan, and nothing runs until an action such as show() or collect().

Aspect	pandas	PySpark
Where data lives	In the memory of one Python process	Partitioned across a cluster's executors
Execution	Operates directly on in-memory data	Builds a lazy plan, optimizes it, runs it on an action
Size limit	One machine's memory, with room for intermediate copies	Grows with the cluster
Index	Central to every DataFrame	None in Spark DataFrames; pandas API on Spark attaches a default index
APIs	The full pandas API, plus libraries that expect pandas objects	DataFrame API, Spark SQL, Structured Streaming, MLlib, pandas API on Spark
Best fit	Exploration, small and medium files, final reporting	Large tables, joins, aggregations, scheduled pipelines, streams

When pandas is the right tool

• The data fits with headroom. The pandas guide notes that "some pandas operations need to make intermediate copies," so a dataset near your memory limit is already a problem.
• You need pandas-specific behavior, such as label-based indexing, or a library that only accepts pandas objects.
• You're analyzing a sample, or an aggregate that Spark already produced.

When PySpark is the right tool

• The input is larger than one machine's memory, or it keeps growing.
• You read many files from object storage, join large tables or write partitioned output.
• You want SQL, streaming and machine learning against the same data in one program.
• The job runs unattended on a schedule and has to survive the loss of a worker node.

Moving data between them

Convert at the boundary, after Spark has done the heavy work:

events = spark.read.parquet("/data/events")                  # distributed read
daily_pdf = events.groupBy("event_date").count().toPandas()  # small result to pandas

psdf = events.pandas_api()                                    # pandas syntax, runs on Spark
top_types = psdf["event_type"].value_counts().head(5)

For per-group pandas logic, applyInPandas passes each group to your function as a pandas DataFrame:

import pandas as pd

def demean(pdf: pd.DataFrame) -> pd.DataFrame:
    return pdf.assign(amount=pdf.amount - pdf.amount.mean())

result = (events.select("account_id", "amount")
          .groupBy("account_id")
          .applyInPandas(demean, schema="account_id long, amount double"))

The Arrow guide warns that "all data for a group will be loaded into memory before the function is applied," so a skewed group can still cause out-of-memory errors.

How Databricks approaches pandas and PySpark

Databricks Runtime includes pandas as a standard Python package, so pandas DataFrames work in notebooks and jobs without setup, and pandas API on Spark is available with import pyspark.pandas as ps. Databricks Connect, which is built on Spark Connect, makes the boundary explicit: DataFrame operations run on Databricks compute, and results are materialized on your local client when you call methods such as collect(), show() or toPandas(). The rule holds everywhere: reduce in Spark, convert at the end.

FAQs

Is pandas API on Spark a drop-in replacement for pandas?

Not entirely. Its docs say it doesn't target full compatibility with pandas or PySpark, and some APIs are left out on purpose: it doesn't implement __iter__(), which stops code from collecting a whole dataset onto the driver. Check the supported API list before porting.

What happens if I call toPandas() on a large DataFrame?

Every row is collected into the driver's memory. The API reference says to use it only if the resulting pandas DataFrame "is expected to be small," so aggregate, filter or limit() first.

What does Apache Arrow do in these conversions?

Arrow is the in-memory columnar format Spark uses to move data between the JVM and Python. The setting spark.sql.execution.arrow.pyspark.enabled applies it to toPandas() and to createDataFrame() from pandas, and the current docs say it's enabled by default.

Can pandas code run inside a Spark job?

Yes. pandas UDFs and pandas function APIs such as applyInPandas and mapInPandas run your pandas code on each batch or group, with Arrow handling the transfer.