How do I register a Python UDF versus a pandas UDF?

Summary

• Register a Python UDF with udf and a pandas UDF with pandas_udf, both from pyspark.sql.functions.
• A Python UDF runs your function once per row on regular Python values. A pandas UDF runs it once per batch on pandas.Series.
• From Spark 4.2, both kinds move data between the JVM and Python with Arrow by default. Earlier releases pickle data for regular Python UDFs unless you opt in with useArrow=True.
• To call either kind from SQL, pass the UDF object to spark.udf.register(name, udf) and do not repeat returnType.
• Inside a Series-to-Series pandas UDF, s.mean() is the mean of one internal batch, not of the whole column.

How do I register a Python UDF versus a pandas UDF?
In PySpark, you register a Python UDF with udf and a pandas UDF with pandas_udf, both imported from pyspark.sql.functions. A regular Python UDF calls your function once per row. A pandas UDF calls it once per batch of rows and passes pandas.Series. Either kind can also be exposed to SQL with spark.udf.register.

How do Python UDFs and pandas UDFs differ?

Aspect	Python UDF (udf)	pandas UDF (pandas_udf)
Function input	Regular Python values for one row	pandas.Series for a batch (pandas.DataFrame for struct columns)
Function output	One value per row	A pandas.Series of the same length as the input
JVM to Python transfer	Arrow by default since Spark 4.2, pickle by default before	Arrow
Return type	returnType as a DataType or DDL string, default StringType	returnType as a DataType or DDL string

How do I register a Python UDF?

Apply udf as a decorator and pass returnType. This example comes from the PySpark udf documentation:

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

@udf(returnType=IntegerType())
def add_one(x):
    if x is not None:
        return x + 1

df = spark.createDataFrame([(1, "John Doe", 21)], ("id", "name", "age"))
df.select(add_one("age")).show()

If you leave out returnType, Spark uses StringType. From Spark 4.1, udf also builds a vectorized UDF when the function has at least one argument, its type hints match a pandas UDF pattern and useArrow is not set. The docs note that UDFs do not support conditional expressions or short circuiting in boolean expressions, so Spark may run them on rows you expected a guard to skip. Put checks such as x is not None inside the function.
Since Spark 4.2, regular Python UDFs use Arrow for serialization but still process one row at a time. To return to pickle, pass useArrow=False to udf or set spark.sql.execution.pythonUDF.arrow.enabled to false.

How do I register a pandas UDF?

Apply pandas_udf with a return type and annotate the function with pandas.Series type hints:

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("string")
def to_upper(s: pd.Series) -> pd.Series:
    return s.str.upper()

df = spark.createDataFrame([("John Doe",)], ("name",))
df.select(to_upper("name")).show()

The type hints decide which pandas UDF variant Spark builds. The functionType argument and PandasUDFType remain for compatibility, and the docs prefer type hints.

Type hint	Variant	What to know
pd.Series -> pd.Series	Series to Series	The output must have the same length as the input
Iterator[pd.Series] -> Iterator[pd.Series]	Iterator of Series to Iterator of Series	Useful when the function must initialize some state first
pd.Series -> float	Series to Scalar	Works with groupby().agg() and windows, has no partial aggregation, and loads all data for a group or window into memory

For a grouped aggregate, use the Series-to-scalar form:

@pandas_udf("double")
def mean_udf(v: pd.Series) -> float:
    return v.mean()

df = spark.createDataFrame(
    [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id", "v"))
df.groupby("id").agg(mean_udf(df["v"])).show()

What does a pandas UDF receive?

PySpark splits columns into batches, calls the function once per batch, and concatenates the results. The input length is the length of an internal batch, not of the whole column, so s.mean() inside a Series-to-Series UDF is a per-batch mean. The batch row cap is spark.sql.execution.arrow.maxRecordsPerBatch, which defaults to 10,000 records.

How do I call a UDF from SQL?

Pass the UDF object to spark.udf.register. Its f argument accepts a Python function, a udf() result or a pandas_udf() result. Set returnType only when f is a plain Python function.

spark.udf.register("py_add_one", add_one)

@pandas_udf("integer")
def pd_add_one(s: pd.Series) -> pd.Series:
    return s + 1

spark.udf.register("pd_add_one", pd_add_one)
spark.sql("SELECT py_add_one(id), pd_add_one(id) FROM range(3)").show()

What mistakes should I avoid?

Mistake	What happens
Registering a plain lambda without returnType	The SQL function returns strings, because StringType is the default
Passing returnType together with a UDF object	The docs say it should not be specified; the UDF's own return type is used
Returning a different length from a Series-to-Series UDF	The output must always match the input length
Returning a dtype that differs from returnType	Spark might convert it, and the docs say the conversion is not guaranteed to be correct
Returning ArrayType of TimestampType or a nested StructType	Not supported as output types for pandas UDFs
Registering a nondeterministic function directly	Build the UDF first, call asNondeterministic(), then register it

FAQs

Which pandas and PyArrow versions do I need?

For Spark SQL, the PySpark install guide lists pandas >=2.2.0,<3.0.0 and PyArrow >=18.0.0. pip install pyspark[sql] brings PyArrow in as an extra dependency. Otherwise PyArrow must be installed on all cluster nodes.

Can I use NumPy inside a pandas UDF?

Yes. The pandas_udf documentation includes a Series-to-scalar example that calls np.average(v, weights=w).

Can a pandas UDF return a struct?

Yes. Annotate the output as pd.DataFrame and give a DDL schema such as "col1 string, col2 long". Nested StructType outputs are not supported.