Is Python the default API in Spark 4?

Summary

• No. The Spark 4.2 docs say Spark "provides high-level APIs in Java, Scala, Python and R", with R deprecated, and name no default among them. You can run SQL from within each language.
• Spark SQL says "the same execution engine is used, independent of which API/language you are using to express the computation".
• Spark 4 releases keep adding Python-specific features: pyspark-client, a plotting API and a Python Data Source API in 4.0, Arrow UDFs in 4.1, and Arrow-optimized Python UDFs on by default in 4.2.
• What did change by default in Spark 4: Scala 2.13, JDK 17 and ANSI SQL mode. spark.api.mode still defaults to classic.
• The typed Dataset API is for Scala and Java only, because "Python does not have the support for the Dataset API".

Is Python the default API in Spark 4?

No. The Spark 4.2 documentation says Spark "provides high-level APIs in Java, Scala, Python and R", marks R as deprecated, and names no default among them. Python is prominent in the docs and release notes: the Spark 4.2 Quick Start starts its shell with ./bin/pyspark and says "for now let's proceed with Python", and the 4.1 and 4.2 release notes both name PySpark among their focus areas. That is emphasis and engineering effort, not a default.

Which language APIs does Spark 4 offer?

API	What the Spark 4.2 docs say
Python (PySpark)	The DataFrame API is available in Python. Spark 4.2 needs Python 3.10 or above. The pandas API on Spark covers "pandas workloads"
Scala	The DataFrame API is a type alias of Dataset[Row]. Spark 4 is built for Scala 2.13
Java	A DataFrame is Dataset<Row>. Spark 4.2 runs on Java 17, 21 and 25
R	"SparkR (Deprecated)", deprecated in the 4.0.0 notes (SPARK-49347)
SQL	Run queries with spark.sql, the command line or JDBC/ODBC

The typed Dataset API "is available in Scala and Java. Python does not have the support for the Dataset API."

What did Spark 4 change in its defaults and requirements?

Area	Spark 4 behavior
Scala	Scala 2.13 is the default and Scala 2.12 was dropped (4.0.0)
JDK	JDK 17 is the default, and JDK 8 and 11 were dropped (4.0.0)
SQL mode	ANSI SQL mode is on by default (4.0.0)
Python	Python 3.8 support was dropped (4.0.0)
Python UDFs	Arrow-optimized Python UDFs and Arrow-based PySpark IPC are enabled by default (4.2.0)
API mode	spark.api.mode defaults to classic (since 4.0.0), so Spark Connect is opt-in

What Python-specific work shipped in Spark 4?

Release	PySpark items in the release notes
4.0.0	A lightweight pyspark-client, "a native plotting API, a new Python Data Source API, support for Python UDTFs, and unified profiling for PySpark UDFs"
4.1.0	Python Arrow UDF and Python Arrow UDTF (SPARK-52214, SPARK-52979), plus Python Data Source filter pushdown
4.2.0	Arrow-optimized Python UDFs and Arrow-based PySpark IPC enabled by default (SPARK-54555), and Real-time Mode trigger support in PySpark (SPARK-54660)

Does the language you pick change the engine?

For DataFrame and SQL code, no. The docs say "the same execution engine is used, independent of which API/language you are using to express the computation". Python code reaches that engine differently. Classic PySpark calls the JVM through Py4J, and the 4.2.0 notes list "Optimize Py4J calls in df.toPandas, createDataFrame, DataFrame, toArrow, and schema inference". The Spark Connect client "does not use Py4J". Data moves between the JVM and Python processes, and "Apache Arrow is an in-memory columnar data format that is used in Spark to efficiently transfer data between JVM and Python processes." The docs call RDDs the "core but old API", and in Python, persisted RDD objects "will always be serialized with the Pickle library".

How do I pick a language?

If you need	Use	Why (from the docs)
A typed Dataset API	Scala or Java	Python has no Dataset API
pandas-style code on Spark	Python	The pandas API on Spark
A remote client	Python or Scala	Spark Connect supports most PySpark APIs since 3.4 and most Scala APIs since 3.5
Plain SQL	SQL	spark.sql, the command line or JDBC/ODBC

How does Databricks fit?

Databricks notebooks support "developing code in Python, SQL, Scala, and R". The default language appears below the notebook name, and a cell can override it with %python, %r, %scala or %sql. Variables defined in one language "are not available in the REPL of another language". The Photon docs list: "Supports SQL and DataFrame APIs across Python, R, Scala, and Java with no code changes required." Serverless and standard compute list "R is not supported", and Databricks Connect is available for Python, R and Scala.

FAQs

Is SparkR still supported?

It is deprecated: the 4.2 docs label SparkR "(Deprecated)", and the 4.0.0 notes list "Deprecate SparkR".

Which Python versions does Spark 4.2 support?

Python 3.10 and above.

Can I use SQL from Python?

Yes. "When running SQL from within another programming language the results will be returned as a Dataset/DataFrame."

Can I make Spark Connect the default for Python?

Install pyspark-connect: "If you want to make Spark Connect the default, you can install an additional library via PyPI (pyspark-connect)." For a pure Python client with no jars or JRE, install pyspark-client, which supports only spark.remote connection URIs.