How do I migrate Spark 2 to Spark 3 or 4?

Summary

• Work through the migration guides one version step at a time. The Spark SQL, Core and PySpark guides have one "Upgrading from X to Y" section per step, and most entries name a config that restores the old behavior.
• Spark 3.0 (June 18, 2020) changed calendars, datetime patterns and table-insert coercion. Spark 4.0 (May 23, 2025) turned ANSI mode on by default.
• Platform requirements moved too. Spark 3.0 removed Scala 2.11. Spark 4.0 dropped Scala 2.12 and JDK 8/11, so Scala 2.13 and JDK 17 are the defaults.
• Treat restore configs as a bridge. Spark 4.0 removed the spark.sql.legacy-prefixed datetime rebasing configs.
• On Databricks, choose a Databricks Runtime built on your target Spark line, such as 17.3 LTS (Apache Spark 4.0.0).

How do I migrate Spark 2 to Spark 3 or 4?

Upgrade in stages and read the migration notes for each step. The Spark SQL migration guide has one section per version step, from "Upgrading from Spark SQL 2.4 to 3.0" through "Upgrading from Spark SQL 4.1 to 4.2". Earlier sections cover 2.3 and below. The last 3.x line is 3.5 (3.5.9 was released on July 16, 2026), so testing on 3.5 before 4.x separates 3.x breaks from 4.x breaks. Spark 3.0.0 shipped on June 18, 2020, Spark 4.0.0 on May 23, 2025 and Spark 4.2.0 on July 14, 2026.

What breaks between Spark 2.4 and 3.0?

Area	What changed	Restore or adapt
Calendar	Proleptic Gregorian calendar replaces the hybrid Julian + Gregorian calendar	Review results for dates before October 15, 1582
Datetime patterns	Spark defines its own pattern strings and checks input strictly; 2.4 and below used SimpleDateFormat	spark.sql.legacy.timeParserPolicy = LEGACY
Table inserts	ANSI store assignment disallows conversions such as string to int and raises on out-of-range values	spark.sql.storeAssignmentPolicy = LEGACY
Dataset as[T]	Up casts are stricter, so Seq("str").toDS.as[Boolean] fails during analysis	spark.sql.legacy.doLooseUpcast = true
Hive	Built-in Hive moved from 1.2 to 2.3	Set spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars for older metastores
Core APIs	TaskContext.isRunningLocally, deprecated ShuffleWriteMetrics methods and accumulator v1 APIs were removed	Use bytesWritten, writeTime, recordsWritten and the v2 accumulator APIs
Shuffle fetch	A new protocol fetches shuffle blocks	spark.shuffle.useOldFetchProtocol = true for old external shuffle services

Spark 3.0 also added Java 11 and Hadoop 3 support, made Hive 2.3 the default dependency and removed Scala 2.11.

What breaks between Spark 3.5 and 4.0?

Area	What changed	Restore or adapt
ANSI mode	spark.sql.ansi.enabled is on by default	spark.sql.ansi.enabled = false
CREATE TABLE	Without USING and STORED AS, the provider is spark.sql.sources.default, not Hive	spark.sql.legacy.createHiveTableByDefault = true
ORC	The default spark.sql.orc.compression.codec changed from snappy to zstd	spark.sql.orc.compression.codec = snappy
Datetime rebasing	The spark.sql.legacy-prefixed rebase configs were removed	Use names such as spark.sql.parquet.datetimeRebaseModeInWrite
JDBC	Type mappings changed for Postgres, MySQL, Oracle, MsSQL and DB2	Per-dialect configs such as spark.sql.legacy.postgres.datetimeMapping.enabled
Shuffle service	spark.shuffle.service.db.backend defaults to ROCKSDB	Set it to LEVELDB

Spark 4.0 also dropped Scala 2.12 and JDK 8/11 (Scala 2.13 and JDK 17 are the defaults), dropped Python 3.8 support in PySpark, removed Mesos support and dropped Hive metastore versions before 2.0.0.

What changed in Spark 4.1 and 4.2?

Spark 4.1 stops assuming a struct is null when a Parquet file lacks every requested field (spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing restores the old read), compresses RDD checkpoints by default (spark.checkpoint.compress) and raises the minimum pandas version for PySpark from 2.0.0 to 2.2.0. Spark 4.2 enables order-independent checksums for shuffle outputs by default (spark.sql.shuffle.orderIndependentChecksum.enabled) and rejects duplicate CTE names that differ only in case.

How do I keep old behavior while I fix code?

Each migration entry names the config that restores the previous behavior. Set them per session while you fix queries:

SET spark.sql.storeAssignmentPolicy=LEGACY;
SET spark.sql.legacy.timeParserPolicy=LEGACY;
SET spark.sql.ansi.enabled=false;

Plan to remove them. The ANSI docs call LEGACY store assignment the only behavior in Spark 2.x, and Spark 4.0 deleted some legacy configs outright.

How does Databricks approach the upgrade?

Databricks Runtime ties each release to a Spark version. As of the September 2026 release notes, 16.4 LTS includes Apache Spark 3.5.2, 17.3 LTS includes 4.0.0 and 18 LTS includes 4.1.0. Serverless compute enables ANSI mode by default. Databricks warns that upgrading Spark versions can bring syntax-breaking changes and points to the Databricks Runtime release notes and the Spark migration guide.

FAQs

Do I have to upgrade through every release?

The guides are written per step. Reading each section between your current and target versions covers every documented change.

Is Spark 3.5 still maintained?

Spark 3.5.9 was released on July 16, 2026. The versioning policy says maintenance releases contain bug fixes and security patches only.

Which Java, Scala and Python versions does Spark 4 need?

Spark 4.0 dropped Python 3.8 and Spark 4.1 dropped Python 3.9. The Spark 4.2 overview says Spark runs on Java 17/21/25, Scala 2.13 and Python 3.10+.