How do I migrate Spark jobs from EMR to Databricks?

Summary

• Databricks' migration guide says most Spark workloads from another Spark offering need only a few changes: remove SparkSession and shutdown boilerplate, drop custom executor and JVM settings, and use the library versions that ship with Databricks Runtime.
• Map each EMR piece to its Databricks equivalent: steps become Lakeflow Jobs tasks, bootstrap actions become libraries or init scripts, S3 data stays in place behind Unity Catalog external locations, and Airflow can keep triggering runs.
• Plan for four snags: RDD APIs and Spark Submit job tasks are not supported on standard compute, init scripts do not run on serverless, an external Hive metastore is read-only through federation, and Spark versions can differ.
• Validate one job in a test environment, compare its results with the EMR run, then cut over job by job.

How do I migrate Spark jobs from EMR to Databricks?

Most Spark code moves with a few changes, not a rewrite. Adapt the code with the steps in the Databricks migration guide, register your S3 data in Unity Catalog without copying it, rebuild each EMR step as a Lakeflow Jobs task, and compare results with EMR before you cut over.

What replaces each EMR concept on Databricks?

EMR concept	Databricks equivalent	Watch for
Cluster built from instance groups or fleets, optionally with managed scaling	Job compute that autoscales between a minimum and maximum worker count, or serverless compute	Custom executor and JVM settings can reduce performance, so don't copy them over
Bootstrap actions	Libraries and environments; init scripts on classic compute	Serverless compute does not support init scripts
Spark step (spark-submit through command-runner.jar)	Lakeflow Jobs task: Python script, JAR or notebook	The Spark Submit task is deprecated and standard compute does not support it
s3:// data paths	Unity Catalog external locations and external tables	Needs a storage credential backed by an IAM role
Hive metastore tables	Unity Catalog tables or Hive metastore federation	Foreign catalogs for an external Hive metastore are read-only
Airflow DAGs	Keep them and trigger Lakeflow Jobs	The Airflow provider is not directly supported by Databricks

How do I move the code and the data?

1. Adapt the code. Databricks creates an isolated SparkSession for each notebook or job, so remove session-creation code, or use SparkSession.builder().getOrCreate() to keep local testing. Remove sys.exit() and sc.stop(), and you can drop awaitTermination(), because Databricks cleans up jobs and streams when a run ends. Set only the Spark configurations that control type handling or functions.
2. Rewrite RDD code. Standard access mode, which Databricks recommends for most workloads, and serverless compute do not support RDD APIs. Use DataFrame APIs, or choose dedicated access mode on classic compute.
3. Compare Spark versions. Check the Apache Spark version of your EMR release against the version listed for each Databricks Runtime in the release notes, and read the Spark migration guide if they differ.
4. Point at the data. Create a storage credential from an IAM role and an external location over the S3 path, then register the existing files as an external table. External tables suit migrations because you can register data without copying it.
5. Adopt Delta Lake. Databricks recommends Delta Lake over Parquet or ORC for writes. CONVERT TO DELTA converts a Parquet table in place:

CONVERT TO DELTA parquet.`s3://my-bucket/path/to/table`
  PARTITIONED BY (date DATE); -- only if the table is partitioned

Avoid updating or appending files during the conversion, and send all later writes through Delta Lake.

How do I rebuild steps and scheduling?

Replace each EMR Spark step with a Lakeflow Jobs task. For a JVM job, create a JAR task, set the main class, point it at the JAR and copy the remaining arguments from your spark-submit command. For a Python job, use a Python script task. The compute table in the Jobs docs recommends serverless jobs for Python script and notebook tasks and classic jobs for JAR tasks.
If you keep Airflow, install the open source Databricks provider and use the DatabricksRunNowOperator, which triggers an existing Databricks job. Databricks recommends it because it avoids duplicating job definitions.
Replace bootstrap actions with libraries or environments where you can. Databricks recommends platform features over init scripts, and on classic compute with Unity Catalog and Databricks Runtime 13.3 LTS or above, it recommends storing init scripts in Unity Catalog volumes.

How do I phase the cutover?

Run a representative job in a test environment and compare its results and run time with your legacy infrastructure, as the migration guide advises. Move one job at a time, and keep the EMR version running until the outputs match.

FAQs

Do I need to rewrite my Spark jobs?

Usually not. Databricks says most workloads require only a few changes. Budget time for RDD code, spark-submit steps, custom Spark configurations and init scripts.

Can I keep my data in S3 and my Airflow scheduler?

Yes. External locations govern access to S3 paths in place, and Airflow can trigger Databricks jobs.

What happens to my Hive metastore tables?

Upgrade them to Unity Catalog tables, or use Hive metastore federation to govern them from Unity Catalog. Through federation, tables from an external Hive metastore are read-only, so register tables that your jobs write as Unity Catalog tables.

How do I know the migrated job performs well enough?

Run it in a test environment and compare results and run times with your legacy infrastructure. Check Jobs run history for duration, and avoid assuming a speedup or slowdown before you measure.