How do I move on-prem Spark to the cloud?

Summary

• Move in stages and change one thing at a time: copy data to object storage, move the metastore, run the same jobs on cloud compute, compare outputs, then cut over workload by workload.
• Object stores are not real filesystems. Spark's default rename-based commit can be slow and unreliable on them, so choose a committer for your store before writing production data.
• Treat a Spark upgrade, the switch from Kerberos tokens to cloud credentials, and any cluster manager change as separate steps with their own test pass.

How do I move on-prem Spark to the cloud?

Work through these steps in order: inventory what runs today, copy the data to cloud object storage, move table metadata, run the same jobs in the cloud and compare outputs, then cut over wave by wave. Keep infrastructure, Spark version, and code changes in separate steps so any difference has one cause.

What are the migration phases?

Phase	What to do
Inventory	List the Spark version, cluster manager, JARs, data sources, scheduler, and Kerberos use. Spark runs under its standalone manager, YARN, or Kubernetes, and spark-submit offers one interface for all of them
Copy data	Bulk copy HDFS data to object storage with Hadoop DistCp, then cross-check source and destination listings
Move metadata	Point cloud Spark at a Hive metastore with enableHiveSupport() and hive-site.xml, core-site.xml, and hdfs-site.xml in conf/, or recreate table definitions. spark.sql.hive.metastore.version accepts Hive 2.0.0 to 2.3.10, 3.0.0 to 3.1.3, and 4.0.0 to 4.1.0. Set spark.sql.warehouse.dir for the default location of managed databases and tables
Commit safely	Pick an output committer for your store, as covered below
Credentials	Replace HDFS Kerberos tokens with cloud credentials. Jobs must authenticate with the object store, and in cloud infrastructure the credentials are usually set up automatically
Validate	Run old and new side by side and compare outputs. The SQL migration guide lists behavior changes by version
Tune	Size executors and test dynamic allocation, which is off by default (spark.dynamicAllocation.enabled)
Cut over	Move low-risk workloads first and verify each wave before the next

Are object stores a drop-in replacement for HDFS?

No. Spark's cloud integration docs say object stores cannot replace a cluster filesystem such as HDFS except where explicitly stated. Directory emulation can be slow, renames may be very slow and can leave the store in an unknown state on failure, and Spark's default rename-based commit is potentially both slow and unreliable. Directory rename safety differs by connector:

Store	Connector	Directory rename
Amazon S3	s3a	Unsafe
Azure Data Lake Gen 2	abfs	Safe
Google Cloud Storage	gs	Mixed

The docs note that S3, GCS, and Azure storage are consistent as of 2021, meaning a written file can be listed and opened at once, but connectors make no guarantees about objects overwritten while a stream is reading them, so avoid overwriting files other clients are reading. To read s3a:// paths, add the hadoop-cloud module (for example spark-hadoop-cloud_2.13) and its dependencies to the classpath. For S3, switch to the S3A committers, which ship in the hadoop-aws JAR in Spark builds with Hadoop 3.1 or later:

spark.hadoop.fs.s3a.committer.name directory
spark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter

For ADLS Gen2 and GCS, the manifest committer needs Hadoop 3.3.5 or later. The S3A committers do not support dynamic partition overwrite, which INSERT OVERWRITE TABLE and Dataset writes in overwrite mode use, and such writes fail with "PathOutputCommitter does not support dynamicPartitionOverwrite". Where rename-based commits are safe, the docs recommend the FileOutputCommitter v2 algorithm (spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2) for performance and v1 for safety; v2 is unsafe without consistent listings.

FAQs

How do I copy the data?

Use Hadoop DistCp, for example hadoop distcp -direct hdfs://nn1:8020/datasets/set1 s3a://bucket/datasets/set1. The -direct option writes straight to the destination, avoiding costly temporary renames on object stores. DistCp recommends cross-checking source and destination listings afterward. With -update on object stores, only modification time and length are generally compared, not checksums.

Should I upgrade Spark during the move?

Not in the same step. Major releases (x.0.0) may include breaking changes and dependency upgrades, while feature releases promise no behavior changes by default. The SQL migration guide lists the 4.0 changes: spark.sql.ansi.enabled is on by default, CREATE TABLE without USING or STORED AS no longer defaults to Hive, and some JDBC type mappings changed. Spark 4 drops Scala 2.12, so rebuild Scala JARs against Scala 2.13.

Can I migrate streaming jobs?

Yes. A checkpoint location must be a path in an HDFS-compatible file system, and the default checkpoint file manager should only write to a store with a fast, atomic rename(). On S3 with Hadoop 3.3.1 or later and the S3A connector, you can set spark.sql.streaming.checkpointFileManagerClass to org.apache.spark.internal.io.cloud.AbortableStreamBasedCheckpointFileManager; then take extra care not to reuse a checkpoint location across queries running in parallel, which could corrupt checkpoint data.

How do jobs authenticate, and what happens to Kerberos?

Spark jobs must authenticate with the object store. For S3, spark-submit reads the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables, and secrets should never be checked into source repositories. Spark obtains delegation tokens for Kerberos-secured HDFS, HBase, and a Hive metastore (when hive.metastore.uris is set), so keep Kerberos configured if cloud jobs still call a Kerberized metastore. Spark authentication is not enabled by default, so secure new clusters before use.

Which Parquet settings help on object stores?

The Spark docs recommend spark.hadoop.parquet.enable.summary-metadata false, spark.sql.parquet.mergeSchema false, spark.sql.parquet.filterPushdown true, and spark.sql.hive.metastorePartitionPruning true, which minimize the data read during queries.