What's a good multi-cloud Spark strategy?

Summary

• Keep data in Delta Lake or Apache Iceberg tables on each cloud's object store. Spark reaches S3, Azure Data Lake Storage Gen2 and Google Cloud Storage through separate connectors (s3a, abfs, gs) with their own settings.
• Expect storage differences. Delta Lake's docs give Azure storage and Google Cloud Storage built-in support, while concurrent writes from several Spark drivers to S3 need an experimental mode that uses DynamoDB.
• Run Spark on Kubernetes, one of the cluster managers Spark supports, with spark-submit --master k8s://... and a container image.
• Use a catalog with open APIs, such as the Iceberg REST catalog or Unity Catalog OSS, a sandbox project whose APIs are still evolving.
• Cross-cloud transfers can incur vendor egress fees. On Databricks, a Unity Catalog metastore is per region, and the docs point to OpenSharing (formerly Delta Sharing) between cloud providers.

What's a good multi-cloud Spark strategy?
Store data in an open table format on each cloud's object store, run Spark on a cluster manager that is not tied to one provider, put a catalog with open APIs in front, and plan for the per-cloud differences in storage connectors and data transfer fees.

Which layers stay the same on every cloud?

Layer	Portable choice	What changes per cloud
Table format	Delta Lake or Apache Iceberg	Delta Lake's S3 write setup; Iceberg's FileIO class per store
Storage access	Spark's filesystem connectors	URI scheme (s3a, abfs, gs), settings, credentials
Compute	Spark on Kubernetes	The cluster and its API server
Catalog	Iceberg REST catalog API, Unity Catalog OSS	The Unity Catalog OSS server's per-cloud storage settings
Pipelines	Spark Declarative Pipelines	The storage checkpoint directory and source paths

How do I store data on each cloud?

Spark reads and writes object stores "through filesystem connectors implemented in Hadoop or provided by the infrastructure suppliers themselves," and each cloud connector "has its own set of configuration parameters." Delta Lake uses "the scheme of the path" to identify the storage system, so the write call is the same on each cloud:

spark.range(5).write.format("delta").save("s3a://<your-s3-bucket>/<path-to-delta-table>")
spark.range(5).write.format("delta").save("abfss://<container-name>@<storage-account>.dfs.core.windows.net/<path-to-delta-table>")
spark.range(5).write.format("delta").save("gs://<bucket>/<path-to-delta-table>")

The guarantees differ. Delta Lake's storage docs give Azure storage and Google Cloud Storage built-in support "with full transactional guarantees for concurrent reads and writes from multiple clusters." For S3, the docs caution that "Concurrent writes to the same Delta table on S3 storage from multiple Spark drivers can lead to data loss" unless you use the multi-cluster mode, which they call experimental and which needs DynamoDB permissions.
Iceberg's introduction lists "Works with any cloud store." It has a FileIO class for each cloud store (S3FileIO, GCSFileIO, ADLSFileIO), selected with the io-impl catalog property. Spark's connector docs rate directory rename as unsafe on s3a, safe on abfs and mixed on gs.

How do I run Spark on every cloud?

Spark "can run on clusters managed by Kubernetes," and its docs call Spark "agnostic to the underlying cluster manager." Submit to the Kubernetes API server with a k8s:// master URL and a container image:

./bin/spark-submit \
  --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
  --deploy-mode cluster \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.kubernetes.container.image=<spark-image> \
  local:///path/to/examples.jar

The cloud-integration docs say to include the hadoop-cloud module (spark-hadoop-cloud_2.13) on the application classpath to reach object stores.

How do I use one catalog across clouds?

Unity Catalog OSS says it is compatible with Apache Hive's metastore API and Apache Iceberg's REST catalog API. With the io.unitycatalog:unitycatalog-spark_2.13 and io.delta:delta-spark_2.13 packages, a Spark session reaches a server through UCSingleCatalog:

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.<catalog_name>=io.unitycatalog.spark.UCSingleCatalog
spark.sql.catalog.<catalog_name>.uri=http://<uc-server-host>:8080
spark.sql.catalog.<catalog_name>.token=
spark.sql.defaultCatalog=<catalog_name>

The server holds the cloud storage settings. Its etc/conf/server.properties takes s3.*, adls.* and gcs.* keys, such as s3.bucketPath.0, adls.storageAccountName.0 and gcs.bucketPath.0 (the current template labels the s3.* keys legacy per-bucket configuration), and etc/conf/hibernate.properties sets the backend database (the deployment guide shows MySQL and PostgreSQL). The project calls itself a sandbox project, and its README says the APIs "are currently evolving and should not be assumed to be stable." The empty token means no authentication, and the template ships with server.authorization=disable.

Can my pipelines stay portable?

Spark Declarative Pipelines (SDP), added in Spark 4.1, is "a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark." spark-pipelines "is built on top of spark-submit, meaning that it supports all cluster managers supported by spark-submit," and the guide lists Amazon S3, Azure ADLS Gen2 and Google Cloud Storage as ingestion sources. A pipeline spec lists the transformation files (libraries) and a storage directory for streaming-table checkpoints:

name: my_pipeline
libraries:
  - glob:
      include: transformations/**
storage: file:///absolute/path/to/storage/dir
catalog: my_catalog
database: my_db

How does Databricks fit a multi-cloud plan?

Databricks "workspaces can be hosted on Amazon AWS, Microsoft Azure, and Google Cloud Platform." Its Unity Catalog best practices say "You can have only one metastore per region" and "Use Databricks-to-Databricks OpenSharing between cloud regions or cloud providers" (OpenSharing, formerly Delta Sharing). Across that boundary, lineage graphs "do not cross region or platform boundaries," and access control "is defined at the metastore level, and does not cross region or platform boundaries."

FAQs

Do I have to replicate data to every cloud?

Not necessarily. OpenSharing "does not require data replication," but "your cloud vendor may charge data egress fees when you share data across clouds or regions." The Unity Catalog guidance is to "Use OpenSharing for tables that are infrequently accessed, because you are responsible for egress charges from cloud region to cloud region."

Should I choose Delta Lake or Iceberg for multi-cloud?

Both document the major clouds, and they interoperate. Delta Lake's UniForm lets you read Delta tables with Iceberg and Hudi clients, and Databricks "Iceberg reads" (UniForm) generate Iceberg metadata so Iceberg clients can read Delta tables.