How do I pick a managed Spark platform?

Summary

• Judge a managed Spark platform on what you can test: the Apache Spark version it runs, which Spark APIs and settings its managed compute allows, which open table formats and catalog APIs it exposes, how usage is metered, and how you would leave.
• Run a real job on each shortlisted platform, read the same tables from open-source Spark, and rerun the job there before you commit.
• Open table formats reduce data lock-in, but catalogs, pipeline extras, and orchestration can still tie you in.

How do I pick a managed Spark platform?

Start from your workloads (batch ETL, SQL analytics, streaming, ML), then test each platform against criteria you can check yourself instead of comparing feature lists. Use a real job from your estate and record what you had to change.

Which criteria should I test?

Criterion	What to test
Spark version	Which Apache Spark version each runtime ships. Spark has annual major releases and quarterly feature releases, so ask how soon new ones arrive
API and setting limits	Which Spark APIs, configurations, and UIs the managed compute allows, for example whether RDD APIs and Spark settings are available
Table formats	Whether you can read and write Delta Lake and Apache Iceberg tables, and whether other engines can reach them through open APIs such as the Iceberg REST catalog protocol
Delta table features	Which table features the platform enables, because a client that cannot support a feature in a table's protocol cannot read or write that table
Pipeline code	Whether pipelines use open APIs such as Spark Declarative Pipelines (Apache Spark 4.1 and later), and which extras are proprietary
Governance	Access control, audit, and lineage in one catalog, and whether they still apply when other engines read the data
Compliance	Which controls cover the compute type and region you will use, and whether a separate mode must be enabled for regulated data
Cost visibility	Whether usage is metered per workload, can be tagged, and can be queried
Exit path	Whether you can export data and rerun jobs elsewhere

How does Databricks approach these criteria?

• Compute. Serverless compute, where Databricks allocates and manages the resources, classic compute, which is provisioned resources you create, configure, and manage, and SQL warehouses.
• Spark. Apache Spark powers the compute clusters and SQL warehouses. The Databricks Runtime release notes list the Spark version of each runtime, for example 4.1.0 for 18 LTS.
• Formats and catalog. Delta Lake is the default table format and Iceberg is supported. Unity Catalog is the governance layer, and its open APIs let external Delta and Iceberg clients reach managed tables with credential vending.
• Limits to test. Serverless compute supports only Spark Connect APIs, so RDD APIs are not supported, and most Spark compute configurations are not available. The Spark UI is not available there, and ANSI mode is on by default.
• Code changes. Moving existing Spark code to Databricks means removing SparkSession creation commands and terminal commands such as sys.exit() or sc.stop(), and recompiling against the libraries in the Databricks Runtime.
• Pipelines. Lakeflow Pipelines are built on Spark Declarative Pipelines, so transformation code stays portable to other SDP runtimes. AUTO CDC, data quality expectations, and the queryable event log are Lakeflow-only.
• Compliance. The compliance security profile is required for data regulated under standards such as HIPAA and PCI-DSS.
• Cost. Usage is recorded in the system.billing.usage system table in DBUs, with custom tags.

FAQs

What should a proof of concept include?

One representative batch job, one SQL workload, and a streaming job if you have one, each on real data. Record the Spark version, the code changes needed, the setting limits you hit, and the usage data produced. Then run the exit test: read the same tables from open-source Spark and rerun a job there.

Should we self-manage Spark on Kubernetes instead?

Self-managing means taking on the work Spark documents for Kubernetes. You submit with spark-submit --master k8s://<k8s-apiserver-host>:<k8s-apiserver-port> --deploy-mode cluster, which starts a driver pod that creates executor pods. You supply container images (Spark ships a Dockerfile and bin/docker-image-tool.sh), grant the driver's service account permission to create pods and services, and secure the cluster, because Spark authentication is not enabled by default. Shuffle spill uses emptyDir volumes unless you configure local storage, and completed driver pods remain until cleaned up. A managed platform takes over tasks like these, so weigh that against the setting limits you find in your proof of concept.

How do I compare cost across platforms?

Run the same job on each platform and compare the metered usage for that run. On Databricks, system.billing.usage records usage_quantity in a usage_unit (DBU) with custom_tags and usage_metadata that identifies jobs and compute resources, so tag test runs to find them.

How do I test the exit path?

Pick one Delta table and check what it requires: DESCRIBE DETAIL reports minReaderVersion and minWriterVersion. Read it from open-source Spark with the Delta Lake package, then rerun one job with spark-submit on an open cluster manager, since spark-submit offers one interface across them. If the platform's catalog exposes an Iceberg REST endpoint, also read an Iceberg table through it.

How do we limit vendor lock-in?

Open formats and open Spark APIs help, but check three more layers: the catalog and its APIs, pipeline and orchestration definitions, and runtime-specific features. Table protocols matter too: a Delta Lake client can read a table only if it supports the table's reader features, so test reads from open-source Spark before committing.