Spark on Kubernetes versus a managed service?

Summary

• Spark on Kubernetes runs the driver and executors as pods that Kubernetes schedules. You supply the cluster, container images, service account permissions, and security setup.
• A managed service such as Databricks allocates compute for you (serverless) or provisions clusters you configure (classic), but serverless and standard compute restrict some Spark APIs and settings.
• Choose by what you already operate and which restrictions your jobs can accept, and test one representative job on each.

Spark on Kubernetes versus a managed service?

Spark on Kubernetes puts you in charge of the cluster and everything Spark needs on it. A managed service takes that over and can restrict some APIs and settings. Spark supports Kubernetes through spark-submit, so the same application code can run on both, with the differences below.

How does Spark on Kubernetes work?

spark-submit starts the driver in a pod, and the driver creates executors as pods and runs your code. Kubernetes handles pod scheduling. When the application completes, executor pods terminate and are cleaned up, while the driver pod stays in the "completed" state with its logs until it is garbage collected or removed. Deleting the driver pod cleans up the whole application.

./bin/spark-submit \
    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=<spark-image> \
    local:///path/to/examples.jar

What do you take on with Kubernetes?

Task	What the Spark docs say
Cluster and access	A running Kubernetes cluster (the Spark 4.2 docs list version 1.34 or later) with kubectl access and Kubernetes DNS
Images	You supply container images. Spark ships a Dockerfile and bin/docker-image-tool.sh, and apache/spark:<version> images exist
Permissions	The driver's service account needs a Role or ClusterRole that lets it create pods and services
Security	Authentication is not enabled by default, and hostPath volumes have known security vulnerabilities, so admins should use the Pod Security Admission Controller
Pod-level control	Pod templates (spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile) cover settings that Spark properties do not, though Spark overwrites some values, and spark.kubernetes.scheduler.name selects a custom scheduler
Shuffle storage	Without local-storage volumes, Spark spills to emptyDir scratch space; persistent volume claims suit large shuffles
Scaling	Dynamic allocation scales executors up and down but needs shuffle tracking or a similar mechanism, and the external shuffle service is listed under future work
Sharing	Namespaces plus ResourceQuota divide cluster resources between users
Debugging	kubectl logs, kubectl port-forward <driver-pod-name> 4040:4040 for the Spark UI, and kubectl describe pod. After the application ends, the history server rebuilds the UI from event logs

What does a managed service take over, and what does it restrict?

On Databricks, serverless compute is allocated and managed for you, with no compute provisioned in your cloud account, and its runtime is upgraded automatically. Classic compute is provisioned resources that you create, configure, and manage. Databricks configures driver and executor settings itself and advises setting only the Spark configurations that are necessary.
The restrictions depend on the compute type. Serverless compute supports only Spark Connect APIs, so RDD APIs are not supported, most Spark compute configurations and Databricks Container Services are not available, and there is no Spark UI. Classic compute in standard access mode also lacks RDD APIs and spark-submit job tasks, and the docs point to dedicated compute when a workload needs them.
Databricks adds Photon, which replaces the JVM-based Spark SQL execution engine for supported operations, and Unity Catalog, which enforces access control and tracks lineage. Predictive optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, which is table maintenance.

When does each option fit?

• Spark on Kubernetes fits when you already operate Kubernetes, or when you need pod-level control, custom schedulers, or APIs that managed compute restricts, and you can staff the tasks in the table.
• A managed service fits when you would rather not run the cluster, images, permissions, and shuffle storage, and your jobs work within the API and setting restrictions of the compute type you choose.

FAQs

Can I move a Spark job from Kubernetes to a managed service?

Most Spark code needs few changes, but check four things. Jobs launched with spark-submit need another task type on standard compute. Executor sizing and other settings you tuned may not apply, because Databricks configures them. Custom container images are not supported on serverless. Tables need their Delta features and file paths checked, because a client that cannot support a feature in a table's protocol cannot read or write that table.

Can I run Spark on Kubernetes on premises?

Yes. Spark needs a running Kubernetes cluster and kubectl access, wherever the cluster runs, and the docs describe minikube for local testing. Databricks usage records list the clouds as AWS, Azure, and GCP.

How do I handle several teams on one cluster?

On Kubernetes, administrators combine namespaces and ResourceQuota to control sharing, and set spark.kubernetes.namespace per application. In Databricks, Unity Catalog enforces access control when users query a table.

How do costs compare?

Measure rather than assume. For Kubernetes, count the nodes you run, the idle capacity, and the staff time to operate them. For Databricks, the docs recommend benchmarking a representative workload and analyzing the system.billing.usage system table.