How do skills and hiring differ across Spark platforms?

Summary

• The Spark docs list Standalone, Hadoop YARN and Kubernetes as supported cluster managers and say "Spark is agnostic to the underlying cluster manager", so DataFrame, SQL and PySpark skills carry across platforms.
• What changes is what you operate: Standalone means starting and securing masters and workers, YARN means Hadoop configuration, Kubernetes means container images and RBAC, and Databricks serverless compute means no compute to provision in your cloud account.
• API reach also differs: Spark Connect "does not support all the execution APIs of Spark, most importantly RDDs", and Databricks serverless compute supports only Spark Connect APIs.
• The documentation describes systems, not job markets, so the interview questions below come from documented answers.

How do skills and hiring differ across Spark platforms?

Spark applications run the same way on every cluster manager: "independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program)". The platforms differ in what sits underneath that driver program. The Spark and Databricks documentation describes systems, not labor markets, so this page makes no claim about demand, salaries or job-posting trends. It maps each platform's documented setup to skill areas and to interview questions with documented answers.

Which Spark skills carry across platforms?

• The application model. Driver, executors, tasks, jobs and stages are defined once in the cluster overview, which applies to Standalone, YARN and Kubernetes alike.
• APIs. Spark "provides high-level APIs in Java, Scala, Python and R" and includes "Spark SQL for SQL and structured data processing". On Databricks, "Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks."
• Debugging. "Each driver program has a web UI, typically on port 4040, that displays information about running tasks, executors, and storage usage."
• Performance tuning. The tuning guide covers the level of parallelism, and in general recommends "2-3 tasks per CPU core in your cluster".

What does each platform ask you to operate?

Platform	What the docs have you set up	Skill area
Spark Standalone	Start a master with ./sbin/start-master.sh and workers with ./sbin/start-worker.sh <master-spark-URL>, or list hosts in conf/workers for launch scripts that, by default, need password-less ssh. Authentication is "not enabled by default"	Cluster setup with shell scripts and ssh, cluster security
Hadoop YARN	Point HADOOP_CONF_DIR or YARN_CONF_DIR at the Hadoop client configuration and submit with --master yarn. Read container logs with yarn logs -applicationId <app ID> when log aggregation is on. The YARN guide has a Kerberos section	Hadoop and YARN administration, Kerberos
Kubernetes	A cluster with kubectl access, permission to list, create, edit and delete pods, Kubernetes DNS, a container image (spark.kubernetes.container.image) and a driver service account	Kubernetes administration, container images, RBAC
Databricks	Serverless compute lets you "run workloads without provisioning any compute resources in your cloud account". Classic compute is "Provisioned compute resources that you create, configure, and manage for your workloads"	Choosing and configuring compute types

What do open-source clusters add: security and monitoring?

On Standalone, YARN and Kubernetes, the Spark docs list these settings and tools:

• Security. The Spark security page says "Security features like authentication are not enabled by default" and that "none are secure by default". RPC authentication is controlled by spark.authenticate, which defaults to false.
• Monitoring after the fact. Spark's history server rebuilds an application's UI from its event logs. You start it with ./sbin/start-history-server.sh, its web interface is on port 18080 by default, and applications must log events with spark.eventLog.enabled and spark.eventLog.dir.
• Metrics. Spark has "a configurable metrics system based on the Dropwizard Metrics Library" that can report to sinks including HTTP, JMX and CSV files.

How does API reach differ by platform?

The Spark Connect protocol "does not support all the execution APIs of Spark, most importantly RDDs", and its client "does not have access to the static Spark configuration or the SparkContext". Databricks Connect "is built on open-source Spark Connect". The serverless compute limitations page says "Only Spark Connect APIs are supported. Spark RDD APIs are not supported." and "R is not supported." It also says "ANSI mode is enabled by default", so invalid casts and arithmetic overflow raise runtime errors instead of silently returning NULL.
In practice, code written with RDDs or SparkContext has to move to Spark Connect compatible APIs, such as DataFrames and SQL, before it can run over Spark Connect or on serverless compute.

Which interview questions have documented answers?

Platform	Question	Documented answer
Any	Where do you see running tasks, executors and storage usage?	The driver's web UI, typically on port 4040
Standalone	What happens by default if the Master crashes?	No new applications can be created. ZooKeeper standby Masters or FILESYSTEM recovery mode are the two high availability schemes
YARN	Where does Spark read the Hadoop cluster configuration from?	HADOOP_CONF_DIR or YARN_CONF_DIR
Kubernetes	What must the driver pods' service account be allowed to create?	Pods, services and configmaps
Databricks	Which Spark APIs are unavailable on serverless compute?	Spark RDD APIs, because only Spark Connect APIs are supported. R is not supported either

FAQs

Do I need Hadoop skills to work with Spark?

Not for every deployment. The cluster overview lists Standalone, Hadoop YARN and Kubernetes as supported cluster managers, and the Hadoop client configuration is a documented prerequisite on YARN specifically.

Do Spark skills from open-source clusters transfer to Databricks?

The application skills do. Databricks Connect connects IDEs such as Visual Studio Code, PyCharm and IntelliJ IDEA to Databricks compute, and the serverless limitations above show where to check API support.