Self-managed Spark versus a managed service?

Summary

• The choice is mostly about who operates the cluster layer and which restrictions you accept. Self-managed Spark gives you control of the Spark release and its settings, and you run provisioning, upgrades, security, and monitoring yourself.
• A managed service takes over that layer but can add components and restrict APIs or settings. On Databricks, Photon is added, and serverless and standard compute restrict some Spark APIs and settings.
• Data in open formats can be read by both, with checks on Delta table features, managed-table access, and file paths.

Self-managed Spark versus a managed service?

Self-managed Spark means you download an Apache Spark release, choose a cluster manager, and operate everything around it. A managed service runs that layer for you, and it may add components or restrict settings, so the two are not interchangeable. This page uses Databricks as the managed example.

What do you run yourself with self-managed Spark?

• Cluster manager. Spark runs under its standalone manager, YARN, or Kubernetes, and spark-submit offers one interface for all of them. Standalone high availability uses standby masters with ZooKeeper.
• Security. Authentication is not enabled by default, so you secure the cluster before exposing it.
• Storage connectors. Object stores need filesystem connectors and a suitable output committer, and they cannot directly replace HDFS.
• Monitoring. Spark provides a web UI, event logs, a history server, and a REST API with a Prometheus-format endpoint.
• Upgrades. Spark ships a major release each year, and major releases may include breaking changes, so read the migration guide before each upgrade.
• Scheduling. Workflow scheduling is a separate tool, for example Apache Airflow.

What does a managed service take over, and what does it restrict?

On Databricks, serverless compute is allocated and managed for you and is versionless, so the runtime is upgraded automatically. Classic compute is provisioned resources that you create, configure, and manage, and you select the Databricks Runtime version, each of which includes Apache Spark. Databricks creates a SparkSession for each notebook or job and configures driver and executor settings, advising that custom executor or JVM settings can reduce performance.
Managed compute also restricts what Spark exposes. Serverless compute supports only Spark Connect APIs, so RDD APIs are not supported, most Spark compute configurations are not available, and the Spark UI and Spark logs are not available. Classic compute in standard access mode, which Databricks recommends by default, also lacks RDD APIs, spark-submit job tasks, and certain Spark properties, and the docs point to dedicated compute when a workload needs them.

Area	Self-managed Spark	Managed service (Databricks)
Cluster setup	Standalone, YARN, or Kubernetes, configured by you	Serverless is allocated for you; classic compute is created and configured by you
Spark settings	You set Spark properties directly	Databricks configures driver and executor settings; serverless lacks most Spark compute configurations and standard compute restricts some
Runtime upgrades	You choose and test each Spark release	Serverless upgrades automatically; on classic compute you pick a Databricks Runtime version
Monitoring	Web UI, history server, REST API	Jobs UI, notifications, and system tables; query profile instead of the Spark UI on serverless
Scheduling	A separate tool such as Apache Airflow	Lakeflow Jobs, workflow automation for Databricks
Execution engine	Spark's own engine	Photon replaces the JVM-based Spark SQL execution engine for supported operations

When does each option fit?

• Self-managed on machines you control. You need a specific Apache Spark release, unrestricted settings, or APIs that managed compute does not expose, and you can staff provisioning, upgrades, and security.
• Self-managed on Kubernetes. You already operate Kubernetes. Spark runs the driver and executors as pods, and you supply the images and the service account permissions.
• Managed serverless compute. You want the platform to allocate compute and upgrade the runtime, and your code works within the Spark Connect API limits.
• Managed classic compute. You want Databricks to provision clusters but need more control, for example dedicated compute when standard compute's limits block a workload.

FAQs

Is the engine the same in both?

Both expose Apache Spark APIs, but the execution path can differ. Databricks extends Apache Spark with custom optimizations, and Photon works with existing Spark API code with no changes required. Very short queries see little benefit because planning and scheduling dominate. Test your own workload instead of assuming parity.

Can I move between them?

Yes, with checks. Databricks creates the session and ends jobs for you, so code moving off Databricks needs session creation and awaitTermination() added back. A Delta client that cannot support a feature in a table's protocol cannot read or write that table, and Unity Catalog managed tables are meant to be read by name, not by file path. Lakeflow Pipelines are built on Spark Declarative Pipelines (Apache Spark 4.1 and later), but AUTO CDC, expectations, and the event log are Lakeflow-only.

Can I use both at the same time?

Yes, when tables are in open formats and both sides support each table's features. Open-source Spark can reach Unity Catalog tables through the Unity REST API for Delta clients and the Iceberg REST catalog endpoint for Iceberg clients.

How do I compare costs?

Count infrastructure, the staff time to patch, upgrade, and monitor, and metered usage. For Databricks serverless compute, the docs recommend running a representative workload and analyzing the system.billing.usage system table. Run the same job on the self-managed cluster and compare.