Is Databricks worth it over open-source Spark?

Summary

• Apache Spark is released under the Apache License, Version 2.0, and runs on a Standalone, Hadoop YARN or Kubernetes cluster that you operate yourself.
• Databricks combines Apache Spark with Delta and manages the cloud infrastructure for you. It adds serverless compute, Photon, Unity Catalog, predictive optimization, Auto Loader and Databricks Connect.
• Open-source Spark fits when you already run Kubernetes or YARN, need Spark on premises, or want every layer open source and replaceable.
• Databricks fits when you want the platform operated for you, with governance and table maintenance built in.
• Existing Spark code needs some changes to run on Databricks, and Delta Lake is open source software that extends Parquet data files with a transaction log.

Is Databricks worth it over open-source Spark?
Whether Databricks is worth it depends on how much of the platform your team wants to operate itself. Apache Spark is a multi-language engine released under the Apache License, Version 2.0. Databricks combines Apache Spark with Delta and, in its own words, "manages and deploys cloud infrastructure for you." This page lists what open-source Spark leaves to you, what Databricks adds, and where each fits. It makes no speed or cost claims.

What do you run yourself with open-source Apache Spark?

You choose and operate a cluster manager. The Spark docs list Standalone, Hadoop YARN and Kubernetes. On Kubernetes, Spark uses its native scheduler. The current docs require a running cluster at Kubernetes version 1.34 or later with kubectl access, Kubernetes DNS, and permission to create pods, and you submit applications with spark-submit --master k8s://<api_server_host>:<k8s-apiserver-port>.
Security is also yours. The Spark security guide says "Security features like authentication are not enabled by default", and RPC authentication is turned on with the spark.authenticate setting.
Tuning is yours too, although Spark ships defaults for common cases. spark.sql.shuffle.partitions defaults to 200. Adaptive Query Execution has been on by default since Spark 3.2.0, and spark.sql.adaptive.coalescePartitions.enabled (default true) merges small shuffle partitions at runtime. The tuning guide covers data serialization and memory tuning.
For tables, Delta Lake provides OPTIMIZE and VACUUM, with a default VACUUM retention threshold of 7 days, and you decide when to run them. For a catalog, Unity Catalog OSS is available under the Apache 2.0 license, but its README says the APIs "are currently evolving and should not be assumed to be stable."
Support comes from the community: the Spark community page lists the apache-spark tag on StackOverflow and the user@spark.apache.org mailing list.

When is open-source Spark the better fit?

Run Spark yourself if any of these describe your situation:

• You already operate Kubernetes or YARN and have engineers who can run Spark on it.
• You need Spark on premises, because Databricks documents its platform for AWS, Azure and Google Cloud.
• You want every layer of the stack to be open source and replaceable.

What does Databricks add on top of Apache Spark?

Each row pairs an area that open-source Spark leaves to you with the Databricks feature that covers it.

Area	Open-source Apache Spark	Databricks
Compute	You operate a Standalone, YARN or Kubernetes cluster	Serverless compute, where "Databricks automatically allocates and manages the necessary compute resources"; classic compute adds automatic termination and autoscaling settings
Query engine	The JVM-based Spark SQL execution engine	Photon, a Databricks-native vectorized engine for SQL, DataFrame API calls, ETL and stateless streaming
Table maintenance	Delta Lake OPTIMIZE and VACUUM, which you run	Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables
Governance	Authentication off by default; Unity Catalog OSS with evolving APIs	Unity Catalog, built into Databricks, enforces access control, tracks lineage and logs activity for auditing
Pipelines	Spark Declarative Pipelines (SDP), a declarative framework for data pipelines on Apache Spark	Lakeflow Pipelines extend SDP and run on the Databricks Runtime
Development	spark-submit and local sessions	Databricks Connect, built on open-source Spark Connect, runs Spark API code on Databricks compute

Photon needs no code changes, but its limits matter: it does not support UDFs, RDD APIs or Dataset APIs, and it supports stateless streaming only. When a workload hits an unsupported operation, the compute switches to the Spark runtime for the rest of that operation. Auto Loader (cloudFiles) incrementally processes new files as they arrive in cloud storage such as Amazon S3, Azure Data Lake Storage or Google Cloud Storage.

Will my existing Spark code run on Databricks unchanged?

Not entirely. The Databricks migration guide "outlines the required changes to adapt existing Apache Spark workloads to run on Databricks". It recommends Delta Lake instead of Parquet or ORC, the library versions packaged in Databricks Runtime, and SparkSession.builder().getOrCreate() so code still compiles and tests locally. It also says to remove sys.exit() and sc.stop(), because Databricks terminates and cleans up jobs when they complete. On serverless compute, "Spark RDD APIs are not supported", Hive SerDe tables are not supported, and "Serverless compute does not support setting most Spark properties for notebooks or jobs."

Does Databricks lock me in?

Your data format is open, but the managed features are Databricks-specific. The Databricks docs state that "Delta Lake is open source software" that extends Parquet data files with a file-based transaction log. Iceberg reads configure Delta Lake tables to generate Iceberg metadata, so Iceberg clients can read them without rewriting files. Photon, predictive optimization, serverless compute and the built-in Unity Catalog governance are Databricks features, so plan for them separately.

Can I run Databricks on premises?

The Databricks architecture docs describe a control plane in the Databricks account and two compute planes: classic compute in your cloud account, and serverless compute in the Databricks account. They describe no on-premises compute plane, and Databricks publishes separate docs for AWS, Azure and Google Cloud. If you need Spark on premises, run open-source Spark.