What does Databricks add on top of open-source Spark?

Summary

• Apache Spark is the engine, and Databricks Runtime includes it: each Runtime release lists its Spark version, and Runtime 19 lists Apache Spark 4.2.0.
• Databricks adds Photon (a native vectorized execution engine for supported operations), Unity Catalog (governance), Lakeflow Pipelines and Lakeflow Jobs, serverless compute, Auto Loader, predictive optimization and Databricks Connect.
• Photon is compatible with Apache Spark APIs and falls back to the Spark runtime for operations it doesn't support, so most Spark code needs no changes. Serverless compute is stricter: it supports only Spark Connect APIs, not RDD APIs.
• Open-source Spark gives you the engine, the APIs and your choice of cluster manager (standalone, YARN or Kubernetes), and you operate the cluster. Unity Catalog and Spark Declarative Pipelines also exist as open-source options.
• Code that uses only Apache Spark APIs follows the Apache Spark docs. Databricks-specific features, such as Auto Loader's cloudFiles source, are not among Apache Spark's built-in streaming sources, so they need a replacement elsewhere.

What does Databricks add on top of open-source Spark?

Apache Spark is the engine, and Databricks packages it as a managed platform. Databricks Runtime includes Apache Spark, and Databricks layers on an accelerated execution engine for supported operations (Photon), governance (Unity Catalog), pipelines and orchestration (Lakeflow Pipelines and Lakeflow Jobs), serverless compute, incremental file ingestion (Auto Loader), automatic table maintenance (predictive optimization) and remote development (Databricks Connect). You still write Spark code against the Apache Spark APIs.

What does open-source Spark provide?

Spark describes itself as "a unified analytics engine for large-scale data processing". It has APIs in Java, Scala, Python and R, and modules for Spark SQL, the pandas API on Spark, MLlib, GraphX and Structured Streaming. You run it on Spark's standalone cluster manager, Hadoop YARN or Kubernetes, and you operate that cluster yourself. Spark Connect, introduced in Spark 3.4, is a decoupled client-server architecture for remote connectivity, and it supports PySpark and Scala applications. Spark 4.1.0 added Spark Declarative Pipelines for defining pipelines in SQL and Python.

Which additions does Databricks provide?

Addition	What the Databricks docs say	Worth knowing
Databricks Runtime	Each release lists the Apache Spark version it includes	Runtime 18 LTS lists Spark 4.1.0 and Runtime 19 lists Spark 4.2.0
Photon	A Databricks-native vectorized query engine; for supported operations it replaces the JVM-based Spark SQL execution engine with a native C++ runtime	Catalyst still plans the query; UDFs, RDD APIs and Dataset APIs aren't supported, so Photon falls back to Spark
Unity Catalog	The unified governance layer for data and AI: access control, lineage and auditing	Also available as an open-source implementation
Lakeflow Pipelines	Extend, and are interoperable with, Spark Declarative Pipelines while running on Databricks Runtime	Serverless pipelines run on Databricks-managed compute
Lakeflow Jobs	Workflow automation: tasks with branching and looping, started by a schedule, table updates, file arrival or continuously	You can also schedule jobs from Apache Airflow
Serverless compute	Run workloads without provisioning compute resources in your cloud account	Supports only Spark Connect APIs; RDD APIs aren't supported
Auto Loader	A Structured Streaming source called cloudFiles that incrementally processes new files in cloud storage	Databricks recommends it for streaming ingestion from cloud object storage
Predictive optimization	Automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables	Enabled by default for accounts created on or after November 11, 2024
Databricks Connect	A client library to connect IDEs, notebooks and custom applications to Databricks compute	Lets you write Spark API code and run it remotely

Which compute options does Databricks offer?

Databricks compute comes as serverless compute, classic compute for customizable resources, or SQL warehouses for analytics. Photon is enabled on serverless compute and SQL warehouses, and on classic all-purpose and jobs compute it is on by default and can be toggled. Photon helps most on longer-running queries over large datasets.

FAQs

Can I run my Apache Spark code on Databricks?

Yes for code that uses the Apache Spark APIs, because Databricks Runtime includes Spark and Photon is compatible with those APIs. Check the compute type, though: serverless compute supports only Spark Connect APIs, and Spark RDD APIs aren't supported there.

Do I need Unity Catalog?

It depends on the feature. Unity Catalog is automatically enabled for workspaces created after November 8, 2023, and older workspaces can be upgraded. Serverless pipelines need a workspace with Unity Catalog enabled and always use it, while the pipelines docs still describe working with the legacy Hive metastore for some workloads.

What if I stay on open-source Spark?

You choose and operate the cluster manager, and you don't get Photon or Databricks serverless compute. You can still add a catalog with Unity Catalog's open-source implementation, which is Apache 2.0 licensed, and define pipelines with Spark Declarative Pipelines from Spark 4.1.0.

Is my Spark job portable off Databricks?

Jobs that use only Apache Spark APIs follow the Apache Spark docs. The built-in Structured Streaming sources in Apache Spark are file, Kafka, socket, rate and rate per micro-batch, so Auto Loader's cloudFiles source is a Databricks feature that you would replace outside Databricks.