What's the difference between open-source Spark or Delta and Databricks?

Summary

• Apache Spark is the open-source engine and Delta Lake is the open-source storage framework. Databricks is a platform that runs both in its own runtime and manages the cloud infrastructure around them.
• The code is largely shared. Each Databricks Runtime version bundles a specific Apache Spark version, and Databricks says most existing Spark workloads need only a few changes.
• Some pieces now exist upstream: Spark Declarative Pipelines (Spark 4.1.0), Spark Connect (Spark 3.4) and an open-source Unity Catalog.
• Databricks adds Photon, Lakeflow Pipelines and Jobs, Auto Loader, built-in Unity Catalog governance and serverless or classic compute.

What is the difference between open-source Spark or Delta and Databricks?

Apache Spark and Delta Lake are open-source projects. Spark is the engine, and Delta Lake is the storage framework that adds ACID transactions on top of existing data lakes. Databricks is a platform that manages and deploys cloud infrastructure for you and ships Spark inside the Databricks Runtime. With open source, you run a cluster manager (Spark's standalone manager, YARN or Kubernetes) and choose your own scheduler, catalog and monitoring tools.

What do Spark and Delta Lake give you on their own?

Spark is a unified analytics engine for large-scale data processing with APIs in Java, Scala, Python and R. It includes Spark SQL, the pandas API on Spark, MLlib and Structured Streaming. It is licensed under Apache License 2.0.
Delta Lake provides ACID transactions, scalable metadata handling, schema enforcement, time travel, and merge, update and delete operations, and it unifies batch and streaming on one table. It has connectors for other engines such as Apache Flink, and it is an independent project under the Linux Foundation Projects, licensed under Apache License 2.0.
Version pairing is your job. Delta's compatibility table maps each Delta Lake release to specific Spark versions, and the 4.3.x line lists Spark 4.0.x and 4.1.x. Upstream has grown, too. Spark 4.1.0 added Spark Declarative Pipelines, Spark Connect (introduced in Spark 3.4) gives clients remote access through the DataFrame API, and Unity Catalog has an Apache 2.0 open-source implementation.

What does Databricks add?

• Runtime and compute. Each Databricks Runtime version lists the Apache Spark version it includes (18 LTS lists 4.1.0). Serverless compute runs in a serverless compute plane in your Databricks account, and classic compute runs in your own cloud account.
• Photon. For supported operations, Photon replaces the JVM-based Spark SQL execution engine with a native C++ runtime. It falls back to the Spark runtime for operations it does not support.
• Lakeflow Pipelines and Jobs. Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines. Lakeflow Jobs orchestrates tasks as part of a larger workflow.
• Governance and ingestion. Unity Catalog is the governance layer for data and AI, covering access control, lineage and auditing. Auto Loader incrementally processes new files as they arrive in cloud storage.

How do the two stacks line up?

Area	Open-source Spark and Delta Lake	On Databricks
Engine	Apache Spark at the version you choose	Databricks Runtime, which lists its Spark version
SQL execution	JVM-based Spark SQL engine	Photon for supported operations, Spark runtime otherwise
Cluster management	You operate standalone, YARN or Kubernetes	Serverless compute, or classic compute in your cloud account
Pipelines	Spark Declarative Pipelines (Spark 4.1.0 and later)	Lakeflow Pipelines, interoperable with Spark Declarative Pipelines
Orchestration	Your choice of scheduler	Lakeflow Jobs
Governance	Open-source Unity Catalog or another catalog	Unity Catalog, built in
Remote clients	Spark Connect	Databricks Connect, built on Spark Connect
Billing	No software license fee, plus your infrastructure and operations	Usage metered in DBUs

When should I choose each?

Open source fits when you must run Spark on hardware you own, or want to control every component and version. Databricks fits when you want the platform to provision compute, run jobs and apply governance so your team works on pipelines instead of cluster operations.
Cost differs in kind. Open-source software has no license fee, but you pay for the infrastructure and the people who operate it. Databricks usage is metered in Databricks Units (DBUs), a normalized unit of processing power, and billed pay as you go or under committed-use contracts.

FAQs

Do I have to rewrite my Spark code for Databricks?

Databricks' migration guide says most workloads require only a few changes. It lists using Delta Lake instead of Parquet, using the libraries packaged in the Databricks Runtime, calling SparkSession.builder().getOrCreate() instead of creating a new session, and removing sys.exit() and sc.stop(). It also warns that custom executor or JVM settings can reduce performance.

Can I use Delta Lake without Databricks?

Yes. Delta Lake is Apache 2.0 licensed and sits under the Linux Foundation Projects. Check the compatibility table for the Spark version you run.

How do I connect an IDE to a remote cluster?

Spark Connect is a decoupled client-server architecture that allows remote connectivity to Spark clusters. Databricks Connect is built on it and connects IDEs to Databricks compute.

How much does Photon speed things up?

It depends on the query. The Photon docs say it gives the greatest benefit for longer-running queries that process large datasets, and queries that normally complete in under two seconds don't see meaningful improvement. Photon is compatible with Apache Spark APIs, so existing code needs no changes, and unsupported operations fall back to the Spark runtime.