Do I need Databricks, or just open-source Spark and Delta?

Summary

• No. Apache Spark and Delta Lake are open source and run without a Databricks account. Delta Lake provides ACID transactions, schema enforcement, time travel, and upserts and deletes on storage such as S3, ADLS, GCS and HDFS.
• With open source you operate the surrounding pieces yourself: a cluster manager (Standalone, Hadoop YARN or Kubernetes), security setup (Spark's authentication and other security features are not enabled by default), monitoring, orchestration and version upgrades.
• Databricks is a managed platform around the same open source projects. It adds serverless and classic compute, Databricks Runtime, Photon, Unity Catalog, Lakeflow Jobs, Lakeflow Pipelines and notebooks.
• Delta tables move between the two paths. Each table has a protocol that lists the features a client must support, so test open source clients before you enable new table features on a table that both sides use.

Do I need Databricks, or just open-source Spark and Delta?

No. Apache Spark is "a unified analytics engine for large-scale data processing", and Delta Lake is "an open source project that enables building a Lakehouse architecture on top of data lakes". Neither requires Databricks. The real question is how much of the surrounding platform your team wants to operate.

What do open-source Spark and Delta Lake give you?

The Delta Lake documentation lists these capabilities:

• ACID transactions on Spark. Serializable isolation levels ensure that readers never see inconsistent data.
• Schema enforcement. Delta Lake handles schema variations to prevent insertion of bad records during ingestion.
• Time travel. Data versioning enables rollbacks, historical audit trails and reproducible machine learning experiments.
• Upserts and deletes. Merge, update and delete operations.
• Streaming and batch unification. A Delta table is a batch table and also a streaming source and sink.
• Connectors. Apache Spark, Apache Flink and Apache Hive are among the engines that read and write Delta tables.

Spark provides APIs in Java, Scala, Python and R (the docs mark R as deprecated), plus Spark SQL, Structured Streaming and MLlib. Each Delta Lake release lists the Spark versions it supports; as of October 2026 the compatibility table shows Delta Lake 4.3.x with Spark 4.0.x and 4.1.x, so check it before you upgrade Spark.

What do you run yourself, and what does Databricks manage?

Area	Open-source Spark and Delta Lake	Databricks
Compute	You run a cluster manager: Standalone, Hadoop YARN or Kubernetes	Serverless compute is managed by Databricks; classic compute runs in your own cloud account and you configure it
Security and governance	Spark's security features such as authentication are not enabled by default, so you configure RPC authentication, encryption and web UI access	Unity Catalog governs data and AI assets: it enforces access control, tracks lineage and logs activity for auditing
Orchestration	You choose and run a separate orchestrator, for example Apache Airflow	Lakeflow Jobs provides orchestration, with triggers that include Scheduled, Table update, File arrival and Continuous
Declarative pipelines	Spark Declarative Pipelines, added to Apache Spark in 4.1.0	Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines on Databricks Runtime
Runtime and upgrades	You pick the Spark and Delta Lake versions	Databricks Runtime includes Apache Spark and adds components and updates; Photon is a Databricks-native engine compatible with Apache Spark APIs

For monitoring, open-source Spark gives you a web UI (port 4040 by default), event logging, a history server that you start yourself, a REST API under /api/v1 and a configurable metrics system.
One storage detail shows what self-managed means in practice. On S3, Delta Lake's default mode supports concurrent writes only from a single Spark driver, and the docs warn that concurrent writes from multiple drivers can lead to data loss. Multi-cluster writes need an extra setup that the docs call experimental.

When is open source enough, and when does a managed platform help?

Open source alone suits a team that already operates clusters or Kubernetes and wants to control every component and version. A managed platform suits a team that would rather not run cluster managers, security configuration, orchestration and monitoring itself, or that wants governance for data and AI assets in one place. Both paths can store tables in the Delta Lake format.

How does Databricks fit around open-source Spark and Delta?

Databricks Runtime "includes Apache Spark but also adds a number of components and updates". Notebooks provide real-time coauthoring and automatic versioning. The control plane, which contains the web application and backend services, is operated by Databricks in the Databricks account. For classic compute the resources are in your cloud account, and for serverless compute they run in a serverless compute plane.

FAQs

Can I run Databricks on my own servers?

Databricks operates the control plane, so you do not self-host it. The docs describe two compute planes: classic compute in your cloud account and serverless compute in a serverless compute plane. Spark and Delta Lake do not depend on that service. Spark runs on any platform that runs a supported version of Java, under Standalone, YARN or Kubernetes.

Does Delta Lake on Databricks match open-source Delta Lake?

Databricks includes non-breaking partial support for table features in all supported Databricks Runtime versions, while open source clients choose how to implement each feature. For table features with backported support, some operations in a Databricks Runtime version might not exist in the matching open source release, so test compatibility before you enable table features on production tables.

How do I bring existing Delta tables to Databricks?

Keep the files in your cloud object storage, configure a Unity Catalog external location for them, and create an external table with a LOCATION. DELTA is a supported external table format. Unity Catalog privileges are not enforced when users read the files from external systems.

Will my Spark code run on Databricks?

Photon is compatible with Apache Spark APIs, so it works with existing code with no changes required, and Databricks Runtime includes Apache Spark. Test your own jobs and check the runtime release notes for the Spark version a runtime bundles.