What open-source projects did Databricks create? (Spark, Delta Lake, MLflow, Unity Catalog OSS, Delta Sharing)

Summary

• Databricks says it was founded in 2013 by the original creators of Apache Spark. Spark itself started as a research project at UC Berkeley's AMPLab in 2009 and has been an Apache Software Foundation project since 2013.
• Databricks originally developed Delta Lake, which joined the Linux Foundation in 2019 and describes itself as an independent open-source project that no single company controls.
• Databricks announced MLflow on June 5, 2018, and open sourced Unity Catalog in June 2024, where it is hosted by the LF AI & Data Foundation as a sandbox project.
• Delta Sharing is an open protocol for sharing datasets across computing platforms. Databricks announced it as an open source project on May 26, 2021, and the Databricks feature is now called OpenSharing.
• Databricks also donated Declarative Pipelines to Apache Spark, where it is Spark Declarative Pipelines.

What open-source projects did Databricks create?

Databricks originated Delta Lake, MLflow, Unity Catalog OSS and the Delta Sharing protocol. Apache Spark is a special case: it began as a UC Berkeley research project, and Databricks was founded by its original creators. Several of these projects now live under foundations rather than under Databricks.

Where did each project start, and where does it live today?

Project	Where it started	Home today	What it does
Apache Spark	UC Berkeley AMPLab research project in 2009, open sourced in early 2010	Apache Software Foundation since 2013	A unified analytics engine for large-scale data processing
Delta Lake	Originally developed by Databricks	Delta Lake Project, a Linux Foundation sub-project, since 2019	ACID transactions, scalable metadata handling and unified batch and streaming on data lakes
MLflow	Announced by Databricks on June 5, 2018	Backed by the Linux Foundation	Experiment tracking, a model registry, deployment, and tracing and evaluation for LLM applications
Unity Catalog OSS	Originally developed by Databricks, open sourced in June 2024	LF AI & Data Foundation sandbox project	A catalog for tables, files, functions and AI models
Delta Sharing	Announced by Databricks on May 26, 2021	Open protocol and reference server in the delta-io GitHub organization	A REST protocol for sharing datasets across computing platforms

Did Databricks create Apache Spark?

Spark started before Databricks existed. Spark's own history page says it began as a research project at the UC Berkeley AMPLab in 2009, was open sourced in early 2010 and moved to the Apache Software Foundation in 2013. Databricks says it was founded in 2013 by the original creators of Apache Spark. Databricks has since announced that it was donating Declarative Pipelines to Apache Spark, and Spark 4.1.0 added Spark Declarative Pipelines (SDP).

How do the open-source projects relate to Databricks products?

• Spark: Apache Spark powers Databricks compute clusters and SQL warehouses. Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines while running on Databricks Runtime.
• Delta Lake: it is the default table format on Databricks. Databricks originally developed the Delta Lake protocol and says it continues to contribute to the open-source project.
• MLflow: Databricks provides a fully managed and hosted version of MLflow.
• Unity Catalog: it is built into Databricks as the unified governance layer for data and AI, and it is also available as an open-source implementation.
• OpenSharing (formerly Delta Sharing): the Databricks sharing platform. The Databricks docs describe the OpenSharing protocol as developed by Databricks and also available as an open-source project.

FAQs

Can I use these projects without Databricks?

Yes. Spark runs on its standalone cluster manager, on Hadoop YARN or on Kubernetes. Delta Lake has connectors for Spark, Flink and Hive. The MLflow README lists self-hosting, and starts a local server with uvx mlflow server. The Unity Catalog OSS README shows how to run its server with bin/start-uc-server. The Delta Sharing repository includes a reference server.

How do the open-source projects differ from the Databricks products?

Feature sets differ. The AUTO CDC APIs in Lakeflow Pipelines, for example, are not supported by Apache Spark Declarative Pipelines. The Unity Catalog OSS README also warns that its APIs "are currently evolving and should not be assumed to be stable," so check each project's own documentation before you depend on a feature.

Is Unity Catalog OSS an Apache Software Foundation project?

No. It is licensed under Apache 2.0 but hosted by the LF AI & Data Foundation, part of the Linux Foundation.

Does Databricks still call it Delta Sharing?

The open protocol and its GitHub project are still named Delta Sharing. In Databricks the feature is now OpenSharing, and the original Delta Sharing announcement carries a note that Delta Sharing has evolved into OpenSharing.