What does Databricks do in the OSS ecosystem?

Summary

• Databricks engineers are the original creators of Apache Spark, Delta Lake and MLflow, and Databricks also created Unity Catalog and the open Delta Sharing protocol. All of them are open source under the Apache 2.0 license.
• Databricks keeps contributing upstream. In June 2025 it donated Declarative Pipelines to Apache Spark, and the framework shipped as Spark Declarative Pipelines (SDP) in Spark 4.1.0.
• Its managed products build on these projects: Lakeflow Pipelines extends SDP, Databricks Connect is built on Spark Connect, and OpenSharing (formerly Delta Sharing) is Databricks' sharing platform. Some engines, such as Photon, are Databricks-only.

What does Databricks do in the OSS ecosystem?

Databricks plays three roles in open source. It created several core data and AI projects and still develops them, it contributes new components upstream, and it builds managed products on top of those projects. Its open-source page says "Databricks engineers are the original creators of some of the world's most popular open source data technologies." Each project below is published under an open-source license, so you can use it without Databricks.

Which open-source projects does Databricks work on?

Project	What it is	License	Databricks' role
Apache Spark	"A unified analytics engine for large-scale data processing"	Apache 2.0, Apache Software Foundation	Databricks engineers are its original creators
Delta Lake	"An open-source storage framework that enables building a Lakehouse architecture"	Apache 2.0, a Linux Foundation project	"Databricks originally developed the Delta Lake protocol and continues to actively contribute to the open source project"
Neon	"An open-source serverless Postgres database platform" that separates storage and compute	Apache 2.0	Joined Databricks in May 2025. Lakebase Postgres is also available on the Neon.com platform.
MLflow	"The open source AI engineering platform for agents, LLMs, and ML models"	Apache 2.0	Databricks engineers originated it
Unity Catalog	"Open, Multi-modal Catalog for Data & AI", with an OpenAPI specification, server and clients	Apache 2.0	Databricks created it
Delta Sharing	"An open protocol for secure data sharing"	Apache 2.0	Developed by Databricks
Redash	A browser-based tool to query, visualize and share data with SQL	BSD 2-Clause	Listed on Databricks' open-source page

Governance differs by project. Apache Spark belongs to the Apache Software Foundation. Delta Lake "is an independent open-source project and not controlled by any single company", and it joined the Linux Foundation Projects in 2019. Unity Catalog's open-source version offers "interoperability across lakehouse formats like Delta Lake, Apache Iceberg".

What has Databricks contributed upstream recently?

The largest recent contribution is Spark Declarative Pipelines. In June 2025, Databricks announced, "We're donating Declarative Pipelines... to Apache Spark." Apache Spark 4.1.0 then added SDP, "a new declarative framework where you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries."

How do Databricks products build on these projects?

Open project	Databricks product	What the Databricks docs say
Spark Declarative Pipelines	Lakeflow Pipelines	"Lakeflow pipelines extend and are interoperable with Spark Declarative Pipelines"
Spark Connect	Databricks Connect	"Databricks Connect is built on open-source Spark Connect"
Delta Lake	Default table format	"Delta Lake is the default format for all operations on Databricks"
The open Delta Sharing protocol	OpenSharing (formerly Delta Sharing)	"the secure data sharing platform in Databricks", and the basis for Databricks Marketplace and Clean Rooms
MLflow	Databricks-managed MLflow	"a fully managed and hosted version of MLflow"
Apache Iceberg	Managed Iceberg tables in Unity Catalog	The Iceberg REST Catalog API "supports read and write operations from external Iceberg engines"

Some parts of the platform are not open source. Photon is "the Databricks-native vectorized query engine", and Lakeflow Pipelines runs "on the performance-optimized Databricks Runtime".

How Databricks approaches open source

Databricks builds managed products on the open projects it created. Its open-source page says that because Databricks built these projects, "the platform stays closely aligned with the open source releases" on spark.apache.org, delta.io and mlflow.org. On table formats, Unity Catalog supports both Delta Lake and Apache Iceberg tables.

FAQs

Can I use Delta Lake without Databricks?

Yes. Delta Lake works with compute engines including Apache Spark, Apache Flink and Apache Hive, and it has APIs for Scala, Java, Rust and Python.

How does Databricks relate to Apache Iceberg?

Apache Iceberg is an Apache Software Foundation project, which its site describes as "a high-performance format for huge analytic tables." Databricks' open-source page lists it next to Delta Lake as a lakehouse format, and Unity Catalog supports it: "Iceberg tables that you create in Unity Catalog are managed Iceberg tables." Iceberg reads (UniForm) also let Iceberg clients read Delta Lake data "without rewriting files."

What is Omnigent?

Omnigent is an Apache 2.0 open-source "meta-harness" that gives coding agents "a common orchestration layer". Its README lists "Status: alpha". On Databricks, "Omnigent is in Beta", and the managed version adds a Databricks-operated server plus model access through the Foundation Model APIs and Unity Gateway.

Where are the source repositories?

Apache Spark is at github.com/apache/spark, Delta Lake at github.com/delta-io/delta, the Delta Sharing protocol at github.com/delta-io/delta-sharing, MLflow at github.com/mlflow/mlflow, Unity Catalog at github.com/unitycatalog/unitycatalog and Neon at github.com/neondatabase/neon.