Is Delta Lake proprietary or locked to Databricks?

Summary

• No. Delta Lake is open-source software under the Apache 2.0 license, and the project calls itself "an independent open-source project and not controlled by any single company." It has been a sub-project of the Linux Foundation Projects since 2019.
• The Delta transaction log protocol is public in the project's PROTOCOL.md. A table is Parquet data files plus JSON log entries in a _delta_log directory, so any engine that implements the protocol can read and write it.
• The project lists Apache Spark, PrestoDB, Apache Flink, Trino and Apache Hive among its engines, with APIs for Scala, Java, Rust, Ruby and Python.
• Photon, Predictive Optimization and automatic liquid clustering are Databricks platform features that operate on Delta tables. They aren't part of the open format.
• Portability depends on table features: a client must support every reader feature a table enables, so check your other engines before turning a new one on.

Is Delta Lake proprietary or locked to Databricks?

No. Delta Lake is an open-source project licensed under Apache 2.0 and hosted as a sub-project of the Linux Foundation Projects, and its protocol is a public specification that any engine can implement. A Delta table is a set of Parquet data files plus a JSON transaction log stored next to them, in the storage you point it at. Databricks originally developed the protocol and contributes to it, and its platform adds features that work on the same open format. None of them converts your data into a proprietary one.

What makes Delta Lake open

• License and governance. The code in the delta-io/delta GitHub repository is Apache-2.0 licensed, and the project has been part of the Linux Foundation Projects since 2019.
• A public protocol. The Delta transaction log protocol defines serializable ACID writes and snapshot isolation for reads. Log entries are "JSON files stored in the _delta_log directory at the root of the table," and because all metadata sits alongside the data, nobody needs a separate metastore just to read a table (catalog-managed tables are the exception).
• Independent implementations. Beyond the Apache Spark connector, Delta Kernel is "a set of libraries (Java and Rust) for building Delta connectors," and delta-rs, "a native Rust library for Delta Lake, with bindings into Python," lists integrations with pandas, Polars, DuckDB, DataFusion, Dask and Daft.

Open-source Delta Lake vs Delta Lake on Databricks: how they compare

Capability	Open-source Delta Lake	Delta Lake on Databricks
License and governance	Apache 2.0; a Linux Foundation Projects sub-project	The same open-source project, which Databricks originally developed and contributes to
Storage format	Parquet data files and a JSON transaction log in _delta_log	The same protocol, used as the storage layer for tables on Databricks
Engines and APIs	Spark, PrestoDB, Flink, Trino and Hive; Scala, Java, Rust, Ruby and Python APIs	Databricks Runtime and Photon, plus external clients through Unity Catalog's Unity REST API
Table features	Deletion vectors (2.3.0), liquid clustering (3.1.0), row tracking (3.2.0), type widening and variant (4.0.0)	Listed by the Databricks Runtime version each feature requires
Iceberg clients	Delta Universal Format (UniForm) lets Iceberg and Hudi clients read Delta tables, read-only	Iceberg reads (UniForm), with Unity Catalog able to act as the Iceberg catalog
Table maintenance	You run OPTIMIZE and VACUUM yourself	Predictive Optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables
Query engine	Whichever engine you choose	Photon, the Databricks-native vectorized engine, for supported operations

Which parts are Databricks-only?

The Databricks-specific pieces are platform services and engines, not the format:

• Photon. "The Databricks-native vectorized query engine," compatible with Apache Spark APIs. For supported operations it replaces the JVM-based Spark SQL execution engine with a native C++ runtime.
• Predictive Optimization. It runs maintenance automatically on Unity Catalog managed tables in Delta Lake and Iceberg formats.
• Automatic liquid clustering. CLUSTER BY AUTO needs Unity Catalog managed tables, Databricks Runtime 15.4 LTS or above, and Predictive Optimization to choose keys. Liquid clustering itself ships in open-source Delta Lake 3.1.0 and above.
• Lakeflow Pipelines. The Databricks product extends, and is interoperable with, open-source Spark Declarative Pipelines (SDP).

You don't need any of these to read a Delta table. Some table features do change what a reader must support, though. Databricks documents that tables with liquid clustering use writer version 7 and reader version 3, and that Delta clients that don't support those protocols can't read them. Before enabling a feature, check it against every engine that reads the table; DESCRIBE DETAIL shows a table's minReaderVersion and minWriterVersion.

How Databricks approaches Delta Lake

Databricks describes Delta Lake as "open source software" that's "fully compatible with Apache Spark APIs," and uses it as the foundation for tables on the platform. Unity Catalog governs Delta Lake and Apache Iceberg tables and opens them to outside engines: Delta clients use the Unity REST API, Iceberg clients use the Iceberg REST catalog, and credential vending lets external clients "inherit privileges on data governed by Databricks." Iceberg reads (UniForm) generate Iceberg metadata for Delta tables so Iceberg clients can read them without rewriting files. Unity Catalog is also available as open-source software, Apache 2.0 licensed and an LF AI & Data Sandbox project.

FAQs

Can I read Delta tables without Databricks?

Yes. Open-source Apache Spark reads them with the delta-spark package, the Delta docs cover connectors for Trino, Presto, Apache Flink and Apache Hive, and the deltalake Python package from delta-rs reads them directly.

Is Delta Lake on Databricks a different format?

No. Databricks uses the same Delta Lake protocol. What differs is the set of platform features around it, such as Photon and Predictive Optimization, and which table features each Databricks Runtime version supports.

Will enabling a new table feature lock my table to Databricks?

It ties the table to clients that support that feature, wherever they run. The Delta docs say "upgrading the read protocol of a table requires that all reader applications support the added features," so check your other engines first.