What's the vendor lock-in risk per Spark platform?

Summary

• Lock-in risk on a Spark platform is not one number. It comes from four layers you can check separately: the table format, the catalog that holds table metadata, the APIs your code calls, and the compute you run it on.
• Apache Spark, Delta Lake, Apache Iceberg and Apache Parquet are open source, and Spark SQL has run in ANSI mode by default since Spark 4.0, so jobs that stay on those formats and APIs have a path to other Spark deployments.
• The catalog and any platform-only table features are the layers to test first, because a second engine has to talk to them before it can read a file.
• On Databricks, managed tables are Delta Lake or Apache Iceberg, and Unity Catalog exposes an Iceberg REST catalog endpoint for external engines. Enabling Iceberg reads upgrades the Delta writer protocol, so check protocol versions before you rely on it.

What's the vendor lock-in risk per Spark platform?

Lock-in risk on a Spark platform is not one number. It comes from separate layers you can check one at a time: the table format, the catalog that holds table metadata, the APIs your code calls, and the compute you run it on. Apache Spark, Delta Lake, Apache Iceberg and Apache Parquet are open source. Look for a platform that keeps your data in those formats, in cloud storage you control, with your code on open Spark APIs. Test the catalog and any platform-only table features first: a second engine must talk to them before it can read a single file.

Which parts of the stack are open?

Vendor lock-in is the cost of moving your data and workloads somewhere else. Each layer has open building blocks you can hold a platform to:

Layer	Open building blocks	What to check
Table format	Delta Lake (Apache License 2.0), Apache Iceberg, Apache Parquet	Format and location of each table: files in storage you own
Catalog	Unity Catalog OSS (Apache 2.0, with an OpenAPI spec), Hive metastore, Iceberg REST catalog API	Can a second engine list and read tables through an open catalog API?
Query and code	Spark SQL (spark.sql.ansi.enabled is on by default since Spark 4.0), DataFrame API, Spark Connect	Does the job stay on open Spark APIs?
Compute	Spark standalone, Hadoop YARN or Kubernetes	Can you run the job on a cluster you operate?
Table protocol	Delta reader and writer protocol versions	Does a feature you enabled raise the versions other clients must support?

How can I test portability before I commit?

Run four checks on one representative table and one representative job.

1. Inventory the tables. DESCRIBE DETAIL on a Delta table returns its format, location, properties, minReaderVersion and minWriterVersion.
2. Read a table from a second engine. On Databricks, Unity Catalog serves an Iceberg REST catalog endpoint, and the docs show this open-source Spark configuration (angle brackets are placeholders; on AWS, set AWS_REGION first because the Iceberg client needs an explicit region):

from pyspark.sql import SparkSession

spark = (SparkSession.builder
  .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:<iceberg-version>,org.apache.iceberg:iceberg-aws-bundle:<iceberg-version>")
  .config("spark.sql.catalog.uc", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.uc.type", "rest")
  .config("spark.sql.catalog.uc.uri", "https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest")
  .config("spark.sql.catalog.uc.token", "<token>")
  .config("spark.sql.catalog.uc.warehouse", "<uc-catalog-name>")
  .getOrCreate())

3. Run the job on Spark you operate. Spark's cluster managers include its own standalone manager, Hadoop YARN and Kubernetes.
4. Check where the files live. A Unity Catalog external table stores its data files in cloud object storage within your cloud tenant. Dropping it removes the metadata but not the files.

How does Databricks approach portability?

• Open formats. Delta Lake is the default format on Databricks. The docs say Databricks originally developed the Delta Lake protocol, continues to contribute to the open source project, and that the transaction log has an open protocol any system can use to read it.
• Delta Lake or Apache Iceberg. Unity Catalog managed tables can be either, and they stay accessible to external clients through open APIs.
• Open catalog API. The Iceberg REST catalog endpoint lets clients such as Apache Spark and Apache Flink read and write Unity Catalog-registered Iceberg tables. It needs external data access on the metastore and the EXTERNAL USE SCHEMA privilege.
• Iceberg reads (UniForm). Delta tables generate Iceberg metadata, so Iceberg clients can read the data without rewriting files. The trade-off is documented: it adds the IcebergCompatV2 write protocol feature, which can affect compatibility with external Delta Lake clients, and protocol upgrades can't be undone.
• Open client protocol. Databricks Connect is built on open-source Spark Connect.

FAQs

If I use Delta Lake, am I locked in to Databricks?

Not by the format. Delta Lake is licensed under Apache License 2.0, and the project lists connectors for engines including Apache Spark, Apache Flink (preview, write) and Apache Hive (read). What can tie a table to specific clients is the protocol version and table features it uses, so check minReaderVersion and minWriterVersion before you depend on a feature.

Can I switch from one Spark platform to another?

Data and code portability are separate questions. Iceberg's multi-engine page says connectors for Spark, Flink and Hive are maintained in the main Iceberg repository. For code, the Databricks migration guide says that when you move from an on-premises cluster, custom cloud infrastructure or another enterprise Apache Spark offering, most workloads require only a few changes. Features that exist on one platform only are the part you would rewrite.

Should I use a Hive metastore or Unity Catalog for portability?

Open-source engines can reach both. Spark supports connecting to a persistent Hive metastore, and Unity Catalog OSS publishes an OpenAPI spec and implements the Iceberg REST catalog APIs. Choose the catalog all of your engines can reach, then confirm it by reading one table from a second engine.

What should I avoid to stay portable?

Avoid burying vendor-specific calls inside job logic. Keep Spark SQL and DataFrame code apart from vendor SDK calls and connection strings, so only a thin layer changes if you move.