Which open formats keep me from re-migrating later?

Summary

• Pair open data files with an open table format: Apache Parquet or Apache ORC for files, and Delta Lake, Apache Iceberg or Apache Hudi for tables. Each table format has a published specification, and Delta Lake keeps its data in Parquet files.
• Check portability against the engines you might move to. Each project documents its engines, and features beyond the baseline can fail on another engine.
• Metadata can be bridged. Iceberg clients can read Delta tables (UniForm in Delta Lake, Iceberg reads on Databricks), but only to read, and Apache XTable (Incubating) translates metadata between Hudi, Delta Lake and Iceberg without copying data files.
• On Databricks, Delta Lake is the default table format, Unity Catalog also supports managed Iceberg tables, and external clients connect through the Iceberg REST catalog.

Which open formats keep me from re-migrating later?

Two layers decide how portable your data is. The file format determines whether another engine can read your bytes. The table format determines whether another engine can understand your tables. Apache Parquet and Apache ORC are open file formats, and Delta Lake, Apache Iceberg and Apache Hudi are open table formats with documented specifications. Store data in an open file format, pick a table format your target engines already read, and register tables in a catalog with an open API.

How do file formats and table formats differ?

A file format defines how records are laid out inside a file. A table format tracks which files make up a table, so changes are atomic and earlier states can be queried. In Delta Lake that tracking is a transaction log in a _delta_log directory, and the data lives in Parquet files. In Iceberg, table state lives in metadata files, and every change swaps in a new metadata file atomically.

Format	Layer	Data files	Documented capabilities
Apache Parquet	File	Itself	Column-oriented, with compression and encoding schemes
Apache ORC	File	Itself	Columnar, with several compression codecs, column projection and lightweight indexes
Delta Lake	Table	Parquet	ACID transactions, time travel, schema evolution and enforcement
Apache Iceberg	Table	Parquet by default, or Avro or ORC	Schema and partition evolution, hidden partitioning, time travel and rollback
Apache Hudi	Table	Base files in columnar formats (Parquet by default), Avro, HFile or Lance	Transactions, upserts and deletes, indexes, clustering and compaction, concurrency control, time travel and incremental queries

What should I check before I commit to a format?

• Who else reads and writes it? Use each project's own list. Iceberg's home page names Spark, Flink, Hive and Impala among its engines, and DuckDB documents an iceberg extension. Delta Lake lists Spark, Flink and Hive plus APIs for Scala, Java, Rust and Python, and DuckDB documents a delta extension that reads and writes.
• Does the other reader support the features you use? Features beyond the baseline can break a cross-engine read. For Databricks Iceberg reads, deletion vectors aren't supported for Iceberg v2 reads, and Iceberg clients are read-only.
• Which catalog holds the tables? The Iceberg REST catalog is an open API specification, and Unity Catalog OSS implements the Iceberg REST Catalog APIs.
• Can the metadata be translated? In open-source Delta Lake, UniForm lets Iceberg and Hudi clients read Delta tables, read-only, with writes from Delta Lake 3.1 or above for Iceberg and 3.2 or above for Hudi. Apache XTable (Incubating) reads a table's existing metadata and writes metadata for other formats, so the same data files can be read as Delta, Hudi or Iceberg.

How does Databricks approach open formats?

Databricks says "Delta Lake is the default format for all operations on Databricks," and that the Delta Lake transaction log "has a well-defined open protocol that can be used by any system to read the log." For Iceberg clients it documents three paths:

• Iceberg reads (UniForm). Delta tables generate Iceberg metadata automatically, so Iceberg clients read the data without rewriting files. The table must be in Unity Catalog with column mapping enabled, minReaderVersion of at least 2 and minWriterVersion of at least 7, and writes must use Databricks Runtime 14.3 LTS or above.
• Managed Iceberg tables. Create them in Unity Catalog on Databricks Runtime 16.4 LTS or above, with serverless compute. Databricks supports Iceberg tables that use Parquet and versions 1, 2 and 3 of the Iceberg specification.
• Iceberg REST catalog. Unity Catalog serves /api/2.1/unity-catalog/iceberg-rest to clients such as Apache Spark, Apache Flink and PyIceberg. They can read and write managed Iceberg tables, but foreign Iceberg tables and Delta tables with Iceberg reads are read-only. It needs External data access on the metastore and the EXTERNAL USE SCHEMA privilege.

CREATE TABLE T(c1 INT) TBLPROPERTIES(
  'delta.columnMapping.mode' = 'id',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg');

FAQs

If I store data in Delta Lake, am I locked in to one vendor?

Delta Lake is open source, a sub-project of the Linux Foundation Projects, its protocol is public, and its data sits in Parquet files. The remaining risk is features: a table that relies on something another engine doesn't support is harder to move.

Should I pick Delta Lake or Iceberg?

Both have public specifications, run on several engines and are supported by Databricks. Decide on the engines and catalog you must support, the features you rely on, and whether your readers need to write. If Iceberg clients only need to read Delta tables, Iceberg reads works without rewriting files. If they must write, create managed Iceberg tables.

Can I keep plain Parquet files instead of a table format?

Yes, and any engine that reads Parquet can read the files. What you give up is what the table format adds: Delta Lake, for example, extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling.