What is the difference between a table format and a file format?

Summary

• A file format defines how data is encoded inside a single file. Apache Parquet, for example, is "an open source, column-oriented data file format designed for efficient data storage and retrieval". Avro and ORC are other file formats.
• A table format manages a collection of files as one table. The Apache Iceberg spec describes its table format as "designed to manage a large, slow-changing collection of files in a distributed file system or key-value store as a table". Delta Lake extends Parquet data files with a transaction log, and Apache Hudi records table changes on a timeline.
• A table format tracks which files belong to the table at each version, so engines get table-level behavior such as atomic commits, snapshots and time travel with VERSION AS OF.
• Table formats sit on top of file formats. On Databricks, Delta Lake is the default table format, and Iceberg tables can be read and written.

What is the difference between a table format and a file format?

A file format answers "how is the data inside this one file encoded?" A table format answers "which files make up this table right now, and what changed?" Table formats keep that answer in metadata, so several engines can work with the same table. They do not replace file formats. Databricks documents that Delta Lake "extends Parquet data files with a file-based transaction log", and the Iceberg spec says it manages tables using "immutable file formats: Parquet, Avro, and ORC".

What does a file format define?

• Parquet is a columnar file format with "high performance compression and encoding schemes", supported "in many programming languages and analytics tools". Spark notes that Parquet files are self-describing, so the schema is preserved.
• ORC is, per the Spark docs, a columnar format with native zstd compression, bloom filters and columnar encryption.
• Avro files carry their schema: "A file has a schema, and all objects stored in the file must be written according to that schema, using binary encoding. Objects are stored in blocks that may be compressed."

When Spark writes Parquet you choose file-level settings such as the compression codec:

df.write.option("compression", "zstd").parquet("/tmp/orders_parquet")

What does a table format add?

• Delta Lake. The protocol says "all metadata for a Delta table is stored alongside the data". Commits are JSON files in the _delta_log directory, and readers use the transaction log "to selectively choose which data files to process". Each added file can carry statistics such as count and min/max values.
• Apache Iceberg. The spec says the format "tracks individual data files in a table instead of directories". A metadata file tracks the schema, partitioning and snapshots, and manifest files list the data files with each file's partition data and metrics. A commit is an atomic swap of the metadata file.
• Apache Hudi. Its timeline "is a log of all actions performed on the table at different instants", and Hudi keeps data "in open file formats".

Spark SQL can read older table versions in both Delta Lake (VERSION AS OF, TIMESTAMP AS OF) and Iceberg. Creating a table selects the table format:

CREATE TABLE orders_delta (id BIGINT, status STRING) USING DELTA;
CREATE TABLE prod.db.orders_iceberg (id BIGINT, status STRING) USING iceberg;
SELECT * FROM orders_delta VERSION AS OF 0;

Spark needs the table format's package and configuration first: the Delta Lake quickstart adds the delta-spark package with DeltaSparkSessionExtension and DeltaCatalog, and Iceberg requires you to "configure Spark catalogs".

How do the two compare?

Aspect	File format	Table format
Scope	The layout of one file	A collection of files treated as one table
Examples	Parquet, ORC, Avro	Delta Lake, Apache Iceberg, Apache Hudi
Metadata	Inside the file: the schema in Parquet and Avro files	At table level: _delta_log, Iceberg metadata files and manifests, the Hudi timeline
Changing data	Write new files	A commit that adds or removes files
Schema changes	Parquet supports schema merging across files	Iceberg schema updates are metadata changes, so no data files are rewritten

How does Databricks approach table formats?

Delta Lake is the default format for reading and writing data and tables on Databricks, and Iceberg tables can be read and written and interoperate with external Iceberg engines. Unity Catalog managed tables are the default and recommended table type for both. Creating an Iceberg table requires USING iceberg, otherwise Databricks creates a Delta Lake table. The docs describe both formats as Parquet data files plus a metadata layer: with Iceberg reads (UniForm), Databricks generates Iceberg metadata alongside the Delta Lake metadata "without rewriting the Parquet data files", and Iceberg clients get read-only access. The Unity Catalog Iceberg REST catalog lets supported clients read and write managed Iceberg tables.

FAQs

Is Parquet a table format?

No. Parquet is a data file format. Delta Lake builds on Parquet data files, and the Iceberg spec also defines Avro and ORC data files.

Can one table use more than one file format?

It depends on the table format. Delta Lake extends Parquet data files. In the Iceberg spec, each data file in a manifest records its own file_format (avro, orc, parquet or puffin).

Do I have to choose between Delta Lake and Iceberg on Databricks?

Not necessarily. Tables with Iceberg reads enabled can be read by Iceberg clients, and managed Iceberg tables are accessible to external clients through the REST catalog. Choose based on which engines must read and write the table.