How to convert Parquet datasets to Delta Lake or Iceberg

Summary

• CONVERT TO DELTA turns Parquet into a Delta Lake table in place. It lists the files, creates a transaction log that tracks them, infers the schema from the Parquet footers and collects statistics, so the data files stay where they are.
• Stop writers first. Any file the Delta log doesn't track is invisible and can be deleted by VACUUM, and every write after conversion has to go through Delta Lake.
• For Iceberg, Apache Iceberg's Spark procedures snapshot, migrate and add_files reuse your existing data files. On Databricks, native Iceberg means a managed Iceberg table in Unity Catalog that you can fill with a CTAS query, which writes new files.
• If Iceberg clients only need to read, keep the table in Delta Lake and turn on Iceberg reads (UniForm) instead of converting again.

How to convert Parquet datasets to Delta Lake or Iceberg

Run CONVERT TO DELTA to turn a Parquet directory into a Delta Lake table in place. For Iceberg, use Apache Iceberg's Spark procedures, which add your existing files to Iceberg metadata, or create a managed Iceberg table on Databricks and load it with CTAS. In every case, stop writers to the source first and send all later writes through the new format.

Which conversion route fits?

Goal	Route	Data files
Parquet directory or registered table to Delta Lake	CONVERT TO DELTA	Reused in place
Legacy Hive metastore managed Parquet table to a managed Unity Catalog Delta table	CTAS	Copied
Hive or Spark table to Iceberg, tested first	Iceberg snapshot, then migrate	Reused
Loose Parquet files into an existing Iceberg table	Iceberg add_files	Reused, not moved
Native Iceberg table in Unity Catalog	CREATE TABLE ... USING ICEBERG with CTAS	Copied
Iceberg clients reading a Delta table	Iceberg reads (UniForm)	No copy, Iceberg metadata is generated

How do you convert Parquet to Delta Lake?

CONVERT TO DELTA parquet.`s3://my-bucket/parquet-data`;
CONVERT TO DELTA parquet.`s3://my-bucket/path/to/table` PARTITIONED BY (date DATE);
CONVERT TO DELTA catalog_name.database_name.table_name PARTITIONED BY (date_updated DATE);

The first line converts an unpartitioned directory, the second a partitioned one, and the third a Parquet table registered in Unity Catalog. With a path, PARTITIONED BY is required for partitioned data. Tables in the Hive metastore load their partition specification from the metastore, but Unity Catalog external tables need it supplied. If the directory structure doesn't match the specification, the conversion aborts. NO STATISTICS finishes faster, and Databricks then recommends liquid clustering to reorganize the layout and generate statistics.
Unity Catalog supports the command for Parquet and Iceberg data in external locations. You need write access on the storage location and, to load the result as an external table, the CREATE EXTERNAL TABLE permission on the external location. Use CTAS for a legacy Hive metastore managed Parquet table. If several external tables share one directory, run CONVERT on each of them, because the others stop working once the directory becomes Delta Lake.

How do you convert Parquet to Iceberg?

Apache Iceberg's Spark procedures need a configured Iceberg catalog and run with CALL catalog_name.system.<procedure>:

Procedure	What it does	Watch for
snapshot	Creates a light-weight temporary copy of a Hive or Spark table for testing	Deleting from the original removes files the snapshot still uses
migrate	Replaces a Hive or Spark table with an Iceberg table loaded with the source's data files	Fails on bucketed tables or unsupported partition formats; the original is kept as table_BACKUP_ by default
add_files	Adds files to an existing Iceberg table without moving them; you create the table first	Schema isn't validated, and a later expire_snapshot can physically delete the files

CALL catalog_name.system.migrate('db.sample');

CALL spark_catalog.system.add_files(
  table => 'db.tbl',
  source_table => '`parquet`.`path/to/table`'
);

The Iceberg docs say not to use add_files when migrate or snapshot are possible.
On Databricks, Iceberg tables you create in Unity Catalog are managed Iceberg tables. Name the format, because Delta Lake is the default, and leave out LOCATION. Iceberg tables can't be created in the Hive metastore. Managed Iceberg tables need Unity Catalog, Databricks Runtime 16.4 LTS or above, serverless compute and predictive optimization.

CREATE TABLE main.default.events_iceberg
USING ICEBERG
AS SELECT * FROM read_files('s3://my-bucket/events/', format => 'parquet');

This copies the data into a new table, so it's a rewrite rather than a metadata-only conversion. Databricks doesn't support PARTITIONED BY on managed Iceberg tables, so use liquid clustering (CLUSTER BY) to optimize the layout instead.
To let Iceberg clients read an existing Delta table without a second copy, enable Iceberg reads. The table must be registered in Unity Catalog with column mapping enabled, and writes must use Databricks Runtime 14.3 LTS or above. The statement below is documented for Databricks Runtime 15.4 LTS and above.

ALTER TABLE table_name SET TBLPROPERTIES(
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg');

Iceberg client support is read-only, and Databricks generates the Iceberg metadata asynchronously after Delta Lake writes complete. Enabling Iceberg reads upgrades the writer protocol, so only clients that support the IcebergCompatV2 feature can write to the table afterward, and protocol upgrades can't be undone.

FAQs

Does conversion rewrite my data?

CONVERT TO DELTA works in place. Iceberg's snapshot, migrate and add_files also reuse the existing files. A CTAS into a managed Iceberg table writes a new copy.

What if writes continue during conversion?

Avoid updating or appending data files while it runs. CONVERT TO DELTA is a one-time conversion. For ongoing sync from a source that keeps receiving writes, the Databricks docs point to incremental clone, which is in Public Preview.

Can I undo a conversion?

Rehearse on a copy of the data first. Iceberg's snapshot leaves the source table untouched, and migrate keeps the original as a backup table by default.

Do I have to choose one format?

No. Databricks supports managed Iceberg tables in Unity Catalog, and Delta tables can expose Iceberg metadata for read-only Iceberg clients.