What is schema evolution in Delta or Iceberg?

Summary

• Schema evolution means changing a table's schema, such as adding, renaming, dropping or widening columns, without rewriting the table or breaking existing queries. Delta Lake and Apache Iceberg both support it.
• Delta Lake validates every write against the table schema and raises an exception on a mismatch. You opt in per operation with the mergeSchema option, or with .withSchemaEvolution() on writes in Delta Lake 4.3 and above.
• Apache Iceberg tracks each column by a unique ID, so add, drop, rename, widen and reorder are metadata changes that don't rewrite data files. Safe widenings are int to bigint, float to double and decimal precision.
• In Delta Lake, renaming or dropping a column without a rewrite needs column mapping, and changing a type in place needs type widening, which is fully supported in Delta Lake 4.0 and above.

What is schema evolution in Delta or Iceberg?

Schema evolution is the ability to change a table's schema as your data changes, such as adding a column for a new event attribute or widening a type, without rewriting the table or breaking existing queries. Delta Lake and Apache Iceberg both support it, with different defaults.

What schema changes does each format support?

Change	Delta Lake	Apache Iceberg
Add a column	ALTER TABLE ... ADD COLUMNS, or automatically on write when schema evolution is on	ALTER TABLE ... ADD COLUMN, including nested fields
Rename a column	RENAME COLUMN without rewriting data requires column mapping	RENAME COLUMN on any field
Drop a column	A metadata-only DROP COLUMN requires column mapping	DROP COLUMN, including nested fields
Widen a type	ALTER COLUMN ... TYPE once type widening is enabled	ALTER COLUMN ... TYPE for safe promotions
Reorder columns	ALTER COLUMN with FIRST or AFTER	ALTER COLUMN with FIRST or AFTER

How does Delta Lake handle schema evolution?

By default, Delta Lake validates that the schema of the DataFrame being written is compatible with the table. A write with columns the table doesn't have, or with mismatched data types, raises an exception. Schema evolution is opt-in. When it's on, source columns missing from the table are appended to the end of their struct, and existing rows receive NULL values for the new columns.

# Delta Lake 4.3 and above
(spark.read.table("source_table")
    .write.mode("append")
    .withSchemaEvolution()
    .saveAsTable("target_table"))

# Earlier versions: the mergeSchema option
(spark.read.table("source_table")
    .write.option("mergeSchema", "true")
    .mode("append")
    .saveAsTable("target_table"))

Delta Lake 4.3 and above also accepts INSERT WITH SCHEMA EVOLUTION INTO target_table SELECT * FROM source_table. Setting spark.databricks.delta.schema.autoMerge.enabled turns evolution on for the whole session, which the docs don't recommend because it can lead to unintended schema changes.
For upserts, the merge builder's withSchemaEvolution(), added in Delta Lake 3.2, lets the target schema evolve from the source:

from delta.tables import DeltaTable

(DeltaTable.forName(spark, "target_table").alias("t")
    .merge(source_df.alias("s"), "t.id = s.id")
    .withSchemaEvolution()
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute())

Renames and drops need column mapping, which lets table columns and the underlying Parquet columns use different names. You enable it by setting delta.columnMapping.mode to name, which requires reader version 2 and writer version 5. The Delta docs label column mapping experimental with known limitations, and the protocol upgrade is irreversible.
Type changes need type widening. Set delta.enableTypeWidening to true, and ALTER TABLE <table_name> ALTER COLUMN <col_name> TYPE <new_type> updates the schema without rewriting the Parquet files. In Delta Lake 4.0 and above, supported changes include int to long, decimal or double, float to double, and date to timestampNTZ.

How does Apache Iceberg handle schema evolution?

Iceberg supports add, drop, rename, update (widen) and reorder, including fields in nested structs. These are metadata changes, so no data files need to be rewritten. Iceberg uses unique IDs to track each column. A new column gets a new ID, so existing data is never used by mistake, and dropping or renaming a column doesn't change the values in any other column.
In Spark, Iceberg allows these safe type updates: int to bigint, float to double, and decimal(P,S) to decimal(P2,S) when P2 > P. The scale can't change.

ALTER TABLE prod.db.sample ALTER COLUMN measurement TYPE double;
ALTER TABLE prod.db.sample RENAME COLUMN data TO payload;
ALTER TABLE prod.db.sample ADD COLUMN point.z double;
ALTER TABLE prod.db.sample DROP COLUMN id;

Iceberg can also evolve the schema on write. Set the table property write.spark.accept-any-schema to true and write with the mergeSchema option, as in data.writeTo("prod.db.sample").option("mergeSchema","true").append(). New source columns are added, and rows already in the table get NULL for them.
With a Hive catalog, operations that shift column positions, such as ADD COLUMN with FIRST or AFTER, fail, because the Hive Metastore validates column types by position.

Which changes still need a rewrite or extra care?

• Narrowing or incompatible type changes, such as string to int, aren't supported promotions in either format. In Delta Lake you rewrite the table with the overwriteSchema option.
• Changing decimal scale isn't one of Iceberg's safe updates.
• Making a nullable Iceberg column required isn't possible with SET NOT NULL, because Iceberg doesn't know whether existing data contains nulls.
• Streams reading a Delta table terminate when its schema is updated, and you must restart them.

How does Databricks handle schema evolution?

On Databricks, MERGE WITH SCHEMA EVOLUTION works in Databricks Runtime 15.4 LTS and above, and INSERT WITH SCHEMA EVOLUTION in Databricks Runtime 18.1 and above. Databricks recommends enabling schema evolution per write operation rather than through a Spark configuration. Dropping a column from metadata doesn't delete its data in files. To purge it, run REORG TABLE to rewrite the files, then VACUUM to physically delete the old files.

What else do people ask?

Can I rename a column?

In Iceberg, yes, for any field: ALTER TABLE prod.db.sample RENAME COLUMN data TO payload. A nested rename only renames the leaf field. In Delta Lake, enable column mapping first, or rewrite the table with overwriteSchema.

Is schema evolution on by default?

Not for Delta Lake writes, where schema validation rejects mismatched writes until you opt in. Iceberg applies DDL changes directly, but schema merge on write needs both the write.spark.accept-any-schema property and the mergeSchema option.