How do I create an Iceberg table with Spark?

Summary

• Register an Iceberg catalog first (spark.sql.catalog.<name> set to org.apache.iceberg.spark.SparkCatalog, plus a type), then run CREATE TABLE <catalog>.<db>.<table> (...) USING iceberg.
• PARTITIONED BY accepts transforms such as bucket(16, id) and day(ts) that create hidden partitions, so queries filter on the source column and Iceberg skips files that can't match.
• CREATE TABLE ... USING iceberg AS SELECT is atomic with SparkCatalog, and the new table doesn't inherit the source's partition spec or properties.
• Iceberg's documented defaults are 536870912 bytes (512 MB) for write.target-file-size-bytes and 134217728 bytes (128 MB) for read.split.target-size.
• On Databricks, the same statement in Unity Catalog creates a managed Iceberg table, but partitioning by expressions such as bucket() isn't supported there.

How do I create an Iceberg table with Spark?

Point Spark at an Iceberg catalog, then run CREATE TABLE ... USING iceberg with a table name inside that catalog. Iceberg plugs into Spark through the DataSourceV2 API, so the name has to resolve to an Iceberg catalog, not just any database. As of October 2026, Iceberg 1.12.0 is the latest release in the Iceberg docs, which list runtime jars for Spark 3.5, 4.0 and 4.1. This starts the Spark SQL shell with a path-based catalog named local, following the Iceberg quickstart:

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local.type=hadoop \
  --conf spark.sql.catalog.local.warehouse=$PWD/warehouse

Then create the table:

CREATE TABLE local.db.sample (
    id bigint NOT NULL COMMENT 'unique id',
    data string,
    category string,
    ts timestamp)
USING iceberg
PARTITIONED BY (bucket(16, id), days(ts), category);

For a shared catalog, set type to hive (a Hive metastore), rest (an Iceberg REST catalog) or jdbc, and set uri or warehouse as that catalog needs.

What does each clause of CREATE TABLE do?

Iceberg's create commands, including CTAS, support the full range of Spark create clauses:

Clause	What it does	Watch out for
USING iceberg	Creates the table in the Iceberg format	The name must resolve to an Iceberg catalog
PARTITIONED BY (...)	Sets the partition spec, including transforms	Transforms create hidden partitions, not extra columns
LOCATION '...'	Sets the table location	Use a fully qualified URI
COMMENT '...'	Sets the table description	Column comments go inline
TBLPROPERTIES ('key'='value')	Sets table configuration	See the defaults below
AS SELECT ...	Creates and fills the table in one statement	Atomic only with SparkCatalog
CREATE TABLE ... LIKE	Copies another table's definition	Not supported by Iceberg's Spark DDL

Which partition transforms can I use?

• year(ts), month(ts), day(ts) (or date(ts)) and hour(ts) (or date_hour(ts)) for timestamps. The older forms years, months, days and hours still work.
• bucket(N, col), which partitions by a hashed value mod N buckets.
• truncate(L, col), which truncates strings to length L and puts integers and longs into bins: truncate(10, i) produces partitions 0, 10, 20 and so on.

This is hidden partitioning. Iceberg derives the partition value from the source column and tracks the relationship, so readers filter on ts and Iceberg skips files for other days. The layout can change later. With the Iceberg SQL extensions enabled (the spark.sql.extensions setting above), adding a partition field is a metadata operation:

ALTER TABLE local.db.sample ADD PARTITION FIELD truncate(4, data);

CREATE TABLE local.db.sample_copy
USING iceberg
PARTITIONED BY (category)
TBLPROPERTIES ('read.split.target-size'='268435456')
AS SELECT * FROM local.db.sample;

Existing data keeps its old layout and new data uses the new spec, though dynamic partition overwrite behaves differently once the partitioning changes. The second statement is CTAS: the new table doesn't inherit the source's partition spec or properties, so declare them. From a DataFrame, the Iceberg docs use data.writeTo("local.db.sample_copy").create(), with partitionedBy and tableProperty for configuration.

Which table properties should I know at creation time?

Property	Documented default	What it controls
format-version	2 (since Iceberg 1.4.0)	The table spec version
write.format.default	parquet	Default file format: parquet, avro or orc
write.parquet.compression-codec	zstd	Parquet compression codec
write.target-file-size-bytes	536870912 (512 MB)	Target size of written data files
read.split.target-size	134217728 (128 MB)	Target size when combining input splits

To give every table in a catalog the same value, set spark.sql.catalog.<name>.table-default.<key>; a create statement can still override it.

How Databricks approaches Iceberg tables

Iceberg tables you create in Unity Catalog are managed Iceberg tables. The statement has the same shape, CREATE TABLE <catalog>.<schema>.<table> (...) USING iceberg, and USING iceberg is required because Databricks creates a Delta Lake table by default. You need a Unity Catalog-enabled workspace, Databricks Runtime 16.4 LTS or above, serverless compute (Databricks uses it to maintain the Iceberg metadata) and predictive optimization enabled for table maintenance.
Unity Catalog then handles snapshot expiration and file compaction. Some open-source habits don't carry over: partitioning by expressions such as days() or bucket() isn't supported for managed Iceberg tables, partition evolution works only from external engines through the Iceberg REST Catalog API, and branching and tagging aren't supported. Open-source Spark can still read and write these tables through that API with credential vending, and Databricks recommends Iceberg client version 1.9.2 and above.

FAQs

Should I use SparkCatalog or SparkSessionCatalog?

Use SparkCatalog for a dedicated Iceberg catalog (Hive, Hadoop, REST, JDBC and other types). Use SparkSessionCatalog as spark_catalog when Iceberg and non-Iceberg tables share one Hive metastore; it hands non-Iceberg tables to Spark's built-in catalog.

How do I avoid typing the catalog name every time?

Run USE local.db; and Spark resolves unqualified table names in that catalog and namespace. SHOW CURRENT NAMESPACE shows the current setting.

Does DROP TABLE delete the data files?

Not since Iceberg 0.14. DROP TABLE removes the table from the catalog, and DROP TABLE local.db.sample PURGE also deletes its contents.

How do I check the partitions and snapshots?

Query the metadata tables, such as local.db.sample.partitions (partitions with data or delete files in the current snapshot) or local.db.sample.snapshots.