Why is my catalog not finding my Iceberg table?

Summary

• Spark finds an Iceberg table only through a catalog registered with spark.sql.catalog.<name> properties, and it resolves your table name against the current catalog and namespace. Check the catalog, namespace and table in the identifier first.
• Then check the catalog settings: the implementation class (SparkCatalog or SparkSessionCatalog), type, uri and warehouse. The Hive-based catalog only loads Iceberg tables.
• Make sure the Iceberg Spark runtime JAR matches your Spark and Scala versions.
• The Iceberg catalog cache is on by default, and entries expire after 30 seconds.
• For a Databricks table read from an external engine, check the Iceberg REST endpoint, the warehouse catalog name, external data access, the EXTERNAL USE SCHEMA privilege, and that views cannot be read.

Why is my catalog not finding my Iceberg table?

Why can't Spark find my Iceberg table?

Spark finds an Iceberg table only through a catalog that you register with spark.sql.catalog.<name> properties, and it resolves the name in your query against the current catalog and namespace. Check the identifier first, then the catalog settings, then the runtime JAR, then the cache.

How does Spark resolve an Iceberg table name?

The Iceberg docs list how spark.table(table) and spark.read.format("iceberg").load(table) read the identifier:

Identifier	What Spark loads
file:///path/to/table	a HadoopTable at the given path
tablename	tablename in the current catalog and namespace
catalog.tablename	tablename from the specified catalog
namespace.tablename	namespace.tablename from the current catalog
catalog.namespace.tablename	namespace.tablename from the specified catalog
namespace1.namespace2.tablename	that table from the current catalog

The docs list these forms in order of priority, so a matching catalog name takes priority over any namespace resolution. Spark 3 tracks the current catalog and namespace. Run SHOW CURRENT NAMESPACE to see them and USE local.db; to change them.

How do I check what Spark can see?

spark.catalog.currentCatalog()
spark.catalog.listCatalogs()
spark.catalog.listDatabases()
spark.catalog.listTables("local.db")

The PySpark Catalog API returns the current catalog, the catalogs available in the session, the namespaces in the current catalog, and the tables in a namespace (the name may be qualified with a catalog). Compare the output with the identifier in your query. The catalog-aware calls need Spark 3.4 or later.

What does a working configuration look like?

The Iceberg Spark quickstart adds the runtime JAR, adds Iceberg support to Spark's built-in catalog, and creates a path-based catalog named local for tables under $PWD/warehouse:

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
  --conf spark.sql.catalog.spark_catalog.type=hive \
  --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local.type=hadoop \
  --conf spark.sql.catalog.local.warehouse=$PWD/warehouse

CREATE TABLE local.db.table (id bigint, data string) USING iceberg;
INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
SELECT count(1) as count, data FROM local.db.table GROUP BY data;

Which catalog settings hide a table?

Setting	What the Iceberg docs say
spark.sql.catalog.<name>	The implementation class. SparkCatalog supports Hive, Hadoop, REST, JDBC, Nessie and other catalogs. SparkSessionCatalog adds Iceberg support to Spark's built-in catalog and delegates non-Iceberg tables to it.
type	hive, hadoop, rest, jdbc, nessie or unset for a custom catalog. If type is null, catalog-impl must not be null.
uri	The Hive metastore URL for a hive catalog, or the REST URL for a rest catalog. For hive, omit it to use hive.metastore.uris from hive-site.xml.
warehouse	The base path for the warehouse directory.
default-namespace	The default current namespace for the catalog. The default value is default.
cache-enabled	Catalog cache, true by default. Entries expire after cache.expiration-interval-ms, which is 30 seconds by default.

Two more checks. The Hive-based catalog only loads Iceberg tables, so use a session catalog to load non-Iceberg tables from the same metastore. And Iceberg publishes one runtime JAR per Spark and Scala version, for example iceberg-spark-runtime-3.5_2.12. Besides vendor dependencies and storage bundles, the runtime JAR is the only addition the classpath needs.

Why can't my external engine find a Databricks table?

Unity Catalog on Databricks uses a three-level namespace, catalog.schema.object. An external Spark session reaches its Iceberg tables through the Iceberg REST catalog at /api/2.1/unity-catalog/iceberg-rest. If the table is not found, check:

• Catalog entry. Set type=rest, the endpoint uri, and warehouse to the Unity Catalog catalog name. Databricks says to configure each catalog separately.
• Access. External data access must be enabled on the metastore, and the principal needs the EXTERNAL USE SCHEMA privilege on the schema.
• Table type. Managed Iceberg tables, foreign Iceberg tables and Delta tables with Iceberg reads enabled are available. External Iceberg clients cannot access views.
• Freshness. Foreign Iceberg tables are not refreshed automatically. Run REFRESH FOREIGN TABLE to read the latest snapshot.

To test the endpoint without Spark, load the table with the REST API:

curl -X GET -H "Authorization: Bearer $OAUTH_TOKEN" -H "Accept: application/json" \
  https://<workspace-instance>/api/2.1/unity-catalog/iceberg-rest/v1/catalogs/<uc_catalog_name>/namespaces/<uc_schema_name>/tables/<uc_table_name>

FAQs

Can I use more than one Iceberg catalog in a session?

Yes. A catalog is created and named by adding a spark.sql.catalog.<catalog-name> property with an implementation class, and its other properties nest under that name.

How do I make Spark's built-in catalog load Iceberg tables?

Configure spark_catalog to use Iceberg's SparkSessionCatalog, as the quickstart does with type=hive. It is useful when spark_catalog must work with both Iceberg and non-Iceberg tables in the same metastore.

Can I read an Iceberg table without registering a catalog?

Yes, by path. The Iceberg docs say the identifier file:///path/to/table loads a HadoopTable at the given path when you call spark.read.format("iceberg").load(table).