Apache Iceberg V3 on Databricks: From Ingestion to Analytics

Summary

• VARIANT data type: Iceberg V3 introduces native support for semi-structured JSON data without requiring a fixed schema upfront.
• Row-level concurrency: Multiple writers can update the same table simultaneously with zero lock contention, enabling concurrent data corrections and pipeline ingestion.
• Cross-platform access: Query the same governed data from Databricks or vanilla open-source Spark via the Iceberg REST Catalog with Unity Catalog governance.

Apache Iceberg V3 on Databricks: From Ingestion to Analytics

See what's possible with Iceberg V3 on Databricks — end to end. We'll ingest semi-structured JSON into an Iceberg V3 table using the new VARIANT column type, run concurrent writes with zero lock contention thanks to row-level concurrency, power a live AI/BI dashboard with incremental materialized views, and query the same data from vanilla open-source Spark via the Iceberg REST Catalog. No vendor lock-in, no data copying — just open formats governed by Unity Catalog.

Chapters

• 0:00 Ingesting raw feedback data
• 0:46 Querying variant data
• 1:55 Enabling row-level concurrency
• 3:01 Automating analytics dashboards
• 3:40 Cross-platform data access
• 5:15 Recap and key takeaways

FAQs

What is the VARIANT column type in Iceberg V3?

VARIANT is a new Iceberg V3 column type that stores semi-structured JSON data natively without flattening into a rigid schema, allowing different records to have different field combinations and shapes.

How does row-level concurrency work in Iceberg V3?

Row-level concurrency allows concurrent reads and writes on different rows simultaneously with zero lock contention, enabling a data steward to correct existing records while pipelines insert new data in parallel.

does Iceberg REST catalog support credential vending and how does Unity Catalog implement it

This video explains that the Iceberg REST Catalog enables credential vending—the system securely distributes temporary, scoped credentials for accessing data without exposing long-lived credentials. This is critical for multi-tenant systems and secure data sharing. Unity Catalog on Databricks implements credential vending natively, managing short-lived tokens for table access. By combining Iceberg REST Catalog with Unity Catalog governance, organizations can securely delegate data access to external systems, partners, and AI agents without sharing static credentials, improving both security and auditability.

What is the Iceberg REST Catalog (IRC)?

The Iceberg REST Catalog is a standardized API for managing table metadata in Apache Iceberg format without requiring a central Hive metastore. This video explains that the REST Catalog enables truly cloud-native architecture: table metadata is accessible via HTTP, enabling distributed systems and multi-cloud setups. The REST Catalog also supports credential vending—providing temporary, scoped access credentials—making it ideal for secure data sharing and multi-tenant systems. Databricks Unity Catalog extends this with additional governance, enabling fine-grained access control on top of Iceberg tables.

What is Iceberg v3, and what does it add?

Iceberg v3 introduces several important features including row-level deletes (efficient updates without full table rewrites), equality delete filters (more efficient data removal), and improved positional deletes. This video highlights that v3 makes Iceberg more suitable for operational workloads with frequent updates, not just append-heavy analytics. Row-level deletes enable use cases like GDPR right-to-be-forgotten, efficient soft deletes for SCD Type 2 implementations, and mutable feature stores for AI. These capabilities narrow the gap between Iceberg and update-heavy databases, making it practical for broader use cases.

Full transcript

0:00 Ingesting raw feedback data

In our scenario, Lumina Technologies runs a training program. They collect session feedback from attendees, including free-form, semi-structured review data that doesn't fit neatly into a fixed schema. They need to ingest this data incrementally, power analytics dashboards, and prove that the data is accessible from any engine, not just Databricks. Feedback is arriving as JSON files in an S3 landing page. Each file contains a batch of submissions.

And here's the interesting part, the review data in each submission has a different structure. Some have text comments, some have sub-ratings, some have device info. It's genuinely semi-structured. We create an Iceberg V3 table. Notice the variant column type. This is new in Iceberg V3, and it lets us store this heterogeneous data natively without flattening it into a rigid schema up front. Autoloader picks up the files and writes them to our Iceberg table.

0:46 Querying variant data

It handles the file discovery automatically. Now, the payoff. We can query into the variant column using dot notation. So, review_data. comments gives us the text comments. reaches into nested structures. So, we can cast these to int and use them like any other column. We can look at review data tags, and this accesses array elements. Different records have different shapes, and that's fine. Variant handles it all natively.
But look at this. Each record has a different combination of fields. Some have comments, some have sub-ratings, some have device info. Variant stores all of it without forcing a common schema. Before we move on, I want to show you something. Look at these rows. Ratings of 7, 8, 9 and on a 1 to 5 scale, a legacy integration sent these in on a 1 to 10 scale. A data steward needs to fix this, but the pipeline has new feedback to load, too.

1:55 Enabling row-level concurrency

In a traditional system, you'd have to choose fix the data first, then load the new batch or vice versa. One blocks the other. Let's see what happens with Iceberg V3. I've kicked off both at the same time. 30 new rows being inserted and a merge correcting the bad ratings on existing rows. Both succeeded. 180 rows total. Our original 150 plus the new 30 ones and zero rows with ratings above five. The correction and the insert ran concurrently touching different rows with no lock contention.
This is row level concurrency in Iceberg V3. Writers don't block writers. Your data steward can run corrections while the pipeline is loading the next batch. So, now let's create a materialized view that aggregates our feedback data. So, average ratings, uh submission counts and trends over time. This materialized view is powering an AI BI dashboard. So, let me switch over to it. You can see the charts reflecting our current data.

3:01 Automating analytics dashboards

Okay, so let's ingest one more batch. I'm going to refresh the And now look at the dashboard. The numbers have updated automatically. The view reflects the latest data without any manual intervention. New data in, fresh insights out. All right, so now here's where it gets interesting. What if your team has existing Spark workloads running outside Databricks? Maybe on EMR, on prem or just a local development environment, can they access this same data?

3:40 Cross-platform data access

So, I'm going to spin up a completely vanilla open source Spark session. No Databricks runtime, no proprietary libraries, just plain Apache Spark with the Iceberg runtime jar. The only thing I need is the Iceberg rest catalog endpoint from our Databricks workspace. First, we're authenticating via OAuth 2 with a service principal. Unity Catalog validates the credentials and sets up the session. Then, we can see the enablement schema and all its tables, just like we would inside Databricks.
Here's the same aggregation query our analyst run on Databricks. Same results, same data, running on vanilla open source Spark. And we can read individual rows, too. What's happening under the hood is credential vending. Unity Catalog is issuing short-lived S3 credentials scoped to exactly the tables this session is allowed to see. So, no S3 keys to manage, no storage credentials to configure on the client side. This is the interoperability promise of Iceberg on Databricks.
Unity Catalog is the governance layer. The data lives in open source Iceberg format, and any engine that speaks the Iceberg rest catalog protocol can access it. So, to recap, we went from raw JSON files in S3 to a live analytics dashboard powered by incremental materialized views with semi-structured data handled natively by variant, concurrent reads and writes thanks to row-level concurrency, and cross-platform access from vanilla open source Spark via the Iceberg rest catalog.

5:15 Recap and key takeaways

No vendor lock-in, no data copying, just open formats governed by Unity Catalog. That's Iceberg V3 on Databricks.

Learn more about the Databricks Data and AI platform.