Databricks + Apache Iceberg™: Managed and Foreign Tables in Unity Catalog

Summary

• Open table formats: Unity Catalog brings Apache Iceberg to the Databricks lakehouse with managed and foreign table support for fine-grained governance.
• Catalog interoperability: The platform eliminates data silos and simplifies access by unifying multiple Iceberg catalogs through a single point of access.
• Performance and governance: Fine-grained governance, predictive optimization, and open lakehouse architecture enable scalable, secure, and performant analytics.

Databricks + Apache Iceberg™: Managed and Foreign Tables in Unity Catalog

Talk by Jonathan Brito, Staff Product Manager, Databricks
Unity Catalog brings open, interoperable table formats to the heart of the Databricks Lakehouse. In this video, we’ll introduce new capabilities that apply fine-grained governance across all data and unify access across catalogs. Learn how Databricks is eliminating data silos, simplifying performance with predictive optimization and advancing an open lakehouse architecture.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to iceberg
• 1:25 Iceberg catalog challenges
• 3:31 Comparing catalog options
• 5:12 Key catalog criteria
• 7:38 Unity catalog overview
• 9:48 Interoperability benefits
• 12:48 Table optimizations
• 15:05 Lakehouse federation
• 16:56 Registering foreign catalogs
• 18:41 Cross-platform data access
• 21:00 Managed iceberg demo
• 24:36 Feature comparison summary
• 27:26 Future roadmap and conclusion

FAQs

What are managed and foreign Iceberg tables?

Managed tables are stored in a cloud location managed by Databricks, while foreign tables reference Iceberg data in external locations, giving organizations flexibility in data storage and governance.

How does Unity Catalog improve Iceberg adoption?

Unity Catalog brings open table formats to the Databricks lakehouse, enabling fine-grained governance across all data and unifying access across multiple Iceberg catalogs for simplified management.

What advantages do Iceberg tables provide for managing data lakes?

Apache Iceberg provides a table format that enables schema evolution, time-travel queries, and ACID transactions on data lakes, addressing limitations of traditional partition-based approaches. Iceberg's hidden partitioning and metadata tracking enable query optimization without requiring engineers to manually partition data, while atomic operations ensure data consistency during concurrent updates. Integration with the Databricks Data and AI platform gives organizations the benefits of a data warehouse with the flexibility and scale of a data lake.

What capabilities does Iceberg provide for working with data across platforms?

Iceberg is an open table format with implementations across multiple engines, allowing organizations to query the same data using Spark, Presto, Trino, Flink, and other tools without data duplication. Foreign tables in Databricks enable querying Iceberg tables stored in external systems while maintaining transactional guarantees and query optimization. This interoperability reduces data silos and vendor lock-in, enabling organizations to use best-of-breed tools while maintaining a unified data foundation.

How do Iceberg's versioning capabilities enable reproducible data analysis?

Iceberg maintains detailed metadata about each table version, enabling analysts and data scientists to query historical snapshots of data and reproduce analyses exactly as they existed at specific points in time. Time-travel queries eliminate the need to maintain separate copies of data for different analysis periods while providing audit trails for compliance. This capability is essential for validating model performance, debugging data issues, and supporting regulatory requirements that demand historical data access.

Full transcript

0:00 Introduction to iceberg

All right, folks. Looks like everyone's filling in. Uh, thanks for joining the session. Uh, first and foremost, I wanted to introduce you guys to my session. uh, product manager here at Data Bricks. I cover lakehouse storage. So for today's session, we're going to cover a handful of uh topics. In particular, we're going to cover a challenge that I see a lot of customers uh hitting today these days as they try to adopt an open lakehouse and that's choosing an iceberg catalog.
What we'll do in this session is cover three things. First, how to choose an iceberg catalog. We'll look at what in fact factors are really important uh to customers. And then second, we'll actually introduce managed and foreign icebro tables. So if you saw the keynote today, these are just some of the latest features that are hitting public preview in terms of our iceberg support that we're offering here at data bricks.
And then lastly, throughout all the presentation, we'll be comparing popular iceberg cataloges. So you'll see a sprinkle of Unity catalog as well as other cataloges and see how they stack up against our criteria for what we think it makes a great iceberg catalog. So to kick things off and to tell our story, we're starting with a protagonist. Uh that being. a platform architect and her organization is actually migrating to an open data lakehouse.
And Angela here has been tasked with leading that migration and making the right architectural decisions to set up her organization for a futurep proof uh architecture. Wow. Okay, great. So, what's our company? Our company is called Dunder Mifflin. Uh, few laughs, few laughs in the audience. Um, and our company, uh, over the years has been trying to solve a variety of different use cases and as a result, they've introduced a handful of different commercial and open source engines to their architecture.

1:25 Iceberg catalog challenges

So, they have data bricks, you have some snowflake, you have some Kafka. Um, and as part of their migration to an open data lakehouse, they actually last year spent a whole host of time looking at different formats they wanted to standardize on. And after careful deliberation, they decided that iceberg would be the direction they wanted to go in. So now it's up to make everything work. Uh, so this should be really easy.
Um, says. She looks at the iceberg ecosystem and as you can see here this whole host of connectors for iceberg has a really expansive ecosystem. Um so what can be difficult here? It's not quite that easy and the reason is because needs to choose an iceberg catalog. Uh for those who are new to iceberg, the iceberg catalog is a top level part of the uh table architecture and it actually has a lot of important tasks that it's responsible for.
Your iceberg catalog you use for things like uh creating tables, changing the name of a table, uh dropping tables, uh but most importantly it actually tracks the current metadata for a table. So all reads and writes to an iceberg table actually have to go through the iceberg catalog. So the iceberg catalog itself introduces a whole host of interoperability challenges if you don't make the right decision. So there's a lot of options on the market today that can choose from.
So let's help look at a few options to start. Let's look at Snowflake Horizon Catalog. Um Snowflake Horizon Catalog obviously has great interoperability with Snowflake itself. You can read and write from Snowflake's engine. And also snowflake horizon catalog can also federate to other iceberg catalogs particularly say glue and iceberg rest cataloges like unity catalog which provide interoperability with say athena or with data bricks.

3:31 Comparing catalog options

However, one big gap is that snowflake horizon catalog doesn't actually allow for rights from external engines. So if you're trying to use different open source engines say uh flink or kafka here at the bottom of the page um you're actually going to get stuck in certain cases. All you really have is reading interoperability with snowflake horizon catalog. All right, let's try Polaris. Another iceberg calendar that's out there.
Very similarly, a lot of inter wrapability with uh snowflake. So with Polaris, you can federate in snowflake hor you can mirror tables from Snowflake horizon catalog to Polaris and fate them back into Snowflake to get full birectional interoperability. as an iceberg rest catalog, you actually have reads and writes from a lot of the open source engines that you see at the bottom of the page. However, that's really where interoperability ends in that Polaris doesn't really offer the same level of interoperability with other cataloges it does with Snowflake Horizon catalog.
And as a result, you have a whole bunch of silos that you're seeing here on the page uh with BigQuery and uh say Athena and Redshift. All right, so this is a lot harder than we thought. Uh we started with a really great picture with all different iceberg connectors and now we have a bit of a shrunken ecosystem. So for, what should she be looking for when she chooses an iceberg catalog? So the first most important uh thing she should be looking at is fully open that catalog is fully open for external rights and reads.

5:12 Key catalog criteria

And this is really important because it provides the ability to choose whatever engine you want to use for the workload. This is kind of the dream of a lakehouse. It also provides a lot of interoperability in terms of future proofing your architecture. So if tomorrow you want to say switch from one engine to the other, you could easily swap back and forth. Second most important uh piece of an iceberg catalog is going to be advanced table optimizations.
So the iceberg catalog, not only does it manage your reads and writes a table, it's also going to op uh manage your optimizations. So ultimately the t the data layout and what sort of query performance as well as storage efficiency you get is going to be driven by what catalog you choose and of course you can do a lot of this work manually but that can be really difficult to do uh for folks who try to optimize iceberg tables if you're using partitioning you have to choose the right partition columns if you're running compaction figure out when exactly you want to run compaction uh to make sure you're getting the best bang for your buck.
Um, so there's a lot of things that go into this and if it's not auto automated, it's a lot of time consuming effort from your team that you can be running into. Lastly, uh, it's interoperated with external cataloges. So this is particularly important just how the ecosystem has evolved over time. Well, what we've seen is that a lot of different platforms actually require using their catalog. So for instance, if I want to write with Athena, I need to use blue catalog.
This results in architectures in which you have usually multiple cataloges at play. So having the ability to interoperate across cataloges becomes really important because it not only gives you the opportunity to choose the right engine that you want to use for your workload, but it also provides a single unified view of all your data because if you can federate from different cataloges and bring all that data into one, you can kind of create a catalog of cataloges um which is really good for data discovery as well as for governance.
Okay, we have our criteria. Let's uh go back to the drawing board and let's look at Unity catalog. So, you've heard a lot about this in the keynote, but Unity catalog is interesting. Two new features that are critical to meeting all these requirements. Uh first is manage iceberg tables. You can think of these as tables in which that are created within Unity catalog. Unique Cal is going to manage the reads and writes as well as the table optimizations.

7:38 Unity catalog overview

These tables are fully integrated with the data bricks platform and are fully open via open APIs. The second feature that's going to public preview are foreign iceberg tables. So foreign iceberg tables are basically contrast to manage iceberg tables. These are tables in which an external catalog so say HMS or glue or snowflake horizon is the catalog that owns that table that manages those reads and writes. These tables are going to be read only within uh data bricks.
Uh but we'll still have access to all the different governance features that you have within Unity catalog. All right. So let's deep dive into each one of these features. So introducing manage iceberg tables. As I mentioned, manage iceberg tables are fully open to the iceberg ecosystem. You can write and read from these tables from data bricks uh as well as from any external iceberg engine. And the way that we get interoperability with the iceberg e ecosystem is through the iceberg rest catalog API.
So for those who aren't familiar, the rest catalog is an open API spec that exists in the iceberg community for interacting with iceberg tables. Uh it basically decouples the catalog backend from the client. So it provides a really nice unified experience regardless of which engine you want to connect to whatever backend. Uni catalog was actually an early adopter of the ISER rest catalog. We actually launched read support for the ICER rest catalog back in 2023.
Uh these new features are basically implementing the right side of those APIs. So through these write APIs, you virtually get compatibility with almost any engine that you would want to use in the iceberg uh ecosystem. So think the Apache uh Spark, Trino, Flink, etc. All right. So let's consider an example and in particular try to determine why this is so important why this is becomes really uh key for customers. So going back to suppose if wanted to write from Snowflake um sets up Soul Rising catalog writes it using uh Snowflake engine but then tomorrow uh her business needs change and she decided hey I actually want to migrate away from Snowflake Engine and use open source spark.

9:48 Interoperability benefits

So the trick uh with this is can she actually do this migration? How easy or how hard would it actually be? It actually is a lot harder than you'd think. And the reason is because because open source spark cannot write to the same table as uh that snowflake is writing to. It not only is a migration of the engine, it's also a migration of the catalog. So for her to get open access with open source spark, she needs to change from snowflake horizon catalog to a whole different catalog, which if you don't have the proper tooling, this can introduce a lot more work to the migration.
This is recreating new policies in your other catalog. This is potentially redirecting downstream workloads. Um, a whole host of things that may not necessarily have been part of her original plan in terms of uh the migration. Okay. Now, let's see if we started out with this exact same example, but let's swap out Snowflake. Let's start instead with data bricks and unique catalog. So in this case you have data bricks writing to table one uh using UC and the nice thing about this approach is that it becomes really easy to swap in data bricks for say open source spark uh as needed and the reason is because they can write to the same table.
So this has this basically gives you the benefits of having no bender lock in all from Unity catalog. Cool. So the fact that now you can write uh iceberg with data bricks, it actually unlocks data bricks uh industryleading performance uh for ETL on iceberg workloads. So what you're seeing here is actually a benchmark that compares data brick SQL to snowflake's gen 2 warehouses and it just compares uh your ETL TCO as your data scales.
And as you're seeing uh from left to right on this slide, data bricks does a really great job of efficiently managing your cost as your data grows and grows in size. While in the case of Snowflake, it actually grows exponentially where at scale you're looking at about a 15x difference in terms of uh your ETL cost. So for folks who are playing to standardize an iceberg, this is a huge uh boon in terms of being able to benefit from these uh TCO savings.

12:48 Table optimizations

All right. Now, our second criteria was advanced table optimizations. Let's look at how Unity catalog stacks up. So, Unity catalog actually offers a whole host of uh automated optimizations through a feature called predictive optimization. So, predict optimization takes a lot of the toil of managing your tables out of the hands of your team to unlock you to do different more highv value uh activities. What pre- optimization does is actually does this really intelligently.
So it analyzes your query patterns and determines the best tables to actually optimize to make sure you're maximizing your ROI. Your manageport tables also uh can leverage liquid clustering. So for folks who are less familiar, liquid clustering um it basically replaces partitioning and zordering um where instead of having to go out and choose your different partition columns uh lib clustering will handle allow you to change your uh allow you to change your clustering cleaves without actually rewriting your data.
So ultimately speaking, you get incremental clustering that is automated by PR optimization. As you're seeing here, quite the benefit in terms of the performance. Okay. And one nice thing about all of this is that these optimizations are actually open to any engine that you plan to use. Um yeah, they're actually open to any engine that you plan to use. So if you're tomorrow decide hey I actually don't want to u use data bricks and I'm optimizing these tables with liquid clustering um any engine that's reading these tables would still benefit.
So for instance in the case of liquid clustering one of the reasons why this works is because with liquid clustering you're actually optimizing the parket files themselves. All those optimizations are then reflected in your iceberg metadata. So any engine that can speak iceberg when they go in to read these tables will benefit uh from these optimizations. Let me fix one thing really quickly. All right. And our last criteria was interoperability with other iceberg cataloges.

15:05 Lakehouse federation

So the way that Unity catalog solves this problem is through Lakehouse Federation. So how does Lakeos federation work? There's really two ways that we connect to external data sources with Lakehouse Federation. First is database federation. So with database federation, you're connecting to um databases that you're seeing on the bottom left hand side of this page. Um and what you're doing there is that when you run a query within data bricks, uh it actually pushes the query down to that external database to run the query.
Ultimately speaking, this is really useful if you're trying to access data in place when that data is stored in a proprietary format. The other way that lakehouse federation works is through what we call catalog federation. This is a bit different in contrasting to database federation where with catalog federation what you're actually doing is uh going directly to object store to reach these tables. So this is great if you're storing your data in an open table format like iceberg because ultimately speaking uh you actually can have a lot more performing queries and a lot of more cost savings because when you run that query using catalog federation it's only using data bricks compute rather than in the case of database federation you need both engines at play and your iceberg tables uh are all now accessible using catalog federation and the nice thing about that is once your iceberg tables have been registered within unity catalog Um, you can even though they're not managed by UC, you can still leverage all of UC's advanced features, things like fine grand access controls, lineage, auditing, all on these tables.
Cool. So, how does this all work? So, this you're seeing in this diagram uh an example of how to register a foreign catalog within UC. And the first step is ultimately creating a connection. So this is creating a connection object that links unique catalog to your foreign catalog. Uh today we support HMS, Glue and Snowflake Horizon for this sort of catalog federation approach. From there you do what's called creating a foreign catalog within UC.

16:56 Registering foreign catalogs

This is basically mounting that foreign catalog within Unity catalog. uh from so if you were to look say for instance in your catalog explorer you would see your UC meta store and then underneath that you'd have your uh foreign catalog listed out and then underneath that you see all your schemas and all your tables and this works across iceber tables this cover is actually all your table that you could have uh delta tables parquet CSV etc everything becomes uh unified and you see another great benefit of this approach is that ultimately Generally speaking, when you read the table, you're guaranteed freshness.
So in the case of an iceberg table, what's actually happening at query time is that UC will ping the other catalog for the pointer to the latest iceberg metadata JSON file. From there has basically everything it needs to run the query directly against object storage. So you're guaranteed freshness and all that is on automatically behind the scenes by UC. And in this case, you can see here that all of your existing workloads remain under uninterrupted.
So this is a great actual first step if you're trying to eventually migrate your tables to UC. This can allow you to migrate readers over to UC before you actually migrate over writers as well. Okay,. Now we're going to look at how we can use federation again to another solve uh another common interoperability challenge I see with customers which is you have customers who are using snowflake and they're using data bricks and they want to make they want to write from both those engines but then make everything accessible to both engines.

18:41 Cross-platform data access

So write from both read everything from everywhere. So to do that you can use federation um from data bricks to snowflake uh you can use lakehouse federation to register a snowflake horizon catalog within UC once you create external locations for all the iceberg tables that are in your federated catalog UC will know to go to object storage to read those tables in the reverse direction snowflake actually offers what they call a catalog integration works very similar to lakehouse federation and that ultimately you're doing is you're registering unique catalog within Snowflake as an external catalog.
From there, you can query these tables uh within Snowflake. And what's nice about this architecture is it's very flexible. Um so if ultimately tomorrow you said, hey, I want to use an open source iceberg engine. Uh as we talked about earlier, you actually do have support for the iceberg rest catalog within UC. So you could just connect uh say in this case trino to uni catalog's iceberg rest catalog API to write or read um to UC and beyond that all the tables that we've federated from snowflake horizon catalog are available in UC so can actually read your snowflake manage iceberg tables from unity catalog and again that's all with unified governance but things could be even simpler um last week at snowflake summit it.
Snowflake actually announced that they'd be adding support for writing to external iceberg uh rest cataloges of which Unity catalog is one. And with now the fact that we have support for writes to Unity catalog, snowflake in this case could actually just write directly to UC. So creating a UC managed iceberg table from Snowflake. So your architecture becomes a lot simpler and a lot fewer hops. Okay, so let's go to a demo.

21:00 Managed iceberg demo

Um, what I'm going to do here is show you guys a notebook in terms of how to use uh manage iceberg tables. Um, creating a manage ice table is incredibly easy. Uh, this is a notebook in data bricks and I've already have a bunch of things set up. Um, as you can see here, the way to create a manage iceberg table in data bricks is really just to append this using iceberg to uh your create st your create statement. So if I run this uh this will take a second to run.
Um as you can see here what this notebook is trying to do is actually it's tracking user impressions for our company Dunder Mifflin. Um so we have two columns uh the user profile which is going to be different users are actually interacting with the website and then the actual impressions that are uh hitting that website. Okay, looks like that's done. I can then from there insert data into my table. Uh this is an example just some dummy data I put together that shows a dummy user and a lot of impressions that she's making to the website.
Um and then you can query the table within data bricks. Okay, cool. And then you can see the result here. Uh so if you were paying close attention, you can see here that I'm actually storing uh a JSON as actually a string. Um, and this actually isn't really great uh for a lot of folks who have who I see who are laughing in the audience. If you try to query this uh particularly querying individual elements in that JSON, your query performance can really suffer.
Um, alternatively, if I wanted to, I could potentially flatten out this table and basically fan out a whole bunch of different columns. That would improve my query performance, but then I end up with a bunch of whole a bunch of empty uh rows and columns in my table. So, that's not great either. Uh iceberg actually has a solution for this uh which they're bringing uh which actually just got adopted by iceberg community and iceberg v3 and it's variant data type.
So we're actually going to have a private preview for uh iceberg v3 including variant data type next month. And in this demo I'm actually going to show you how it works. So how we can do this is if I scroll down and then open up this. So you can see here I'm going to create a new table using iceberg v3 uh version 3 within data bricks and then from and I'll see task from the original table but in this time I'm actually going to parse the JSON so I can store this string uh as a variant data type.
Just let that run for a second. Cool. Looks like that worked. And then if I try to query the table, you'll see here now that it's stored as a variant data type. So now this is an object rather than a string. And this has a lot of great performance benefits because if I decided, hey, I actually want to just query uh say the user ID column uh element. I could run this query and you can see in less than a second I got that result.
So I'll be coming to private preview. uh and um so definitely check out that and reach out to your uh account rep if you want to get part of that private preview. Let me switch back to my presentation. All right, now bringing everything together. Um this slide here is comparing Unity catalog to other popular iceberg cataloges across the metrics that we were discussing throughout um all this presentation. So external access being able to read and write from external engines, table optimizations, there's a variety of different optimizations that exist in the iceberg community. uh storage optimizations like snapshot exploration or file cleanup, file compaction, familiar I imagine most folks and then obviously clustering and then the federation of external cataloges and as you can see here Unity catalog is checking all those boxes for while the other cataloges that we have pictured here have limitations that you need to consider.

24:36 Feature comparison summary

Also one important thing to note here is that these iceberg tables are going to be adding a lot of great functionality that is popular within the open source iceberg community. Uh in particular adding support for things like hidden partitioning partition evolution. So we actually do this in a really creative way uh in which it actually allows you to use liquid clustering from an external iceberg engine. So what that means is if you open up the iceberg rest catalog and you're using an external iceberg engine, if you create a table with a partition column, we actually interpret that partition column as a clustering key.
So then under the hood, you can actually use look clustering from outside of data bricks with an iceberg engine. And then you can use partition evolution to actually evolve those uh to change those clustering keys because every time you change a partition using the iceberg rest catalog will translate that into uh changes to clustering keys. So this really unlocks liquid clustering for again the entire iceberg ecosystem.
And then lastly with iceberg v3 you just saw the demo a second ago. Uh these are some of the features I'll be coming to manage iceberg tables and as part of this private preview in particular deletion vectors which for those who aren't familiar this is bringing sort of merge on read semantic toolles um this really can greatly improve your right performance row ids these are really important for incremental processing uh in particular with row ids and deletion vectors you actually can unlock row level concurrency so that's one thing we'll be bringing to these manage ice tables as well as part of this preview because through that you actually can reduce the chances of having conflicts uh as you're writing to your various manage iceberg tables variant we talked already a bit about in terms of the advantages you get from that in terms of processing um semiructured data and then write defaults is another great feature I'd be bringing to these tables so it's been a long journey uh we have had these features in private preview and these are some actually the most well adopted private preview features that we've had here at data bricks.

27:26 Future roadmap and conclusion

So it's been really fun leading the team uh to bring these features to public preview and as I mentioned manage tables and manage ice tables and for tables are fully available now uh for public preview. check out the documentation uh to start testing. And beyond data bricks, um we actually did a lot of work leading into this launch to get a lot of various partners uh involved uh particularly folks trying to land iceberg tables uh with unity catalog.
So here featured is a quote from Red Panda who actually launched an integration as well with our um Uni Calig iceberg managed tables which allows you to basically take data from Kafka topics and land them directly into Uni Calig managed iceberg tables and these optimizations benefit from all these tables benefit from all the optimizations I discussed earlier. So you'll have liquid clustering on these tables. So it's a really great win for uh for really customers across the board.
Cool. And then closing things out in terms of the full picture. Uh this kind of brings everything together. So as you can see here, uh lake the unity catalog is really provides a unified governance solution uh for really everything you can have in a lakehouse. So starting at the storage layer, we obviously have support for Delta Lake. With these announcements, you have full support for iceberg as well in unity catalog as well as any of the legacy formats you may have in a lakehouse.
For a lot of folks, we see they still have some parquet. They still have some CSV tables, JSON and avoided within Unity catalog. Moving a layer up, uh we go a lot further than just the tables within Unity catalog. It's a really unified governance solution for everything in a lakehouse. So you have AI models, files, notebooks, dashboards, all of that can be governed by Unity catalog. In terms of features, uh we don't just stop at the traditional access controls and auditing.
Um Unity catalog actually provides a lot of other features, advanced features, uh that you can bring to these various tables. So lineage, cost controls, quality monitoring and this is all accessible to really any engine that you want to use in the lakehouse through the uni rest or ice APIs as well as with any external catalog through federation. So that's my talk. All right, thank you folks.

Learn more about the Databricks Data and AI platform.