Get Data Into Databricks - Databricks Uniform Iceberg Integration

Summary

• Iceberg Interop: Convert existing Databricks tables to Apache Iceberg format to consume them in any Iceberg-compatible engine.
• Managed Tables in Iceberg: Create new Iceberg tables directly in Databricks with full compatibility for reads and writes across engines.
• Foreign Catalog Support: Connect to existing Iceberg data from external sources and federate it for consumption in Databricks.

Get Data Into Databricks - Databricks Uniform Iceberg Integration

Try Databricks today: https://dbricks.co/3EAWLK6.
Discover the power of Apache Iceberg in Databricks
Explore three key use cases:

• Iceberg Interop: Seamlessly add Iceberg to existing tables, and consume data both inside and outside Databricks.
• Managed Iceberg: Create and manage Iceberg tables effortlessly, with full compatibility inside and outside Databricks.
• Foreign Iceberg: Federate existing Iceberg tables, enabling consumption across platforms.

Simplify your data workflows while replatforming with these bite-sized examples of Iceberg in action!

Chapters

• 0:00 Introduction to Iceberg
• 1:19 Iceberg interop basics
• 6:30 Managed Iceberg tables
• 10:58 Foreign catalog integration

FAQs

What are the three main Iceberg integration approaches in Databricks?

Iceberg Interop allows you to convert existing Databricks tables to Iceberg format for multi-engine access. Managed Iceberg lets you create native Iceberg tables with full read/write support. Foreign Iceberg allows you to connect to external Iceberg tables.

Can I query Iceberg tables outside of Databricks?

Yes. With Iceberg Interop and Managed Iceberg, you can consume tables using any Iceberg-compatible engine, not just Databricks. This video uses PyIceberg as an example for external querying.

How does Unity Catalog support Apache Iceberg tables across systems?

Unity Catalog provides unified governance for Iceberg tables through managed Iceberg support, enabling fine-grained access controls and data lineage tracking for tables in Databricks lakehouse. The integration handles both managed Iceberg tables stored in Databricks and foreign Iceberg tables in external catalogs, providing consistent governance policies across hybrid data environments. This capability allows organizations to maintain Iceberg tables as critical assets while ensuring compliance and preventing unauthorized access.

What are the differences between managed and self-managed Iceberg tables?

Managed Iceberg tables in Databricks are automatically created, stored, and maintained within the lakehouse with full governance through Unity Catalog and monitoring capabilities. Self-managed Iceberg tables require users to provision and maintain the underlying storage infrastructure independently while still allowing Databricks to read and write them. Managed tables simplify operations and provide better governance enforcement, while self-managed tables offer flexibility for organizations with existing Iceberg deployments outside Databricks.

How can I track data lineage and audit Iceberg table changes?

Databricks provides data lineage tracking for Iceberg tables showing upstream data sources and downstream consumers, supporting compliance and impact analysis for data changes. Audit logging captures all operations on Iceberg tables including reads, writes, schema changes, and access patterns. This comprehensive tracking enables data teams to understand data dependencies, troubleshoot data quality issues, and demonstrate compliance with regulatory requirements.

Full transcript

0:00 Introduction to Iceberg

Hello uh welcome to this video about iceberg support and data bricks as of March 2025 just some bite-size examples um to really give you a gut feel of how to get started it's really cool um okay so the first thing is just Iceberg interop this is how you would take um really new tables any existing tables and make them consumable as iceberg in any engine you could do it of course you can consume in data BRS but you can consume in any other Iceberg consuming engine um then we have manage Iceberg and that's where you could write um from any external engine you could have iceberg in unity catalog and then you you can consume it you can use it you can mix a match and then um so so that gives you uh for your existing data in data bricks you have Iceberg support for uh engines outside of datab bricks read and right you have Iceberg support and then if you have um an existing Lake an existing Iceberg Lake and you want to make that available to data bricks we have foreign Iceberg support and that just means like it's it's managed by something else um other than data breaks and we can connect to that as well so uh you should be able to mix and match and use anything you want um get the job done okay let's go uh first we're going to install some some libraries um I've already done this just so it's um we don't have to watch on the video but a little tip Pi Iceberg uh it changes a a lot adds new features pretty regularly and I've been burned just a few times go ahead and pin that version if you do build anything you want to schedule um because it it changes adds new features it changes apis so uh just keep that in mind all right first let's create a table and this shouldn't feel any different than any other table you would create this is a new one so I have a space data set uh I'm going to create space stuff Iceberg so we'll run that and I'm running this all on serverless I didn't have to install any sort of like spark libraries or anything wild like that it just works um cool all right so this created the table the table exists it has uh space stuff in it but you haven't seen that so let's query it but instead of doing the normal thing where I'd query it with data bricks I'm going to query it with um pi iberg and you know kind of important here yeah I'm running it in datab bricks just because datab bricks runs python but you could run this anywhere you put this in a g action in a Docker container you run it in uh any number of clouds um go wild right run it locally that's cool um all right yeah so this is the connection details you don't don't memorize this this is just for you know we have docs so let's run this so I'm going to the important stuff is I'll I'll get the unity catalog like object by loading the catalog with these connection details um I'm using a token just because I grab it from The Notebook but you can do this with uh service credentials as well uh that works fine and I'm going to go ahead and just get a pandas data frame good old pandas uh by loading that so let's get this and see what it looks like very cool um we've got our stars and galaxies and nebulas and okay so we got all kinds of stuff here um all right so 66 rows not huge uh go ahead and try this though with massive massive data sets that's fine it's Iceberg um maybe don't collect them all into a single pandas data frame unless you have a big machine but go for it right um all right next let's modify this table so I'm going to go ahead and insert the moon um one of my personal favorite space things and I don't know if um if this works as expected maybe we'll see 67 rows up here so we'll do the insert and then I'll go ahead and just run this and yeah I'm reconnecting it's not as efficient but it's fine and if this works we'll see a nice Moon um let's see here or let me run this 67 rows there's the moon perfect just another object um and it's not just inserts you can also do deletes um so down here we'll delete from there where the object is equal to the Moon here be back to objects 66 the Moon is nowhere to be seen um great okay so those are the basics you can do updates you can do whatever you want to to the table um now one thing up here to keep in mind you need these three properties I don't know why it takes three properties that seems absurd to me um we should design that better but uh don't forget some of these like this one especially um is easy to forget but it's all in the docs uh not that you read docs I know you don't read docs that's why you're watching this video you don't read docks get out of here all right cool um the last thing is let's go ahead and take our um our original table the uh space stuff uh that wasn't Iceberg aied and I'll grab the full name here and let's just convert it right instead of having to rewrite the whole table uh to make Iceberg we can just uh apply it in place uh cool and that is um done and you can see you can see anything going on and it doesn't actually change the way the table appears in data breaks the other things that I'm using with this table don't care they don't know uh and now if I go up here I realiz it's kind of boring which is maybe that's what you want right with your Iceberg support it's boring so now I'm going to query just original space stuff as a Iceberg table this time and it's the 66 rows perfect um that's it that is uh Iceberg interop uh directly in data Bricks now let's do something maybe a little more fun let's do some manage Iceberg so uh duct B and py arrow I already installed those above I'll delete that and okay let's do something kind of fun so I'm G to like to create the data you can do anything you want you can use pandas you can use polers you can use uh there's a ton of stuff I'm using duct B really um you can use any Ang like you could use trino you could use uh open source spark and um star rocks does this and I mean just so many engines support this so yeah whatever you want um here I'm just creating like a very simple table with duct B in memory and then I'm getting the arrow table out of it um so fetch is an arrow table I like arrow for this you could do it with uh pandas I think you can do with a few things and then I will create the table using the schema of this and then a pen data to it um in pi Iceberg I think it would be kind of cool to just like do that one maybe there is a way and I'm just kind of dumb and um okay let's create it so I will pull open actually my catalog here and go in here default all right we have space stuff we have space stuff Iceberg um and when we're done we'll be creating hello Iceberg um but completely from an outside engine not using data bricks um I mean I'm kind of using datab bricks compute but you could run this python anywhere it doesn't like it's not material to What's Happening Here uh so I'll run that and oh that was really fast okay I'll do the refresh I guess I could just close this and there's hello Iceberg um with its structure um let's see uh yeah it doesn't even indicate whether or not it is like something special or anything if you go and click into it you can see it's an iceberg table so that's kind of cool and like all all the normal stuff works here like you would you would have to care that this is um Iceberg versus not Iceberg like your here space stuff this um this was a Delta table when we started and I've added uh all that Iceberg interop stuff it just kind of as is added so it can be consumed by anything all right let's go back back and uh I thought it would be cool now we're going to query it in data brick so we created it with something else um and now I'm going to query it just native sequel here for hello Iceberg and there's our message um let's see what happens when we do stuff with it so let's add the same data from before let's just add it four more times um and it just depends whoever wrote that Pi Iceberg uh Library pretty cool pretty cool people working on that fako especially is super cool and I love working with this Library all right so now when I go to query this I would expect to see five total records and look we got five records and it's not just external rights we can overwrite the table so that'll um it's like a trunk en load type thing and go up here and we'll run it again so now I would expect it just the one record cuz we truncated all the previous ones and we added just one all right perfect that's so so simple is to be kind of boring uh maybe and then I like to look at the history of the table um so we're describing the history of this Iceberg table and if you are someone who likes to look in here so we did the first create uh then we did a bunch of aens and I actually don't know how Pi Iceberg does those all in one versus I thought it would be like four individual pens and then we have the um overwrite and the these are kind of cool to look into so like they're sourc from just like a general Iceberg client uh let's see here so we did a create we did the append we did the append then we did the um it removed some files this time and then added some more all right that is that is it as far as the external ice for catalog um I've got to do uh more setup for that but if there's a specific one you want to see um we support I've not found one we haven't supported the big ones are uh Hive metastore that's probably the biggest just like by presence in the world glue uh glue those are both really really simple and easy to set up in here uh glue is AWS glue um then there are the two other bigger ones but they're not as big as those first two they're Snowflake and like the iceberg they create um which we have a really we have a good UI for that as well and then the last one is rest cataloges external rest cataloges um so that could be that could be open source Polaris it could be like if you had tabular back in the day that was back in the day geeez yeah time's flying um or or who knows that's an open spec anyone can use it so any any number of vendors might be using that um and that one right now a little less easy to do um you could do it with a a small script that I have but we're adding that to the UI as well but the first three are all in the UI right now so you're good to go so wherever your iceberg is whatever you're doing with Iceberg um whether you care about iceberg or whether you just care about cheap uh mix and match of compute and and moving data loading data you should be able to do anything you want right now today uh in here check check check okay now wait till you see the real cool stuff right the real cool stuff I legally cannot tell you about and I refuse to tell you about I'll make another video for that all right thank you

Learn more about the Databricks Data and AI platform.