Skip to main content

Incremental Iceberg Table Replication at Scale

Summary

  • Scale replication: This video presents an end-to-end workflow for replicating Apache Iceberg tables at scale using Apache Spark, addressing hierarchical metadata complexity.
  • Open-source contribution: The team contributed replication libraries back to the open-source community, enabling data engineers and practitioners to replicate Iceberg tables reliably.
  • Disaster recovery: Incremental replication ensures backup tables remain identical to source counterparts, providing critical disaster recovery and data resilience capabilities at scale.

Incremental Iceberg Table Replication at Scale

Watch: Incremental Iceberg Table Replication at Scale
Talk by Hongyue Hongyue, Software Engineer, Self-Employed ; Szehon Ho, Software Engineer, Databricks
Apache Iceberg is a popular table format for managing large analytical datasets. But replicating iceberg tables at scale can be a daunting task — especially when dealing with its hierarchical metadata. In this talk, we present an end-to-end workflow for replicating Apache Iceberg tables, leveraging Apache Spark to ensure that backup tables remain identical to their source counterparts. More excitingly, we have contributed these libraries back to the open-source community. Attendees will gain a comprehensive understanding of how to set up replication workflows for Iceberg tables, as well as practical guidance on how to manage and maintain replicated datasets at scale. This talk is ideal for data engineers, platform architects and practitioners looking to apply replication and disaster recovery for Apache Iceberg in complex data ecosystems.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

Why is Iceberg table replication challenging?

Iceberg's hierarchical metadata structure makes replication complex—users must replicate metadata files, data files and maintain consistency across multiple versions and concurrent writes.

How does the open-source solution work?

The solution uses Apache Spark to handle the complexities of Iceberg metadata and data file replication, ensuring backup tables remain identical to source tables.

What catalog works best for Apache Iceberg table replication?

Databricks and Neon both support Apache Iceberg with strong replication capabilities. This video demonstrates open-source replication libraries that work with any Iceberg implementation. The Databricks Data and AI platform integrates with Unity Catalog for managed Iceberg governance.

How does Iceberg table replication compare to Delta Lake?

Both Iceberg and Delta Lake support efficient table replication, but Iceberg's open-source nature enables broader interoperability across engines. This video demonstrates replication patterns that work across Iceberg implementations. The Databricks Data and AI platform supports both formats with consistent governance.

How does Unity Catalog support Apache Iceberg tables across systems?

Unity Catalog provides unified governance for Iceberg tables through managed Iceberg support, enabling fine-grained access controls and data lineage tracking for tables in Databricks lakehouse. The integration handles both managed Iceberg tables stored in Databricks and foreign Iceberg tables in external catalogs, providing consistent governance policies across hybrid data environments. This capability allows organizations to maintain Iceberg tables as critical assets while ensuring compliance and preventing unauthorized access.

Full transcript

0:00 Introduction to Iceberg

Hi everyone. Um thanks a lot for coming so late in the day. Um today um we're going to talk about incremental iceberg table replication at scale. So. Uh currently I'm a software engineer at data bricks. I work on the spark team on the iceberg delta spark integration. Uh, prior to this I was actually at Apple for four years where I worked on Apache iceberg and I'm iceberg PMC member. Uh, this is my friend Hungu. Hey everyone.
Uh, this is Hongu. Um, I used to work at Apple as well and work with Cih on the Apache iceberg. U did some contributions. I'm really glad today I had the opportunity to share this with the rest of folks. Yeah. So we will present some work that we did at Apple called incremental iceberg table replication at scale. Um and so uh yeah please complete your surveys after the talk. So first question like uh do you how many of you use Apache iceberg?
Oh wow. Oh wow. Okay. So I can go kind of fast through this deck. Uh then um yeah I'm you know a big fan of Apache iceberg. Um so iceberg uh as you all know is open table format spec designed for huge analytic data sets. Basically, it's a spec how to lay out metadata files and data files in a data data lake and make it at like a data warehouse, right? So, and the advantage of it is it's all communitydriven. It's owned by Apache software foundation and it's built on technologies that themselves are owned by Apache software foundation like Parket, Avro, etc.
And with iceberg you get all these really cool features of to for a data uh warehouse like highly performant asset transactions row level updates delete safe schema partition evolution rollback etc etc. Um, iceberg is a spec first so has a lot of implementations under the Apache iceberg uh umbrella like the PMC. We uh govern the projects that uh implemented in Java, Python, Rus, Go, C++. And there's a ton I think growing number of OSS projects and proprietary projects uh proprietary vendors that read and write iceberg.
So that's a really huge growing ecosystem. And I think one of the strengths was like there was such a demand for you know analytic um data warehouse storage on kind of like open formats. Um so yeah so with that introduction um so that's all the good stuff about iceberg. One of the weird stuff about iceberg for me was when I first started is something very simple is actually very hard. So as of iceberg 1. 7 until 1. 8 there was actually no good way to copy an iceberg table.

2:27 Challenges of table copying

And that's kind of one of the um artifacts of the spec uh in the initial design. So iceberg table spec uh I think one of the strengths was it always prioritize uh being specific. So you don't ever want ambiguity in the spec. That's how you have like tons of engines able to read and write the same format. And as part of that it chose the absolute paths uh for all the file references of iceberg. So like metadata files point to other files and all those references are absolute paths.
And that actually makes it super hard to copy the table from one place to another because once you copy it all those references are broken. So things like uh you know just migrating a table from data center A to B or or even copying a table from this directory to that or uh more like I guess for Apple like backup disaster recovery that became really hard um surprisingly and um to show you the extent of the problem uh we're dealing with this is the iceberg spec uh it's a spec of metadata files and data files right so uh the problem starts at the very top even the top pointer the catalog pointer it points to the table version file the metadata file and that itself is absolute path.
Then you have that pointing to the child which are the manifest list you know for the snapshots. Those are absolute paths. They point to the manifests which are absolute paths. They point to data files which are absolute paths. And keep in mind this is actually kind of an older diagram. I don't think it's been updated. There's things in V2 and V3 like position delete files point to data files which are absolute path references.
And then in V3 there's delete vectors DVs that also point to uh again absolute paths to data files. So copying an iceberg table like of this to another location breaks all those references. So that's something um pretty frustrating for users. And so how do people kind of work around this problem? Um there were three main things people tried or at least what we tried. The first is kind of like you can always use a your favorite query engine spark trino. you do you know create table as select or alternative create table and then insert into select.

4:27 Review of existing workarounds

Basically you just like select all the rows and then write it back um to the second location. Um this works okay but I think the drawback is obviously this is quite expensive. You have to kind of read all the rows um process them like kind of like deserialize them potentially shuffle them reerialize them and write back to disk. What you really want is kind of like a file copy which is a lot faster. So this is kind of like you know processing all the data and what's actually worse is you lose all the metadata too.
So all those metadata files that I showed before they all had a point like they all was keeping histories of the iceberg table like different snapshots different schemas um you know things like that you probably don't want to lose. So if you do this while the data looks good all the history is gone. Um so you know you can try optimizing and I think some people tried. Um one way you can also do this on your own is maybe or this um was an attempt to do it.
Uh you can always find you know all the source table source files of the of the source table. So basically uh you have to look for files in the current snapshot of iceberg. Then you can use a you know a file level copy tool like this CP or some thirdparty tool that does kind of a quicker file copy. Um and then you create the target iceberg table and then iceberg has already a procedure called add file which generates a metadata file for a data file.
So then you generate all the metadata and then you you can call register on the target catalog which says oh this is now the target table. Um this sounds like it's it's good in practice right like it's it's it's performance. Um but there's actually a lot of pitfalls for this. first after they introduced delete files in V2, this whole thing kind of fell apart because now the delete files, you copy them over, but their references are broken.
Um, and so you actually have rows that become undeleted, which is quite dangerous. Um, and actually worse is um, you know, you you lose all the metadata, as I mentioned before, you lose all the history and stuff, but if you don't do the process of reading and writing the data back, you actually can't read a lot of the files that you just copied over blindly because maybe they need the old schema or old partitioning spec and you don't have those anymore. if you're just doing the file level copy of the data files and similarly to the previous one, you just lose history in general.
You cannot roll back to previous snapshots and things like that. So this is kind of closer but uh not yet there. And then a workaround three I think was pushed by AWS in the community was using MRAP and MRAP is kind of like a AWS feature that you can uh have a kind of a global access point uh where you kind of say all the objects are MRAP uh paths. All the iceberg references are MRAP paths and this kind of works like a relative path thing because behind the scenes um AW AWS can kind of like switch from a primary to a secondary bucket.
Um so this is kind of like more optimal like having relative paths in iceberg. It's kind of workaround for the fact that iceberg doesn't have relative paths but it's really um limiting too. Obviously only supports AWS. um the failover is requiring you to configure it and I think worst was actually like AWS has no knowledge of the iceberg file layout so it just copies when it feels like it and then so when you query the iceberg table on the target side you have sometimes incomplete data which is really terrible and again this is only for or not like this is only for disaster recovery case um if you ever you know if you you can't use this just to copy a table to your friend for instance right this is more like heavy duty uh BDR um so then That's why we decided in Apple to build a table replication workflow and what kind of I think uh there's different I think Netflix also had one but in Apple what kind of worked for us was this three-step process.

8:04 Implementing the replication workflow

Um so we know metadata is going to be broken. So we kind of first rewrite all the metadata of ice for everything that's going to be broken and we fix that. Uh we put it in the staging place. Then we uh essentially copy all those stage metadata and the original data to the target. And then uh on this on the target side we we check if all everything looks good and if everything looks good we call register that metadata file to the target catalog.
Um if you know the catalog API there's a register table where you can kind of make a table from a metadata file. So this is the three-part uh process that we use in Apple to um do kind of like iceberg replication and uh we were able to contribute to the community in version 1. 8 ates with this new action called rewrite table paths which is a spark action. Uh you may know iceberg has a lot of spark procedures like rewrite data files rewrite you know manifest.
So this is another one and essentially it's the first part of the workflow basically rewriting all the metadata files um so that all the references are fixed and um and then yeah as I said it kind of uh well the arguments you take the source location uh it's like a source you know bucket directory and a target location like a target bucket directory and then a staging location and what it does is it kind of replaces all the slashsource to slashtarget and all the metadata puts it in the staging area And then it's kind of like serves as an input to the DCP or kind of like the copy tool.
Basically, it returns you back both the metadata that I just rewrote as well as the data files that those referenced. So that now um the DCP can just take that and copy it over. And another thing it returns is the name of the kind of the root metadata version, the last metadata version file it copied. So that you can on the target side just take that and register it with the target catalog and everything works pretty well. um to give kind of an example of how this works or how it looks like on disk after this operation.
Um this is like a example iceberg table. So you already see kind of how messy this this thing is. If just by two snapshots I couldn't fit any more in the slide, but like just two snapshots you already have a ton of references that you need to fix, right? Um each metadata layer points to the bottom one. And then uh by having two snapshots, you actually have a ton of back pointers as well because you know like uh if you're adding a file, you need to refer back to the original files. you're adding a snapshot or you need to refer back to the original snapshot.
So you can roll back and it just keeps going on. And so what you what when you run this operation the first step is it um yeah so you run it with saying I want to replace slource with slashtarget. I want to move this to slashtarget. Um it goes ahead and puts everything in staging and then and it replaces all the paths. So the previous one was slource it replaces it with slashtarget. Uh notice that we don't do anything with the data files.
We don't have to rewrite them because we want to be more performant. So we just pretty much rewrite all the metadata and then leave the data files. But then we return all the paths to DCP. Um, and then DCP can take that one, copy it over, and then everything actually works pretty well because now you already rewrote all the metadata. So when you rewrite on the source side, those kind of like don't they're not really that meaningful, but once you copy it over, everything snaps back into place and then because everything is now the right path and then you can just do register table.
In this case, target metadata 2 was kind of the root of the table in the source. So we return that one and then uh you can use that to register to the target catalog. Um yeah so actually um I think that's pretty much all I had and then like uh Hong will can keep going about like different problems that we try to solve. Okay can I get this? Thank you so much. So let me expand on the second part of this. So one of the problem we had for the replication or the disaster recovery is that we need to do this often and frequent.
Uh we're living in the world where there's a lot of data change. If we only need to replicate them once that would be ideal but that's not the case. So take some example think about the streaming ice table we had. Uh oftent times we have a single tables getting its data from a cop topic uh and data being constantly appended to the tables every hour or even minally because some of the data sets has a high data refreshment requirements.

11:55 Incremental table replication

We need the data to be real time or at least near the real times. So there's a lot of change on the source table and we need to uh periodically apply such change to the target table. So um how do we handle the incremental will be the problem? Uh let me start with a really uh simple example. Let's say I just create my own tables. This got dv. source and I'm writing a single row into the tables. Um as mentioned earlier the iceberg maintain its hierarchical metadata structures.
So for writing a single row we'll end up have four different files. The top three files are the metadatas and the bottom one is the park data files. when we trying to replicate them for the first time uh we will create a new target tables and uh we will expect the same data and same snapshot history uh as the source table. So when we trying to read we see there are single rows and we saw on the target tables we have four files they're basically identical the only thing different uh different is their path.
So the source table start from source target table from target. So that's good. That's easy. How about I continuing right into my source tables. So for the second right um I'm just trying to uh provide a very simple example. I'm also inserting but this can be either updating or deleting. From the iceberg perspective it's same. Um all the data change will generate new snapshots. So on the right hand side we can see that um the latest metadata is version two.
We have a new snapshots. Uh snapshot one corresponding to the first row we inserted. The snapshot two is the second row we inserted. There's a few ways we can replicating this. So start with the most straightforward one. Let's say um if we want to uh replicate this, we can look at what's the current uh version of the source table metadata v2. So let's see what are the files necessary in order to answer u the query of the table.
So apparently we actually need all eight data files because uh each row will need four files to handle. So if we do the dumb way, we need to replicate all eight files. But um we know that we already have some files on the target side. How can we leverage that? Um we notice that uh the data change are being encapsulated into the iceberg snapshots. What if we can just copy them by the snapshots? Does that save us some times on the replication side?
So over here uh the way how we do the incremental replication is we specify both start version and end version. Those two version does not necessarily to be adjacent. It can be a range from one to 100 over here because we only have a two right. So the star will be the first metadata and the end version will be the second uh metadatas. And between those two metadatas we can see what's changing in term of snapshots and each snapshot will uh resolve to the files the manifests and the data files so that by uh identifying the difference in term of the snapshots we are we have the ability to do this incrementally.
So once we identify such uh snapshots and list of file to copy we do we basically repeat the same process as mentioned for the metadata files uh we rewrite them put into a staging location for the data files we copy them directly from the source to the target. So if we done everything correctly in the end we only need to copy four files which is second snapshots and then in the target tables we will have the same data as our source tables.
They're in sync their history and the snapshots are also um uh in synchronized. So that's the part about the incremental replications and uh one might be curious how does that um work with other ice square features you know iceber has a lot of uh concept around the branch like you can write more branch similar to the uh version control softwares and how do we handle with snapshot expirations so let's uh dive into the snapshots Earlier I mentioned um iceber snapshot has this u a beautiful feature called immutabilities.

16:59 Handling advanced Iceberg features

Every data change will create a new snapshot. And if you apply the row level operation like inserting uh updating deleting rows will also generate a new snapshot. We always uh we always move forward. So we never go back and modify the existing snapshot. That's why for the replication we can just look at what's the difference in term of snapshot and put them onto the target site. But something to notice here is that snapshot itself has some overhead like um it helps with time travels.
We can go back to a point in time and look at the state of tables but we cannot uh keep accumul accumulating snapshots forevers. we need to um keep the metadata and data size in check. The snapshot expiration is one of the maintenance uh procedure in the iceberg where they can um unlink the reference and also delete those files are no longer used. So over here um the example on the right hand side is we have empty table we insert uh twice.
So we have a two snapshots on the latest V3 of the tables we expired the first snapshot. So as part of the snapshot expiration we also removed the S1 so all the file asso associated with the first snapshots and this can be a problem for the uh uh table replications. The first problem is that uh what if when we trying to rewrite the metadatas the files already been deleted. So this will uh incur some runtime exception when when we trying to uh rewrite the metadata files.
The second part is that what if the rewrite succeed but when we try to copy them over using DCPS the files are already being deleted. then this is much harder to catch because um at the DCP stage we expect other file to be there and this uh file not found will um require a little bit more deep dive to understand why what's the external process are removing the files so um based on our experience um I think we need to schedule them differently and also we need to apply the referential integrity That means uh when we trying to replicate them over before we register into the target catalog, we need to examine that all the files we copy are actually present on the file systems.
There can be um some runtime corruptions or um user errors where we delete files are not intended. Now uh let's look into the branch and the tags and uh ice support the branching as a way to do the data experimentations tagging are used as weight for the data auditings. When we uh think about branching we think uh now we have a non linear history about the snapshots. So another example over here um we have our tables we write some snapshot S1 and we append additional data S2 over here we can create another branch which generate snapshot S3 but S3 it's special here it's not the latest snapshot so for user when they trying to read the table they will not get to the S3 unless they specifically ask for this branch but from the replication persp perspective it's the same because in the uh version four of metadata of the tables we still have all the snapshots it's just that uh for the read of table we go to S2 instead of S3 so when we trying to replicate them we take all three snapshots into considerations and the branch will be taken care of and similarly um sometimes when when we do the data experimentations we'll run into the problems uh we want to roll back or reset to a previous version.
So ice square support is through the uh start procedures like you can call ice square to roll back to a specific time stamp or older snapshots over here. Um, we also has very similar tables. We have snapshot S1 on the first right and S2 on the second right. Then we realize, hey, there's something wrong with my datas. I I don't want S2 anymore. So, let's roll back to S1. And that's the beauty of iceberg allow you to move the pointer only without doing any destructive operations.
So S2 is still being tracked and can be replicated to the target site, but the other future read will only look at S1. That's what's intended to have the uh source table and target table uh being uh disaster recovered. So um I I guess I'm I'm trying to repeat myself. As long as there's no destructive operation or anything resolved to the link uh reference of the ice tables existing replication workflow will taking care of them for both branching and roll back.
So uh the next problems we touched bit a little bit about the concurrent modifications. I want to clarify this. Uh the modification to the iceberg table can be c categorized in a few ways. The first is the rowle operations. When you insert, update, delete rows into tables, those are being taken care of because they're all generating a new snapshots. As long as the snapshot based modifications, uh we are able to handle them pretty well.

23:09 Managing concurrent modifications

But there's always something else like the file level of uh modification where some files gets deleted or removed from the disk without knowing the by the iceberg. So the problem with this is uh as we mentioned expire snapshot um that's where we are removing the files from the disk and we want to avoid running this together or concurrently with the rewrite table and DCP. So what we find out that um it's probably better to run the snapshot expiration before the uh replication workflows that give us two benefits.
First uh it will avoid any runtime surprise where the files are supposed to be rewritten or copied will always be there. The second part is that if we expire snapshot ahead of time that actually reduce number of files we need to copy it over. So that also save on the IO. Um other than that um I think concurrency is not u giving us any more surprise. So uh with both uh incremental replications and ability to handle with concurrent modifications the problem we see last is about performance.

24:56 Optimization and future outlook

We all know that uh once we have this uh disaster recovery solutions readyies, we want to apply to a lot of our production tables because those are the important ones. But usually those are the tables with much larger in size and uh sometimes these can cause the uh cluster uh resource problems. Uh one of the common example is uh we had one production tables. It has tens of thousands of snapshots. Um there been they have been building data sets for years and it's very hard to apply any u regular maintenance to the table but we still want to have the DR on top of those tables because the data loss will be the last thing they want to see.
So instead of uh so we have to make some trade-off here. Either we want to preserve the entire history like any other tables or we just want to u have the minimal available into the target tables so that um we can build on top of this in future and uh let's let's say uh look at example belows. Let's say we have a 1003 uh snapshots. Instead of replicating entire tables one by ones, let's just look at the latest snapshot snapshot snapshot 103s.
If we can just replicating this, all the future read on the target side will still work but it lose the ability to go time travel. Um the way how we do this is very similar to the incremental replications. uh we examine the uh snapshots but instead of uh finding the difference between the start and end versions we just pick the latest snapshots and replicating them overs. One thing to notice that is uh this kind of filter need to apply consistently.
So the source table can change. We can have snapshot 104 200 1,000 and when we replicating them we want to make sure we're forward looking. we're not trying to replicate in the S1 S2 which is doesn't never exist in the target tables. Um so when we leverage a technique like this uh we can speed up our replications for the large production tables but those are the um selectively enabled for the tables where we cannot the vast majority of tables they still need to go through the all of the snapshots.
So we have the exact or identicate copy between the source and the target. So the current state um as mentioned in iceberg 1. 8 we have contributed the first out of three step into the ice open source. We have the ability to rewrite the uh table metadatas. Uh it's been used into spar actions and also spar store procedures. There's a second and third step where we're still continue to working on. One of the part we want to highlight is we need the referential integrity check to make sure all the file links are still there and that they're uh consistent before we register them.
We don't want to see the problem when we query the target tables. that showed up. Some of the file are not found on systems and also uh when we're trying to register them especially for the incremental copies we want to make sure we have the ability to uh do it atomically so that either all of the register succeed or not at all. We don't want to stuck in the middle state. Uh once we have this in the ice 1. 8 We actually uh get a lot of feedback from the community.
There are surprising way of using these features. Uh we have one of the team uh from China. They're are using this to do the online hive to iceberg migrations. Uh they can help solve this problem because they do not incur any downtimes and require zero user actions to migrate 5,000 of hive tables to the iceberg. There are some bugs uh being reported from the community where some of the manifest we are rewritten has a different uh size.
So it's not a problem for the spark but some engine like a trino they rely on the ice mania to provide accurate sizing and that's something we're looking to uh to solve. And lastly, uh I think the best way to solve this uh problem is probably using the relative path very similar to delta did where the path in the metadata is based on the table location instead of the absolute path everywhere. So that will increase the portability of the iceber tables.
However, uh we need to think uh twice about the location ownership. What if someone is not authorized? they're trying to copy your tables by uh replicating them to their own location. So, we're looking forward to that in the ice v4. I think that's all we have. We're ready for any questions. Yeah. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.