Delta + Iceberg, Better Together

Summary

Delta Lake 5.0 adopts the Apache Iceberg v4 adaptive metadata tree as its native content metadata layer, creating a single on-disk structure that both Delta and Iceberg clients can read and write directly without translation layers.
Iceberg v4 introduces sub-linear commit overhead through an adaptive manifest tree, native deletion vectors for faster deletes and merges, and typed statistics that fix data-skipping reliability gaps in Iceberg.
The unified approach means users no longer need to choose between Delta and Iceberg table formats, since the same open on-disk structure serves all Delta- and Iceberg-compatible engines and tools.

Delta + Iceberg, Better Together

For years, Delta Lake and Apache Iceberg co-evolved as parallel standards. That era is ending. this video goes deep on the technical “how” of this convergence. We first walk through Apache Iceberg v4’s adaptive metadata tree: what is changing in the redesign and how the new structure enables single-file commits to significantly accelerate performance. We then cover how Delta Lake 5.0 adopts the Iceberg v4 metadata tree as its native content metadata. This produces a single on-disk structure that both Delta and Iceberg clients read and write directly. No translation layers, no conversion overhead. Just one format.
Talk By: Anoop Johnson, Principal Software Engineer, Databricks ; Ryan Blue, Member of Technical Staff, Databricks ;

Chapters

FAQs

How are Delta Lake and Apache Iceberg converging?

Delta Lake 5.0 adopts the Apache Iceberg v4 adaptive metadata tree as its native content metadata, producing a single on-disk structure that both Delta and Iceberg clients can read and write directly. This eliminates the need for separate translation layers, conversion tools, or format-specific connectors.

What is the Iceberg v4 adaptive manifest tree?

The adaptive manifest tree is a redesigned metadata structure in Apache Iceberg v4 that achieves sub-linear commit overhead, meaning metadata operations scale better as tables grow large. It enables single-file commits that significantly accelerate write performance compared to the linear overhead in earlier Iceberg versions.

Why do users currently have to choose between Delta and Iceberg?

Delta and Iceberg are currently independent open-source projects with different content metadata structures, forcing teams to pick one and then manage interoperability manually when they need to use both ecosystems. The unified metadata in Delta Lake 5.0 and Iceberg v4 eliminates this choice by making table format an implementation detail rather than an architectural decision.

What improvements does Iceberg v4 bring to deletion vectors and statistics?

Iceberg v4 adds native deletion vector support for faster row-level deletes and merges, typed statistics to fix data-skipping reliability, and native column mapping for cleaner schema evolution. These improvements bring Iceberg closer to feature parity with Delta while sharing a common on-disk representation that serves both ecosystems.

Full transcript

[00:09] So welcome everyone. Thanks for coming. This is the Delta and iceberg better together talk. I am Anup Johnson. I'm joined here by Ryan Blue. And Ryan is the original creator of Apache iceberg. In this talk, we are going to present a
[00:22] new unified metadata structure that improves both Delta and iceberg projects. I'll skip this one. One housekeeping note before we start. Uh at the end of
[00:36] the session, you will get a survey in the datab bricks events app. Your input helps shape what gets into the summit next year. So please take a moment in the end. Let's dive right in.
[00:50] So here is the thesis for the talk. One ondisk metadata. So the key idea here is that Delta and iceberg currently are independent projects with different metadata structures. So a customer has
[01:04] to pick between them, right? So we don't think that needs to continue that way. So in this talk, we'll show you how this new metadata unifies both Delta and iceberg meanwhile improving both
[01:17] projects. So this is the first point we want to land. As a user, you should not have to worry about picking table formats. You
[01:29] don't pick a database engine on details of how it lays out files on disk. Table format should be that way. They should be an implementation detail. Instead, what you should worry about is picking
[01:42] the right architecture. Does the table format allow fast writes into the table? Does it perform well with the cure engine of your choice? Does it interoperate well with the rest
[01:56] of the data ecosystem? So the rest of the talk is going to show you how this new metadata kind of helps get to the right architecture.
[02:10] So here is the unification approach at the 30,000 ft level. So we have two different independent projects, Delta and iceberg that grew up independently. How do you unify these two projects together? And can you come up with a unified state that improves both
[02:24] projects together? Here's a situation today. So Delta and Iceberg are independent open source projects. They are scalable
[02:36] table formats. So customers store vast amounts of data using Delta and iceberg. The data and most of the metadata is sitting in open formats in cloud storage. And the most important thing right now is with iceberg v3 the data
[02:50] layer is unified. So data is stored as paraveet files and rowle deletes are stored as deletion vectors. But the content metadata representation is different between these two projects. So
[03:02] what is content? Content metadata is information about files, deletion vectors and column level statistics. So because the metadata layer is different between these two projects. If
[03:14] you want to bridge these formats, you need to use a translation layer such as uniform. And this translation is not quite cheap. So you need to run this translation layer that translates
[03:26] between two media representations. And there is some overhead in in keeping these two representations in sync and also there is some lag sometimes between this.
[03:38] So why do we need to unify the formats? So the key reason is there's no reason why the content metadata needs to be different. They are effectively representing the same information. They just happen to use two different formats
[03:51] currently. It doesn't have to be that way. Most importantly, we can design a new merit representation that can improve both projects. And most importantly, we can do this in the open with the
[04:04] involvement of both Delta and iceberg communities. and Delta and Iceberg both are hitting the same scalability ceilings. Instead of having to solve this twice in two different projects, two different ways, we can do this once
[04:17] and lift up both projects. So here's our vision of the unified stack layer by layer. At the bottom you
[04:29] have the data, right? So data is stored as I mentioned in park and deletion vectors. This data layer is massive and it's currently unified with iceberg v3. So data layer is exactly same between
[04:42] iceberg and delta as of today. The content metadata has two different representation currently with the unified content metadata. This becomes the same and on top of the content
[04:54] metadata we have the two different cataloges. Delta clients talk to unity catalog and iceberg clients talk to IRC. So that information can still be the same and on top of that we have delta
[05:07] clients and iceberg clients. So none of that changes. So you can continue to use your iceberg clients or delta clients on top of this tables. So also if you look at the stack there
[05:20] is diminishing returns in unifying as you can go higher in the stack. So the data layer is very large so it needs to be unified and that's what happened with iceberg v3. The focus of the talk is the next layer the content medity
[05:33] unification. So content medity is very large um and you can translate it but translation is expensive. With this new structure we are we don't need to do translation anymore.
[05:47] We don't need to necessarily unify the catalog or the ecosystem because the data stored in the cataloges are typically tiny.
[06:00] So all of this is done on top of the foundation of iceberg v4 and Ryan will go deeper into the V4. Thanks. Hey everybody, thanks for coming to this. Great to be here. So the the heart
[06:14] of iceberg v4 um is something that we we call the adaptive metadata tree. This is this is small. So okay um so the
[06:27] adaptive metadata tree is trying to solve a problem um in iceberg that we have lots of files that need to be written in order to commit once to any table. Um and that has has been a a
[06:42] contrasting point with delta in the past because delta commits using just a single file. So delta writes the changes or deltas um into a single file where iceberg would need to write a uh
[06:55] manifest list and a manifest and uh anything else that you need. So one of the things that we wanted to do was to decrease um the amount of work that you needed to do in order to commit to a
[07:08] table. Um so we are making this so that you can take data files and uh and delete uh files the DV references and store those in the root of the iceberg
[07:20] metadata tree. Um that way any commit only needs to touch that one root file which now serves two purposes. It serves the purpose of like delta taking the
[07:32] changes from uh the latest commit and applying those uh to just the single file. And just like iceberg, it takes the place of the the manifest list uh where we store all of the references to
[07:46] the larger tree of metadata or content metadata as we're calling it. Um so that is one pretty big difference. The other thing that we're doing is that uh we
[07:58] found actually um this was a big thing earlier uh this year that that Anup and I worked on. Uh oh, sorry. Uh we found that um we can actually get rid of
[08:12] two-phase planning. Um so this new structure allows us to do some things differently. Um, so in iceberg we had previously a separation between data
[08:25] files and delete files as well as data manifests that track those data files and delete manifests that track the delete files. We were storing them separately. And what that meant was when you went to go read from a table, you
[08:39] needed to find all of the potentially matching delete files, sort of index them, and then stream through all of the data files and find the matching deletes. That was a readtime operation.
[08:51] Um, and that was to keep the uh the right amplification down because we wanted to be able to change the delete manifests very rapidly without necessarily changing all of the or
[09:04] rewriting all of the the data metadata. What we found though was that we we also wanted to add um uh metadata deletion vectors. So much like you can uh soft
[09:18] delete rows in your tables, you can now soft delete actual data files to replace them. And that led to uh being able to uh very cheaply go and and use the same
[09:32] files, the same metadata files um and just move things into the root file again. Um and by really decreasing that cost we were able to say well now we can
[09:44] actually get the same benefit of decreasing the overall cost of um of changing files uh without separating the data and delete uh metadata. So by
[09:56] colllocating we actually make um all of this cheaper on the read path. So when you read if you find a a data file, you instantly know whether or not it has a
[10:08] deletion vector. So that is actually going to be a a really powerful uh outcome as well. So this is what the adaptive metadata
[10:20] tree is going to look like. Um for a small table, you rewrite just one single manifest file. So before we would have uh you would write a manifest and then
[10:32] another file on top of it. But here our snapshot points to just the single manifest. As the table gains more files, you know, you change it, you modify it. You keep
[10:44] rewriting the content from that initial manifest. So nothing between snapshot one and snapshot 2 in this metadata layer is actually shared. we rewrite the entire thing every single time. And that
[10:57] might seem counterintuitive, but um we've found that um what really matters is not the size of these metadata files. It is the amount of time that it takes
[11:10] uh is pretty much dominated by that round trip to object storage. So you can put a lot more content in this file and not pay a performance penalty. If you're not paying a performance penalty to
[11:23] write all of the metadata again in this route, you may as well rewrite as much as you can. So that way the reads are a lot faster because you only need to read a single file in order to reconstruct
[11:36] the state of that table. So now you only need to scan one manifest in order to know the total uh state of the table that you have to read. That of course only goes so far. And so
[11:51] what we eventually need to do is kick out submanifests. And this is why we call it adaptive. Um we can change a lot of the tuning settings and and uh change when a leaf manifest gets produced or um
[12:06] or how many how large the the root manifest is and things like that. So here we needed to have a root manifest because copying that data eventually does get to the point where you're paying a latency penalty. So when you
[12:20] start paying a latency penalty for your commits, you kick out a leaf manifest and then you have a very very small entry. Um and the root manifest the size is is once again very small. Um, again,
[12:35] we can also then encode our deletion vectors against the leaf manifests, making updates to the table much more cheap and also allowing us to get rid of that two-phase planning.
[12:48] All right, another big piece of what we're doing in in uh V4 is we're moving to relative paths. This is the exciting yet boring part of the talk because this
[13:03] is great for a lot of use cases. You need this for disaster recovery. Um, you need this for easily cloning tables. It's much easier to work with data, but it's honestly mostly plumbing. So, what
[13:17] we're doing is we're introducing uh table locations that work much like Delta. In fact, this is almost identical to uh the way Delta stores its its uh information. Um so the catalog is now
[13:31] responsible for providing a table location that gets pretended to all of the paths that happen to be relative. So we can detect a um an absolute path, those are unchanged or you could have a
[13:44] relative path and we just uh concatenate them together with a slash in order to get there. So this unlocks uh quite a few nice features and also gets us a lot closer to that uh unification with
[13:58] Delta. All right. Third major change that we're making to the iceberg format is moving from a row-based representation for metadata to a columner representation. So uh I guess
[14:13] not a
[14:26] Um and we mostly wanted to be able to iterate quickly. Um but what happened was we had to store all of the statistics for data columns in maps. Um, and so we had uh a map from field ID to
[14:41] say some uh binary blob representing a lower bound or an upper bound. And those were extremely hard to work with because we had to serialize things that had already been serialized in the AVO
[14:54] format. Um, and we actually didn't track uh the initial type. So we would see four bytes uh in a binary and know from the column type that we should interpret
[15:07] that as a float or as an int. Um similarly if the column was a long and we saw four bytes we would say oh that was probably promoted from an int. Um so we were constrained on uh how we were
[15:20] actually able to evolve types. So in V4 we're actually taking all of those maps that take the lower and upper bounds column counts and things like that and we are putting them into strrus and then
[15:34] we have a strruct of each uh statistic strct for a given field. Um and we we store all of that in a columner fashion and then by moving also to parquet we
[15:46] get to be able to skip those things very easily. So rather than scanning through all the statistics for every single column, we can skip them effectively using our column format. So that
[15:59] major uh major improvement. The fourth thing I want to talk about is we're adding the ability to overwrite and append individual columns without
[16:11] actually rewriting the data files themselves. So this works much like DVs where DVs can go and mark rows as deleted and then you replace them by adding new rows and new files. What
[16:23] we're doing is we're actually marking uh or replacing columns or adding columns um by associating additional columner files with the original data file entry.
[16:36] So you could have a data file with a columns A, B and C. And if you write a new file with columns C and D, you just attach that to the original data file entry. This has uh a lot of applications
[16:49] primarily in like very wide tables where you might have a feature uh you know a feature set and you want to keep adding features for some ML application without
[17:01] rewriting you know a pabyte of data or something like that. There are also applications for overwrite where you want to continuously replace like a score column as you update your algorithm. So this is a a really handy
[17:14] feature for AI and ML use cases. Um and then I think this is the last one. The the >> Yes, it is good. I was just making sure
[17:27] it was the previous slide I'm seeing. Um, indexes are another addition um, not really in the V4 timeline because these are going to be uh, backward compatible or in other words, it's not actually a
[17:40] breaking change to add an index. Um, but indexing is something that we want to add to the formats so that we can really easily uh, find individual rows. So this helps with the um the needle in a
[17:54] haststack problems and it also helps us speed up um some of the merge queries and things like that where if you know exactly what rows are being changed from the input data um you can more
[18:07] effectively skip and and avoid reading uh too many files to find rows that might match your query. So we're adding indexes. This is actually a a really exciting uh thing because people have
[18:19] been asking for secondary indexes for years. Um and it's also going to allow us to get rid of um equality deletes for streaming applications and move those uh to deletion vectors which is just a a
[18:32] much quicker and better format for applying those deletes. Right? So that's the the basis that we are putting into um into iceberg and then uh Anoop is going to come back up and cover how we
[18:45] are also merging and and using that in Delta. Thanks Ryan. So what Ryan covered is the foundation of the unified metadata. So
[18:57] to understand how this unified metadata improves Delta, let's dive deeper into how Delta works currently. So Delta is built on top of the foundation of a transaction log. So
[19:10] every commit appends to the log and currently the logs are stored as JSON files on cloud storage. So um it's very quick to write. So in this case we have
[19:23] the V1 of the table and you commit to the table by writing a file conditionally writing a file called v1.json JSON and that JSON file contains a bunch of add actions that point to the
[19:35] data files, right? So as the table changes, you know, the log kind of grows. The second the the writer that wrote the second version of the table writes a V2.json that might
[19:48] add more files to the table or remove files to the t from the table and this continues. So V3 for example writes another JSON file that might add or subtract files from the table. So to
[20:03] reconstruct the table state as of table version V3 we have to replay the logs. So the client delta client will have to kind of go and read V3, V2 and V_sub1 to reconstruct the table state. Right? So
[20:15] as the table grows the replay cost is proportional to the the log size. So to bound the size of the log we tend
[20:28] to produce what's called checkpoint files in delta. So in in this case for example when the table reaches version 200 delta will go and produce what's called a checkpoint file.
[20:44] So for very large tables the checkpoint itself will be partitioned into a bunch of shards. We call it sidecar in delta protocol. There are paret files that contain information about all the files. So the checkpoint that's produced as of version 200 is self-standing. It
[20:57] contains all the table state as of version 200. Right? So you don't need to go and look at any delta logs before version 200 of the table. So and then new commits come in and they keep on
[21:10] appending to the log and produces new JSON files. So for instance, if you want to read version 200 of the table, you would read the checkpoint at version 200 and then apply the changes that happened to the table since then.
[21:24] So one of the problem we have in Delta is that the checkpoints are the way we write checkpoints, it's not incremental. So the checkpoint to produce the version 200 checkpoint we go and read the
[21:37] previous checkpoint apply the changes that happened on the table since then and produce the new v200 checkpoint. So the the 200 checkpoint does not share any shards with the previous checkpoint.
[21:50] So writing a checkpoint is heavy weight. So we don't do it in the commit path. Generally what happens is we write the v200 json file and do an asynchronous generation of the checkpoint.
[22:02] Um so the other problem is that because a checkpoint is full u the shards are all new files. So if any systems that cache metadata we'll see a full cache eviction when new med when a new
[22:14] checkpoint lands. So if you have to compare delta and iceberg as of today here is a scorecard. So the strength of delta is fast
[22:27] commits. To commit to a table, all you have to do is write a simple JSON file and the change detection is pretty efficient because the log tells you
[22:39] exactly what changed and the delta checkpoints as of today they are columnar. They are stored in park and the statistics are stored as trucks.
[22:52] Also the in the checkpoint files the data and DV are collocated. So from a given record in the checkpoint files you know which data file and what's a DV that attaches to it and delta is already
[23:05] a relative path but delta has some downsides one is that there is this cost of relatency from log replay because checkpoints are expensive to create we
[23:17] don't create them often enough and as a result any client will have to kind of replay a large number of JSON files on top of the checkpoints so we mitigate this by having distributed log replay. So when you run we run a spark job
[23:29] itself to kind of read these JSON files and apply them on top of the checkpoint. The second problem is that the bulk commits are expensive. Let's say you're running a very large DML job that changes many many files on the table. So
[23:42] currently we have to write a large JSON file serially and that takes some time. The last remaining problem is this checkpoint writes problem that I
[23:54] mentioned. Checkpoints are not incremental. So we have to kind of rewrite the whole table when you whenever you generate a checkpoint. Also u this is more of an implementation detail. We also keep track of deleted
[24:06] files or tombstones in our checkpoints currently for vacuum purposes. Um the last problem is uniform translation. So if you want to read a delta table as iceberg you had to enable this feature called uniform that does
[24:19] this translation. So as you can see iceberg v4 as Ryan presented earlier kind of incorporates pretty much most of the strengths that Delta already has. And if we incorporate
[24:32] the iceberg v4 metadata tree into data delta itself we can actually mitigate almost all of these weaknesses.
[24:44] So how how so? because we can produce this new checkpoint incrementally and we can produce this much more often and bound the the cost of the log replay. So the delta 5 is basically introducing
[24:57] the iceberg v4 metadata into the heart of delta. Let's see how that works. So here's the key idea. So we are delta itself takes in the new iceberg merger
[25:10] tree and can read and write from delta clients right. So the if you want to use iceberg clients on the same table you don't need to run uniform they are operating on the same metadata on disk.
[25:22] So they we don't need to do any translation anymore and the delta's current log based commit model stays. So all the delta features that you know and love you know continues to work and this
[25:35] preserves all of Delta strengths and key features and this is implemented as a new reader writer feature in delta. So once you enable this feature even on an existing table the table will start producing iceberg v4 compatible metadata
[25:48] tree going forward. So as I mentioned this new metadata tree is identical to what iceberg stores on
[26:01] disk to do this delta continuously have two different commit patterns right so we have what's called manifest commits let's say you're making a very large change to a table um delta writer itself
[26:13] can incrementally produce this metadata tree so unlike the current checkpoints which is an asynchronous thing on the side this is right in the commit the comm the writer itself produces the V4 metadata tree incrementally. So it knows
[26:26] what's the last version of the tree and it applies the changes on top of that and produces a new version of the tree and the pointer to the tree will be in the delta commit itself. Right? Um we
[26:39] also have what's called log commits. So log commits are for very small changes. So if you're making small changes, let's say you have a steady stream of like small inserts going into the table. So
[26:51] we don't need to rewrite the root manifest on every commit. Instead, what you can do is you can stack a bunch of delta logs on top of that tree. So log commits works identical to delta commits
[27:03] as of today where the content media changes goes to the log and a reader comes in. It applies all the log commits that happened since the last tree and
[27:15] produces a table state. So here is how the two commit types will work together. So let's say version 200
[27:29] is a large change. It sees that instead of having a very large JSON file, Delta client will figure out that it is more efficient to go and produce this changes on the tree than it is to write a very
[27:42] large JSON file. So it produces this new tree structure um which is the root manifest 200 which points to a bunch of leaf files. Note that many of these leaf files could be shared with the previous
[27:55] version of the tree. So it's fully incremental and that is done as part of the v200 commit itself right and then the 201 is a small DML that you ran and
[28:07] it looks at this and says oh yeah it's a very small change instead of producing a root manifest file on disk I could just simply write a JSON file as delta does today. So this log commits and manifest
[28:20] commits you know the delta client makes an intelligent decision on what makes sense and you know does that
[28:33] so if you're a delta user what do you actually get from the system so one of the primary benefits is it will improve read performance how because we can produce this tree incrementally and much more frequently than delta produces
[28:46] checkpoints today. So that kind of bounce the size of the log and you know you it's almost as if your checkpoints are much more frequent, right? So you reduce the cost of the log replay cost
[28:59] and the iceberg before root manifest contains aggregate stats. So it show it knows what are the lower bounds and upper bounds of the leaf manifest files. So if you have a query that is looking
[29:11] at a small amount of data by reading the root manifest you can prune out the the the leaf manifest without even opening them. So this should speed up point queries on delta you know significantly
[29:25] and more importantly the because the tree is incrementally produced the the manifest files are much more cachable a new tree will not cause a full cache eviction or cause performance issues.
[29:37] It should also improve right performance. So there is lower right amplification. Instead of paying a full cost of rewriting the entire table state in every checkpoint, we are doing this incrementally. If you're doing a very bulk write,
[29:50] instead of producing a very large JSON file, it will go and produce a tree and that's a paralyzable operation. So in in a short like the it improves read and write performance in delta and
[30:03] more importantly if you want to read these tables as iceberg you don't need to run uniform this is the same metadata that iceberg uses and iceberg clients and can operate on the same metadata
[30:15] so net all the improvements to delta there are strict improvements they improve both reads and rights on existing delta tables
[30:28] Migration is intentionally painless. It's as easy as flipping a new reader writer feature on your existing delta tables and going forward it will produce the iceberg comparable metadata instead of delta's u you know current metadata
[30:41] right so this is all kind of implemented inside delta and if you're upgrading from existing iceberg v3 tables the same v3 to v4 upgrade cycle applies so it can
[30:55] do an in place migration of the existing iceberg metadata for example For example, if you have V3 manifest, all you have to do is the upgrade has to do is write a new root manifest which can continue to point to the V3 manifest and
[31:08] it just works. More importantly, if you care about both Delta and iceberg, you don't need to run uniform anymore. Uniform is sunset and there is no need to have duplicate
[31:20] storage or conversion costs and strictly improves both Delta and iceberg. So to wrap up the if we have to take one takeaway point which is that you don't
[31:33] have to choose anymore between Delta and iceberg. The Lakos ecosystem between these two projects are converging into a identical metadata on disk and this unified metadata improves both Delta and
[31:45] iceberg as a step function and you get seamless read and write interoperability from both Delta and iceberg clients and you don't need to worry about picking the formats and no more uniform.
[32:02] So that concludes the talk. We're happy to take questions. Um, also please fill up the survey.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.