The Future of Open Table Formats: Delta Lake, Iceberg, and More
Summary
- Format unification strategy: This video explores the unification roadmap for Delta Lake and Apache Iceberg, revealing how these open table formats are converging to serve diverse data and AI workloads.
- Key innovations: Learn about deletion vectors for efficient data removal, enhanced metadata capabilities and V3 features that advance open table format innovation.
- Open ecosystem collaboration: Understand how Databricks contributes to open source communities while maintaining strategic direction for format evolution on the Databricks Data and AI platform.
The Future of Open Table Formats: Delta Lake, Iceberg, and More

Talk by Daniel Weeks, MTS, Databricks ; Ryan Blue, Member of Technical Staff, Databricks
Open table formats are evolving quickly. In this video, we’ll explore the latest features of Delta Lake and Apache Iceberg™. Join us to learn about what’s driving format innovation in open table formats.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction1:42Agenda5:55What motivated this12:20Data unification13:42Deletion vectors20:29Iceberg metadata25:08Delta metadata37:28Single file commits
FAQs
What is the Delta Lake and Iceberg unification plan?
Leaders from Databricks are working with community members to unify the table formats, sharing innovations like deletion vectors and enhanced metadata capabilities while respecting both communities' independence.
What are deletion vectors?
Deletion vectors are an efficient mechanism for marking rows as deleted in open table formats without rewriting entire data files, improving performance for updates and deletes.
Will open table formats like Iceberg and Delta Lake become the default standard for enterprise data?
This video indicates that open table formats like Delta Lake and Apache Iceberg have already become the standard for serious enterprises. Both formats solve critical problems: ACID transactions, time travel, schema evolution, and compatibility across tools. The question isn't whether open formats will win (they have), but how they'll coexist and interoperate. Delta Lake's performance optimizations and deep Databricks integration give it advantages for analytics and AI workloads, while Iceberg's vendor-neutral approach appeals to enterprises with multi-vendor stacks. Expect both to remain dominant rather than one single standard.
comparing open table formats
Both Delta Lake and Apache Iceberg are modern table formats solving critical data reliability challenges, but they have different strengths. This video explains that Delta Lake, built into the Databricks Data and AI platform, provides superior performance through optimized storage layouts, fast point updates, and deep AI integration. Apache Iceberg emphasizes vendor neutrality and read/write symmetry across different tools. The choice depends on your ecosystem: Delta Lake excels in Databricks environments and for AI workloads, while Iceberg works well in multi-vendor data stacks. Both support time travel, ACID guarantees, and schema evolution.
open table formats and data governance - how do they fit together
Open table formats (Delta Lake, Iceberg) and data governance are complementary: formats provide the technical foundation (ACID, time travel, schema tracking), while governance adds policy enforcement (access control, lineage tracking, compliance). This video shows how Unity Catalog governance layered on top of Delta Lake creates a powerful system—you get reliable data management (guaranteed by the table format) combined with policy enforcement (governed by Unity Catalog). Together, they enable enterprises to scale data infrastructure while maintaining compliance and preventing accidental data exposure.
Full transcript
0:00 Introduction
Okay. Well, looks like we have a few people uh coming in, but we'll get started anyway. Hi everyone. Um Um I am a member of technical staff at Data Bricks. I joined a year ago through the tabular acquisition. um where I have been working on open data formats and more specifically Apache iceberg and. I'm also an employee at data bicks and I am also working on Apache iceberg. I was uh one of the original creators with back at Netflix where we designed and built it originally and uh been working on storage table formats and cataloges for a long time now.
And actually what we're talking about today is very exciting because it is uh basically the unification plan for iceberg and delta that we've been putting together over the last year or so. So um hopefully you are into you know very deep in data formats uh because we're going to go there. But first everyone please read this uh statement about uh forward-looking statements. Um, we we of course uh are not making promises.
I never make promises. So, yeah, read carefully, please. All right. Um, also just want to say please complete the surveys through the app. Uh, this helps us put together a better program for the next time. We really appreciate you guys taking the time to do it. All right. And with that, we can get started. Um we're talking about the future of open table formats. So this is um basically the work that we're doing uh at data bricks with uh both delta and iceberg uh community members to unify the table formats and uh we just want to present talk about where we're headed today.
1:42 Agenda
Now um there there are actually quite a few pieces of this that we'll cover. Um so first of all we're going to talk about what unification is, what it means and our our strategy for uh unifying the formats. Um and of course anytime we're talking about unification um keep in mind much like the forward-looking statements that um you know we are contributors and we're participants in the open source communities. So while we as data bricks might have opinions or you know preferences on how things go um we we don't claim to control you know those communities or be able to even make promises.
Um so yeah I don't make promises. Um we're also going to cover uh V3 um and sort of what that contributes to data level unification um and then go into you know a strategy for the rest. So looking at both Delta and iceberg table formats, the opportunities for unification and then sort of the strategy that we're putting together for how that could take place. Um, of course, uh, I also always always want to mention that I don't want you guys well I mean maybe the people in this room seems like a small enough group, but I don't want many people actually caring or thinking about data formats in general. and you know the course of your your day as a data engineer or a data analyst uh you know maybe building ML models, you should generally just have a a platform that works without needing to think about it without um being surprised by random behavior of like zombie data when you rename columns and things like that.
So please don't care about table formats. That's not the point. Um but there is a a big component here that you do need to care about which is architecture. So what is the format that you choose? Uh you know what is the effect of that on the architecture that you want. Um, I think that, you know, most people here are, you know, pretty sold on the lakehouse architecture and that's really what we're trying to unlock, right?
The foundation and bottom layer of a lakehouse architecture is open data formats that allow you to use your, you know, central copy of data that is well-governed through a central catalog with basically any compute that you need in order to do uh your job. So hopefully you care more about architecture than formats. Um but we are going to talk pretty much about formats today. Now you might be thinking well uh why and I think it's a a good topic to just go through and and talk about how we are going to do this.
But if you are someone who just cares about architecture and capabilities and things like that, I would really encourage you to go to Bridto's talk later today. Uh so, he's giving a talk on you know essentially uh this phase two here in the middle. Um this week I think today our announcements are that um you know we have used Unity catalog to effectively unify table formats. If you come in through Unity APIs, everything looks like Delta.
If you come in through iceberg APIs, everything looks like iceberg. And we try to make that as seamless as possible. If that is the thing that you are primarily interested in, I encourage you go check out Bridto's talk as well. Um, but right now we're going to skip ahead, you know, to phase three to the very end and talk about what we're doing to unify the table formats hopefully in the next couple of years. Okay. So you've you've been warned.
We're gonna go and I will for uh unification and our approach hand it off to. Thank you. So I'm going to go ahead and start from a very high level view of why are we even talking about this? What motivated this? How did we get to where we are today? And a lot of that really has to do with distributed processing, how it evolved over time. It's been 20-ish years since we had like the map reduced papers and HDFS. And that's really the origins of a lot of this even cloud-based distributed processing.
5:55 What motivated this
But at the time it was an engineering challenge, not an analytic challenge, not a SQL challenge. It was how do we process, sort, do all of these challenges with large amounts of data and cheap commodity hardware. But then over time, what happened is we had the introduction of the cloud. cloud brought object stores, you know, infinite storage, a bunch of different capabilities that really didn't mesh well with those original technologies that we built for data processing.
And that's because they were built off of things like file system constructs. And over time, as we started to rely on the processing mechanisms and looking for guarantees from the data, it's not so much about like how big is it, it's that how correct is it? Are we getting, you know, uh the results we're expecting? Do we have the guarantees about how it's changing and evolving over time? Do you know do we have the constructs that we need for data engineering, data analytics, data science?
We saw that there were major gaps and this is where the community uh you know in many different areas uh started exploring you know ways to address this through cloudnative or cloud first solutions. We did this at Netflix with the introduction of Apache iceberg. Data Bricks was uh addressing similar challenges uh with the introduction of Delta Link. Different communities were looking at these problems from different perspectives and it's only natural that you know people find those problems.
They're addressing their localized challenges and they're fixing things either in correctness or performance or something related to this dramatic shift that we had in the infrastructure. Um so multiple technologies evolved uh over that period of time. And where we ended up is that we actually now there are many projects that have you know disappeared they're exploratory but now they like iceberg and delta have emerged as incredibly successful projects and just to give you an idea there are exabytes and exabytes of data that are stored in these two formats.
Both of them provide many of the same capabilities now because as they started in different locations, they started closing gaps in terms of the behaviors, the guarantees that they provide. You know, asset operations was one of the very first things that everybody was interested in because of the you know the uh the kind of object store guarantees, you know, eventual consistency. You know, these were kind of the first things were addressed.
But then you move up the stack and you start solving things like schema evolution and its performance and a bunch of other features around these table formats. But that caused a rift in terms of as time goes on, you've got huge amounts of data in both. And then there's a question of what is the kind of matrix of things that I need to do to solve my problem? Which one is the right one? Does this solve this problem better?
Now I've got both of them. So this has become a problem for the industry as a whole where you have similar capabilities across two different representations and now you have to make a choice and what we want to do is we want to alleviate that choice. So this is why we talk about unification. So what does unification look like? Well, you know, we want to address this working with both communities in a way that advances the state of for everyone. uh we don't want to be making you know negative trade-offs in one place or another in order to achieve unification and one of the best things about this is we've got many different open source communities that are tackling the same challenges we work very closely together the parquet community arrow uh Delta Iceberg these are all related technologies and we're all solving similar problems so we've been working together but we also want to make sure that we're learning from each other's mistakes and taking the best of everything to move forward We also want to make sure that we're making strategic investments so that we can make the smallest amount of change and get the most amount of value out of really niche features, really niche capabilities.
Like we do want to address those over time, but we really want to get to the value first. So when we think about this in terms of unification, we've kind of deconstructed the table or this, you know, the storage area into three main areas. One is the data layer. going to talk a lot about V3, but this was a major focus of what we were doing in V3. Making sure that data that you're going to write and you're going to keep for 10 years, looks the same, can be interpreted the same, has the same types.
And that's really critical because that's the most painful part of any sort of migration or anything. It's like, hey, go rewrite that pabyte of data to change something. That's not the next one is what we're calling kind of the content metadata layer, which is how do we represent the constructs of the actual underlying table representation? what's active, what's, you know, old, how do you time travel? These these kind of things are all part of the metadata content.
And then at the top level, we've talked a little bit about Unity catalog. You have the catalog layer, which has a bunch of behaviors and capabilities that are built around the table formats themselves. And so there's these distinct areas that we're focusing on in terms of what does it mean to actually unify formats. There's also ecosystems. uh there's lots of libraries that interact with you know iceberg data there are lots of libraries that interact with delta data uh tools uh capabilities that you know these different libraries bring integrations with engines and what we're saying here is that like these things are part of the surrounding ecosystem and they each provide kind of unique value so what we don't want to do is remove that value what we do want to do is have a common representation so that when somebody reads that data manipulates in some way and then produces a new set of data that the next tool despite what ecosystem or language or capabilities it has can interact with that data in the same way.
So it's really getting to a common representation. With that I'll hand it to and he'll go through B3. All right. Yeah. So um we're going to hop into data unification um which is that lowest level uh essentially parquet files um that persists for so long. We wanted to get a start on that. Um, so there are uh not too many differences here. Um, not surprisingly, you store an in32 and paret underneath delta the same way you store an in32 underneath in parquet underneath iceberg.
12:20 Data unification
But there were differences um and we were uh actually starting to diverge um in in some key areas. So what we first did was we looked at you know what are the types that are in iceberg but not delta or delta but not iceberg and you know where do we need to uh get our hands around this problem and what we're going over here is the iceberg side of the problem. Um there are some gaps elsewhere that we're going to cover uh at the end like you know time type but those are are actually harder problems because Spark itself doesn't support time type right now.
So although iceberg does um you know you haven't been able to use it spark for a long time but on the iceberg side we found that we could introduce u some some really valuable changes um that get us a lot closer to unification. Uh so we're going to go through those. Um one of the marquee features of Delta Lake is deletion vectors. Now, iceberg also supported uh a rowle delete format. Um but we encoded uh on from the iceberg side, we encoded the deleted rows as file reference and position within that file rather than having a DV per file and then a position DV that's deleted.
13:42 Deletion vectors
Um now this was just a a incompatibility for a long time. Um but we also had other issues with iceberg v2. So uh V2 in Iceberg introduced the rowle delete format and it actually was too permissive. So we thought you know you might want to be able to encode another file of deletes and then send that off and then asynchronously compact that later. Uh and it turns out that when you allow people to do things asynchronously sometimes people just never do them at all.
So we had tables that had thousands of delete files against data files and you know essentially people complaining and when people complain about table formats we have a big problem because people should not be thinking about these things. So uh what we did was we introduced synchronous maintenance and uh better tracking for DVs. Um so we we made the design a lot more similar to Delta and we actually incorporated the same DV representation um so that the DV files that Delta uses are compatible with iceberg and vice versa.
So this is basically making our our representation of rowle deletes across the formats um nearly identical um which is a great step forward for anyone trying to use deletes. Um, what's actually going to happen in a month or two is we're going to, again, I'm not making any promises, no forward-looking statements here. Um, but in a month or two, we're going to release uh the ability to take your uh your delta tables and use them with uh you know the iceberg interface using a v3 structure for those delete files.
So, it gives us you know tangible benefits right away. Um, I should also say that this was a great example of collaboration across communities. We got Delta people involved in the iceberg mailing list talking about, you know, yes, it is important to have synchronous maintenance. Uh, you know, here's what we learned from the Delta side and it was a really great example of the communities coming together as was row lineage.
Um so row lineage uh is a lesser known feature within Delta called row tracking uh where you get synthetic row ids uh for basically every single row in the data set. And you also have a label that tells you what version uh of a of the table a row was last changed in. So that's actually really important for really high quality change data feeds. So if you want to know exactly what changed uh even through like a copy on write operation um you can either go and compare every field of every row to figure out you know what rows were just caught up in this copy on write operation and unchanged but but were moved to a different file or you can actually use uh this metadata to uh to target the rows and get really fine grain changes.
So another great thing uh that we're adding to iceberg um that is important across all formats. Uh next I want to highlight um a couple new data types. So variant data type is probably the biggest one. Um where for the longest time people just kept putting JSON into their columns and despite you know those of us on the side of you should always have a schema people just kept doing it. And then when I caught myself doing it, you know, we all do it.
Um so uh actually this is a really great story where um we uh as data bricks uh brought to the parquet sorry park community the spark community um a a new variant type variant format. Essentially, let's uh let's attack this problem and be able to store a binary representation that you can access super quickly and uh have some heruristics for pulling out data as top level parquet columns so that you can actually get the the parquet benefits.
Um so we did that. Uh then actually the iceberg community and snowflake in particular picked up that work and said hey we should use this everywhere. Um and so we said yeah that sounds like a great idea and we all decided to move the specification into parquet so that every parquet file and you know downstream derivative of parquet has this ability as well. Um so we're delivering this in v3 it's also in spark 4. 0 uh coming in uh I think delta 4.
0 as well. Uh so uh it was a really good story and a fantastic feature for those of you who you know just have semistructured data and don't want to use JSON parsers all the time. Um lastly uh we've added geospatial types and you know like the other things here this was a collaboration and really uh represents uh you know something new that was not part of unification. Um but if we added it separately then it would have made the unification problem harder.
So we decided to pull it into iceberg v3 um again standardized the types in the up uh upstream parquet community work with the delta and iceberg communities to get those specs updated and now we have uh geospatial uh data types coming. Um so now you you don't have if you have map data you don't have to you know just use hive or or uh some binary representation. you can actually have you know a a data lake architecture or sorry uh lakehouse architecture uh for geospatial data.
So we're really really excited about that. And of course there are more things um so we're standardizing default values uh you know write defaults initial defaults across um across both formats. Uh we're working on additional type widening use cases and things like that. And we're also delivering these on the delta side through iceberg compat v3 features uh that should allow you to take advantage of these uh as you access your your tables as iceberg tables um even though they they are um you know they were originally delta metadata.
All right. And with that we're going to go and talk a little bit more about the metadata layers. Oh, am I good? Okay. back on. Um, we that's the quick update on the data layer. So, a lot of good progress there. The next uh layer is metadata. So, how do we take the content metadata or this tree of metadata files um that we use and figure out how to move to a common representation that is a benefit to both communities.
20:29 Iceberg metadata
In order to do that, we want to talk about the state of, you know, both projects right now. So, for iceberg, I'll hand it. Thank you. And what we're going to do is we're going to take a kind of quick visual peak through what is this iceberg metadata that we're talking about so people can kind of have a an a general understanding of what it looks like today and that's going to be really helpful in the next section. So when we talk about iceberg metadata, we often refer to it as a tree structure.
And everything is navigable from one central point at the very top of the top of this tree. And that tree actually encompasses, you know, both like historical versions of the table, but most of what we're going to be focusing on is what we call the content metadata. So this is the metadata and how it points to files in the actual object store. Now these are files as well but the way that we construct the tree currently in versions like one through three is that we have this two level structure of metadata.
The top level is the is the manifest list and the manifest list is pointing to a series of manifests. In that manifest list, you have information about the manifest like which partitions they pertain to, statistics about those um and other relevant information for you know maintaining the format information. Below that you have the manifests and those manifests all point to data files or DVs or delete files and this makes up the structure.
So if you think about a tree structure when we want to go and update that tree we perform treel like operations. So what do we do when we want to create a new commit or a new snapshot of the table? What we do is we try and reuse as much of that old tree as possible. So you're creating new nodes and attaching data files and manifests within that update. So now you can see that we've added another manifest to the table.
It's got new data files, but it's also referencing information from the previous manifests. So this is all just kind of treebased manipulation in order to progress the state of the table. So then if you take a look at this over time, you have this representation of a tree structure and your next iteration of that is always going to be a mutation reusing components of the previous version of the table. And this largely makes up every mutation of iceberg metadata.
You're you're using this tree structure to point to the underlying data files. You're carrying statistics at the different levels. We're using different statistics at different levels of the tree, but that allows us to reuse as much as possible without having to rewrite the data. Even if you're doing delete operations, dropping data, it's always a mutation on the next version, which also allows for time travel by walking backwards through those snapshots.
So, at the end of the day, we have this kind of general representation of a tree structure. And this is largely what encompasses everything about iceberg metadata at a very high level. So there are some strengths and weaknesses to this approach. It's not that this is perfect. Otherwise, we wouldn't be here talking about how we can improve, but you know, this actually allows for very fast planning because all you're doing is you're actually navigating this tree.
You're able to prune large portions of the tree using statistics or partition information. You know, it's stable. every representation of the tree from you know the root to all the leaf nodes is captured for every state of the table and it's very efficient in order to update that structure because you're reusing so many components of it. There are some drawbacks though. There's right latency. We actually have to produce multiple pieces of metadata in order to add just like a single file.
So sometimes there's a bit of right amplification and depending on like the small scale or how fast the data data is changing that can have an impact on you know the amount of metadata you're tracking and performance of the table and you know there's also issues with the levels like I mentioned that we're keeping different stats at different levels and that means we potentially have to navigate through levels to take advantage of stats like we can't do that at the top level so we're not being as efficient as possible in terms of how we skip metadata.
25:08 Delta metadata
With that, we'll move on to Delta. Headed. All right. So, let's go through the same thing on a format I just started learning about a year ago. No, we we we had uh looked at Delta before that, trust me. So, um Delta has a very different model for metadata. Um and again we're talking about the content metadata here not you know how to partition not what columns are in the table or anything like that just how do we keep track of the actual data files which data files are in a table to do that delta has this system of deltas uh and each delta is a JSON file like you see here so v1 points to or actually is is represented by a v1. json JSON file and that file contains uh uh basically events like add file or remove file that uh encode changes to the set of files in the table.
So here we're adding file A and file B and so we just write a new delta file with file A and file B referenced in it. Similarly, if we want to go to the next version, we simply say, okay, well, now I'm adding files C through M and I write a new delta. Now, note that these have no pointers to one another. These these have uh, you know, no relationship. I don't have to read all prior deltas in order to create a new delta.
And that's one of the really nice things about delta is we can uh with very minimal state uh we can express just the changes and have a a new version of the table. Um of course the drawback there is that once I get to this state where I have multiple versions I have to go read v1 v2 v3 and so on to get the entire state of the table. And I can't just say, "Oh, okay. Well, I see file A is in V1 and go release a task and start processing that because I might have a delete event or a remove file event in a later version.
So rather than, you know, constantly read, you know, thousands and thousands of versions, we've introduced a checkpoint. So periodically, and I think this is usually every 200 commits or so, we turn one of those commits into a checkpoint. And to do that, we take previous checkpoint or any information about that commit uh and we squash all of the changes on top of it to get a representation that we can write out and read quickly.
Now, this representation, of course, we don't want to have just a single data file because these tables could be really huge. So we've uh we've divided in checkpoint B2 all of the um actual file references across multiple shard files. And then we also have a checkpoint file that contains um both locations of all the shards as well as any protocol or like table level information. Now we're going to put the table level information aside here for a second um and just focus on those checkpoints.
Um but uh you know notice that this is also surprise a tree structure that contains the entire state of the table as of some point in time and that's going to be important. So now you know like iceberg there are some trade-offs there there are good things about this bad things. Um so first of all we get fast commits where you don't have you know quite so much state um to keep around. Iceberg commits are actually, you know, fairly complicated because you're making a change and it's not like you're you're making a change and making sure that you can still do that.
You may have altered physical files. You may have uh you know need you may need to rewrite multiple metadata files. Um so they can be generally uh more expensive. Um so with Delta you avoid those problems. Um it's also a lot easier to detect changes because the only things that have changed are in those delta files. Um and then these are also using parquet for the uh shards of those checkpoints. So you get uh essentially column scan planning which is a a really good thing.
Um still there are some weaknesses as well. So, uh, that log replay, so getting all of the events since the check checkpoint, um, and then smooshing them into the current table state is actually a fairly expensive process because you're reading up to 200 small files. Um, you know, there are analoges of that problem on the iceberg side, too. Um, but you have to you have to do that. There's no metadata filtering operation that allows you to skip any one of those uh deltas against the last checkpoint.
Um checkpoints are also fairly substantial. So you're rewriting the entire table state every 200 commits as a checkpoint. And so we would like to make that process a lot more lightweight. Um and also like if you have a really really huge commit, this is kind of minor. Um you have to write the whole thing as a JSON file. So if you're writing a million new data files into a table, that can be quite slow because you have to put them in a single uh single delta file.
So we we've sort of set the stage with both the background on iceberg and the background on delta. Well, what do we do uh to bring these two different worlds together? And the idea here is pretty easy. We've we've hit it in the slides already. It's iceberg as checkpoints. So let's go into what that might look like. Well, here is iceberg metadata and here is a delta checkpoint. And as you can see, we can kind of just change the labels.
So that's that's our plan. We're going to rename a bunch of classes in Java. Just see what happens. No. Um it's actually not this simple. Uh but let's let's take a minute and think through if we were to do this uh what what would we gain and where would we go? So if we just said okay the unified table format is going to be delta with current iceberg metadata underneath as checkpoints we would actually get some really cool properties out of this.
So first of all we would get incremental checkpoint updates. iceberg is really good at reusing work and making changes to the tree structure that we use because we don't have deltas. So, you know, we try to make the updates to our our version tree as efficient as possible. Um, and we've gotten pretty good at that over the years. So you know where we don't have incremental uh you know like cheap cheaper checkpoints on the delta side we can get that for free by integrating iceberg metadata iceberg also because we have a a different way of cleaning things up we get to omit deletes from checkpoints or omit deletes when we rewrite any data file.
Um so rather than carrying these delete file entries around for seven days uh and then using that as a you know signal that we can delete data files if we go with the iceberg model then we can actually make the entire checkpoint metadata tree more efficient by storing less. So essentially every checkpoint get rid of what churned out and just have the data that is still in the table. And that actually can be very very uh impactful if you have uh high churn in your tables.
Um in addition the metadata inherits iceberg's property of being fairly cache friendly because we're reusing the data file show you the metadata files across checkpoints. Um and you know cache friendly is a really good thing if you want fast query planning and things like that. Um but then it's not all free stuff in this world. So if this were our plan for unification moving forward, there are some substantial problems.
So first of all, from an iceberg perspective, we would be saying, okay, well, iceberg versions then we have this other kind of way to encode changes to the table. And that's a a new concept of adding deltas to iceberg and then brings in the latency of log replay. So it would make the iceberg read path uh you know significantly slower to do it this way. Um so that's that's not a a great outcome. Um and al honestly probably not something that the iceberg community would agree to. you know, like, hey, we we want to unify the formats, but we're going to make reads slower and we're gonna increase the complexity.
So, we need to figure out a way to work around that. Um, also, iceberg metadata is not like uh Delta's checkpoint metadata in that it's not yet column. Um, so we actually have to work around some deficiencies of iceberg in this world. So, like maybe we should do something about that. And then we have other problems like file locations in iceberg are absolute in delta they're relative so that you can pick up all the files in the table move them somewhere and the table just works.
So you know we can't really do this because writing out checkpoints in this world would mean you can no longer move your table around which you I've heard people are doing for disaster recovery but it's S3 who needs it. All right. So, you know, what we need to do is, I think, expand on this idea of iceberg checkpoints. You know, use the things that look alike um but solve some of these deficiencies. So, how do we do that?
Well, this is the heart of the plan that we have. Um number one we need to make some iceberg changes so that can have relative paths column metadata and sort of you know get iceberg metadata to the point where delta checkpoints are so that we're not regressing again we want to be improving both formats at the same time with unification so that people have a reason to do this. So we will do that. We will also introduce one file commits and this is our plan for really making metadata supercharged and basically better than delta is better than iceberg is at the same time.
Now we need that in delta because iceberg commits are slower, right? We we can't have a checkpoint and keep them updated and actually like get rid of the deltas unless we're producing a checkpoint on every commit operation in delta. In order to do that, we have to have something that is about the same uh efficiency and cost for you know quick writes to that table. So delta having single file writes that's a really big step.
So we want to introduce single file rights on the iceberg side as well. Now that is roughly what we want to do. So take this idea of checkpoints with iceberg metadata improve iceberg metadata to the point where we can produce a checkpoint really quickly and incrementally like do an iceberg. So make it even faster to keep these up to date and then improve delta by having a a checkpoint for every single commit. get rid of log replay and essentially uh remove the requirement for distributed planning and have faster metadata operations overall.
It's also going to uh basically make uh cachability a whole lot better because we're going to have uh longer term persistent metadata files rather than rewriting them all the time. So that's the idea. How do we get there? Well, here is the basics. So, we're going to start and say, okay, we want to like iceberg maintain the metadata tree, the content metadata every single commit. Okay. Now, we also want single file commits.
37:28 Single file commits
So, let's take advantage of the fact that we can um write more in a commit than we necessarily need to given the latency uh to and from S3. So rather than just writing the files that change in the data file that we write for every commit, we're going to just keep copying stuff. So as I write a single manifest, my next commit rewrites the entire thing rather than referring to the last manifest. And that gets us this single file commit.
This is great. But again, as we talked about in iceberg metadata, we know that we're going to need a tree structure because we can't just carry everything forward every commit. So what we want to do is actually uh make the tree adaptive. So for when the root file fills up when it gets too large to be a very quick commit operation and you start having uh latency due to the throughput and size of that file rather than just the round trip to S3, then we cut another manifest file.
So you should get this sort of saw to behavior where your your commit latency slight slightly creeps up and then we cut another manifest file and it drops back down. So that is our our rough plan for how to introduce uh single file commits. Of course we're also going to make uh you know other substantial changes here. Uh basically, um everything that we need to change here in iceberg v4, the iceberg side of this is allow data and delete files in the root manifest.
Um make it super fast for those single file commits to happen. Um we're also going to um oh, another idea that we we think is important is using DVs at the metadata level so that we don't have to rewrite manifests all the time. So rather than rewriting one of the leaf manifests and committing a a new root manifest, we can just encode a DV against one of the existing manifests. Again, increasing our ability to cache metadata uh have faster scan planning and also this this removes the uh the large number of files that both formats accumulate.
Every single commit in iceberg creates a manifest and those stack up and accumulate until you uh eventually compact them. Every commit in delta creates a new delta and you have to read all of those. But if you're continually rewriting that root level, then you keep things nice and compact and again um you know really speed up that uh uh commit path as well as the read path. So, that is and I'm I'm almost perfectly at time.
That is the plan. I wish we had time for questions, but I think we're getting kicked up. Thank you everyone. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.