A Unified Future for Delta and Apache Iceberg™

Summary

Delta Lake and Apache Iceberg share the same data layer as of Iceberg v3, storing data in Parquet files with deletion vectors, but still maintain separate content metadata structures with different trade-offs for write speed versus read performance.
The proposed Advanced Metadata Tree (AMT) is a unified metadata format where Delta Lake commits store content metadata directly in Iceberg v4's adaptive metadata tree, giving Delta efficient tree-structured manifests and Iceberg interoperability while preserving Delta's transactional guarantees.
This video includes a live demo of the unified format in action and explains how Delta Kernel in both Java and Rust enables cross-engine adoption, with sub-linear write overhead and improved statistics pruning efficiency as the key performance improvements.

A Unified Future for Delta and Apache Iceberg™

Delta Lake and Apache Iceberg™ have converged on similar ideas: columnar metadata for efficient pruning, manifest trees for scalability, and deletion vectors for fast updates. Yet today, we maintain two separate metadata structures, duplicating work and diverging in capabilities. The next major evolution of Delta Lake reimagines metadata from the ground up. This talk introduces a unified metadata architecture where Delta Lake commits store content metadata directly in Iceberg v4's adaptive metadata tree. Delta Lake gains efficient tree-structured manifests and Iceberg interoperability while preserving the transactional guarantees that Delta users depend on. Join us to see the future of lakehouse metadata.
Talk By: Anoop Johnson, Principal Software Engineer, Databricks ; Micah Kornfield, Software Engineer, Databricks ;

Chapters

FAQs

What is the Advanced Metadata Tree and how does it unify Delta and Iceberg?

The Advanced Metadata Tree (AMT) is a proposed unified metadata format where Delta Lake commits store their content metadata — file lists, deletion vectors, and column statistics — directly in Iceberg v4's adaptive metadata tree structure. This gives Delta the benefit of Iceberg's efficient tree-structured manifests while preserving Delta's transactional guarantees.

What are the current metadata differences between Delta Lake and Apache Iceberg?

While both formats store data in the same Parquet-plus-deletion-vector format as of Iceberg v3, their content metadata layer differs: Delta uses JSON-based transaction log commits with columnar statistics, while Iceberg uses a manifest tree with map-based statistics. Each has distinct performance trade-offs for fast writes versus fast reads.

How does the unified metadata format improve performance?

The Advanced Metadata Tree improves statistics pruning efficiency and achieves sub-linear write overhead for metadata operations by replacing the linear scan of Delta's flat transaction log with a tree structure. The video shows performance improvements in both stats pruning speed and metadata write latency.

What role does Delta Kernel play in adopting the unified metadata architecture?

Delta Kernel, available in both Java and Rust, provides a single implementation of the Delta Lake reader protocol that any engine can embed, ensuring consistent behavior across Spark, Flink, DuckDB, and other systems. The Rust implementation in particular enables lightweight, cross-language adoption in environments that cannot run a JVM.

Full transcript

[00:07] So today we're going to be talking about the unified future for Delta Lake and Apache iceberg. Um I'm Mike Kornfield. This is my colleague Koop Johnson. We're both software engineers at Data Bricks uh working on Delta Lake and iceberg stuff. Um so before we dive into things,
[00:20] uh lawyers want to give a disclaimer that uh this is things might change. Don't be surprised if not everything here materializes but this is our best approximation of how we're thinking about unification today.
[00:32] Uh so the agenda for the talk um we're going to review uh do a review of how Delta Lake and iceberg formats work at a relatively low level. Um and then talk about what the future holds for medave
[00:44] unification. Um we'll talk a little bit about implementation details do a short demo and then um have some time at then for uh questions and answers. So um to get started, Noop's going to talk about uh Delta Lake and iceberg
[00:57] today. >> Okay, let's dive right in. So the high level the architecture for both table formats look a lot similar, right? So there are four layers in your
[01:09] typical data stack. At the bottom layer you have data and both formats iceberg and delta stores the data in paret files and if there are rowle deletes they are
[01:21] stored in deletion vectors. So with iceberg v3 this da layer is fully unified. So they are exactly the same on disk. This is a big deal because data tends to be really really massive and
[01:34] without unifying the data layer it's really hard to kind of have any sort of interoperability between these formats. So with iceberg v3 we got to unifying the data layer. The next layer on top of that is the
[01:48] content metadata. So content metadata is information about files deletion vectors and column level statistics. So currently iceberg and delta they have different designs for the containment
[02:01] data. So they roughly represent the same information but the actual the metadata representation is different. So we'll go deeper into this layer in in the coming slides. This is primarily the focus of
[02:14] this talk. Our thinking is that currently delta and iceberg have different pros and cons. We can come up with a unified metadata representation for both delta and iceberg that kind of
[02:26] improves the interoperability between these formats and also kind of fixes some of the existing issues in both formats. On top of this is the catalog layer. So there we have Unity catalog and then
[02:38] there is iceberg rest catalog. So Delta clients typically talk to the Unity catalog for catalog interactions. Iceberg clients talk to the iceberg rest catalog interface. In fact Unity already provides an iceberg rest catalog
[02:51] interface for if you have external iceberg clients talking to the table. On top of that you have the ecosystem. So Delta has multiple implementations of the protocol. So there is deltas spark
[03:03] which is the most commonly used form of delta and then we have the kernel which is a standalone implementation of the delta protocol and the kernel is implemented in java and also in rusk. Currently iceberg itself has multiple
[03:16] implementations. The most common implementation is the reference implementation in Java. This is what is used for spark and flink and other jvm based query engines. And also iceberg has implementations in other languages.
[03:29] There is Go, Rust, Python and C++. So the way we think about this unifying, it's very important that the data stays unified. So we already are at that
[03:41] state. The next thing to that makes sense is unifying the content metadata because the content metadata is fairly large. So think about information about your data files and statistics. On large
[03:53] tables, the content metadata itself can be very large in the order of gigabytes or hundreds of gigabytes. for very large tables. So because the the representation is different between both projects, if you want to bridge between
[04:06] these two projects, you have to do some sort of a translation layer currently. So Delta has a feature called uniform which exports Delta's content metadata into an iceberg compatible form. So this translation is not cheap. you have to
[04:19] run it and typically this layer translates makes a copy of the delta metadata incrementally into iceberg. So the focus of this talk is how can we
[04:31] come up with unified content metadata that both formats understand it's a same bytes on disk so that you don't have to pay for this translation cost.
[04:46] So let's dive into how delta lake works today. So delta is based on the abstraction of a transaction log. So a writer whenever it needs to make any changes to the table, it must describe what changes it's making to the table into a log. So it makes an append to the
[04:59] log saying, "Hey, here's what I changed on the table." And the log is stored as a bunch of JSON files currently. So if you open your cloud storage where your table is stored, there will be a
[05:11] directory called underscore delta log where all the logs are stored. So we'll start with a simple table. The writer is writing the first version of the table. You wrote some data into the table. Delta stores the the actual data as
[05:25] paret files on cloud storage. Then it writes a first version of the JSON file that says I added these data files XYZ paret and UVW parket and writes out this
[05:39] JSON. So if the right was successful, the the transaction committed, right? So the second writer comes in. So they're trying to change one of the they're trying to delete a small number of rows
[05:52] from this file that we added in the previous version, right? So Delta uses what's called deletion vectors. So deletion vectors are used for doing rowle deletes. So the idea is that if you're only deleting a small number of rows from a park file, you don't need to
[06:05] rewrite the whole file. Instead you can just represent this as a bit map that says what rows are alive and not alive. So the second writer says remove the previous version of the UVW and read
[06:18] that file with the new deletion vector. The deletion vector says here are the rows in the file that are deleted. So that change got written as the second version of the JSON file and this
[06:31] process continues. So new writer comes in any changes to the table they're committed as this JSON files everything you can think of each of these files as an append to the log and this kind of
[06:43] continues right so one ramification of this is that if you want to reconstruct the table state at any point of time you must read the JSON
[06:55] files the the delta client has to read all the files so for example if you are reading this table as of version 4 the delta client has to go and read four, three, two and one JSON files and
[07:08] reconstruct the table state. It goes through the log and it can figure out what is the table state. What are the files that that are alive on on the table at any point of time. So the downside is this log replay cost kind of
[07:22] grows as the as a log grows. So to mitigate this, Delta has a concept of checkpoints. So the idea is that you periodically create this checkpoint which kind of captures the full table
[07:35] state as a bunch of parquet files. So these paret files are containing metadata not data. Right? So in this case the writer who wrote 2.json might create a checkpoint for version two.
[07:49] Right? So the checkpoints contain the entire table state as all the as of version two. And then the future writers can add
[08:01] append to the logs as it happened today. So once the checkpoint is created the the readers can don't need to go and read the version one and two of the table anymore. They can just go to the
[08:13] checkpoint because that checkpoint is for version two and then they only need to apply the deltas that happened since a checkpoint. So checkpoint version is two. So you must read three and four and
[08:25] apply on top of the checkpoint. So this kind of bounds the the amount of work the readers have to do right. Um one the current delta checkpoints are
[08:38] full in the sense that a checkpoint itself may have lot of files and when whenever you create a checkpoint these files are not shared with the previous version right so for example let's say you are creating a checkpoint
[08:51] for version 200 it kind of reads the previous checkpoint applies the deltas and then reconstructs a new checkpoint so when you reconstruct the checkpoint the the entire parket files are new right So it's an expensive operation. So
[09:04] we don't checkpoint very frequently. We checkpoint probably based on the implementation somewhere between 30 and 200 commits been created checkpoints. So it's an expensive operation. The
[09:16] second issue is that checkpoints because they don't share any parts with the previous checkpoint will cause a full cache eviction. So some of the query engines will cache the metadata. So because a new checkpoint is generated
[09:28] that cached metadata is kind of completely evicted. So iceberg on the other hand does not have a concept of a log. The entire metadata is based on a tree based
[09:41] representation. Currently the t tree has two levels. So the root level is called what's called a manifest list file and the leaves are manifest files. The manifest list contains pointers to
[09:54] manifest files and the manifest contain pointers to the data files or deletion vectors. So we starting with the similar table structure. The first writer wrote the
[10:07] manifest list v0 which points to the manifest m0. And currently in iceberg all of the metadata it's all in avo format. That's has been iceberg the way it's been working from
[10:19] the beginning. So averro is row oriented unlike park. So the nice thing about iceberg is that the tree is incrementally maintained. Every writer produces a new tree but the
[10:33] tree shares the leaf manifest with the previous version of the tree. So every writer will produce a new manifest right and it always had to produce a new manifest list file.
[10:47] But as you can see the the leaf manifests that are common between the trees are shared. So in this case it doesn't need to rewrite the manifest m0 which which already exists. Another difference is that in iceberg
[11:00] there's a separation of data and deletes right. So for example and here the manifest contains all the data files the m0 and the delete manifest contains all
[11:12] the deletion vectors. So this process continues each version produces a new tree and it's all incrementally maintained and there's
[11:24] also the separation between data and delete manifest. So this brings us to one challenge iceberg has which is that if this during reads basically any reader will have to resolve the deletion
[11:39] vector corresponding to a data file. So in this case the way iceberg works it reads all the delete manifest prunes them out then builds an index of any unpruned delete manifest and then it
[11:51] streams through the data manifest files and attaches data files with the deletion vectors. So this is an expensive operation on and this is done during reads.
[12:09] So this continues every uh every version of the table has to produce at least a new manifest file and a manifest list file. So this is because the manifest files can point to data files. Manifest list cannot. Manifest list can only
[12:22] point to manifest files. So if you're making a small insert, iceberg has to first write a new manifest file which points to the new data file. Then produce the new manifest list file that points to the manifest files. So you
[12:34] need to do this operation serially because the manifest needs a pointer to the to the to the manifest files. So as a result iceberg the rights are relatively slow.
[12:53] There's also difference in how iceberg and delta represents statistics. So statistics are maintained for mainly for pruning files for a both formats maintain the min and max of columns for each file. So the idea is that you you
[13:05] can use this this min and max to prune out files that are not relevant to the query. It's the same idea but implemented differently in delta lake the statistics are typed. So for
[13:17] example, if you have an int column, the checkpoints will contain the lower and upper bounds as intel and it's a strct. In iceberg, currently the lower and
[13:30] upper bounds are serialized as a map representation and it's stored as bytes and the format interprets the bytes based on the scheme of the table. So this has one downside which is that if you are having statistics about several
[13:43] columns and you your query only touches a few columns the cl the iceberg library has to go and dialize the statistics for all columns while in delta because they're stored as strcts you can project out exactly what
[13:56] you want. Another difference is that how paths are stored in iceberg metadata. All the paths, paths to manifest files, pass to
[14:09] data files, pass to decision vectors, they're always absolute currently. So this is a focus they have been having which is that they focused more on data integrity from the beginning, right? So they have been very paranoid about
[14:21] having relative paths. But it has a downside which is that it's very hard to relocate an iceberg table. For example, let's say you have data in one region and you want to replicate that data into another region for disaster recovery
[14:34] purposes. All of the metadata is invalid because all the metadata points to the the files in the other the original bucket, right? Unlike delta paths are always relative by default. So you can
[14:46] simply copy a delta league table into a different S3 bucket and everything just works. So if you have to do a scorec card, you know, here are the things that Delta
[14:58] optimizes. Delta is optimized for fast commits because to make changes, all you have to write is write a simple JSON file and that's a very fast operation.
[15:10] The checkpoints are columnar and the types are st the the the bounds are typed. So this is very efficient for retrieving only the bounds that you're interested in
[15:23] because iceberg has the combination of a row oriented avoiles and also the map which makes it hard to retrive exactly what you want. The other nice thing is delta the data
[15:35] files and DVs are one record. So in a checkpoint file that says here is the data file and here is its corresponding DB while iceberg has to do this join between data files and DVs and so that's
[15:47] great for read performance but delta also suffers from some read performance penalty because of the log replay but the log replay has to be done by the reader right and sometimes the logs
[15:59] there can be a lot of logs that have not been checkpointed yet and that can cause some performance issues so we currently mitigate this by doing a distributed replay in the sense that reading all these delta logs dealing applying them
[16:12] on top of the checkpoints it's all done as a distributed spark job. So we kind of mitigate this by kind of having the full power of spark kind of do this metadata resolution. The second problem is this checkpoint
[16:24] amplification problem that I mentioned because checkpoints are full and it has to kind of create this entire table history whenever we create a checkpoint and it's has all the problems with not being very cach friendly also. Another
[16:37] problem is that the changes are always described as in in the log using a JSON file. So imagine that you have a very large DML job that changes large part of the table. Delta always have to kind of
[16:50] materialize the change as a large JSON blob and this is not exactly paralyzable.
[17:03] So iceberg the trade-offs iceberg make currently they're almost inverted in the sense that there is no log replay cost right because it's always the tree but again there's a problem with read performance because we have to do the join between data and delete manifest.
[17:17] It's great for bulk rights because if you're making changes to the table that affects large number of files you can always paralyze this operation. you write out multiple manifest files and then write the the manifest list file.
[17:32] So the weakness is that you have to write multiple files even for small changes. Let's say you're inserting one row into the table. Iceberg has to first write a new manifest file that contains
[17:44] the pointers to the new data that has been inserted. Then write the manif manifest list file and then call the catalog about the new manifest list file. the read performance. There is a problem with the two-phase planning that
[17:56] I mentioned earlier. The other issue is that it can only the manifest list only contains partitioned summaries. So if the table is partitioned, it can prune out manifests but not based on let's say
[18:08] other bounds, right? Non-partitioned column bounds and the there's no way to prune out um columns also for pruning, right? Because the everything is stored as a map. So
[18:20] you have to read the whole map and descalize it. So this gets us to where Delta and Iceberg are today. And Micah is going deeper into how we are thinking about evolving both formats and how we can
[18:32] improve them both. >> Um yeah,
[18:44] if we uh reconsider how the metadata is going to be organized for um iceberg and delta. So um the key concept that uh we're trying to introduce in iceberg v4 there's been a lot of discussion on the mailing list already about this and um
[18:58] uh an has actually started opening up some prototype PRs for this is uh what's called the adaptive metadata tree or um AMT for short and um there's two main changes from the current iceberg uh
[19:10] format that um exists today and kind of re going over the same example that we've been going over for the same time you kind of have a starting tree. And the main differences are one at the
[19:22] root, it's no longer a separate data structure manifest list. We're just going to have another manifest as the root. And we'll see why that's important in a minute. And then um the other big difference is that instead of using AVO,
[19:34] we're going to start using parquet in iceberg. And so um what makes it special to having a manifest as a root is it means you can now put data files directly in the
[19:46] manifest. So um in the previous example, this is adding a DV to the file UVW. So what ends up happening is we end up um being able to add the deletion vector
[19:58] info directly to the root. We still don't need to rewrite the um leaf manifest. Instead, um the AMT also introduces a concept of uh uh manifest deletion vectors. And this is very similar to your data deletion vectors
[20:11] where if you're only changing a small part of your manifest, you don't need to um rewrite your entire manifest to remove a particular entry. You can just update a bit map which is stored in the root manifest to uh remove those files.
[20:24] So um that reduces uh you know if you're doing lots
[20:36] operation because you can do that all by just rewriting the root. Um so again this process continues and now you know a smart iceberg writer might realize I only have two files in my table. I don't
[20:48] really need a leaf manifest. So as I remove all the files from the original leaf, I can just keep it in the root, which means that I'm only writing one file uh for content metadata on any
[21:00] given commit. And again, this process continues. We can just add a third file to the root and it all um stays nice and compact for reading things back. So one thing that if um you're kind of
[21:14] paying attention uh sorry one other point here is that in iceberg v3 we are going to start colllocating the data files with the DV information. There's not going to be separate manifests or for um storing the DV information. And
[21:27] so if you're paying attention that's actually one of the relative strengths of iceberg because um you know a lot the data file is where the statistics are stored. So if you um have to rewrite uh your data file each time or move that
[21:41] around, you're rewriting your statistics and that causes um larger write amplification where existing iceberg if you're only updating DVs, the DVs are smaller, right? They're only really just pointers to what the deletion vector is and the referenced file. And so the
[21:54] solution for this in the AMT is for larger updates, we're going to support um logical column level updates. So if this is um on the bottom left, if this is your manifest leaf that has um
[22:07] deletion vector info in it already um and you have a root pointing to it, what we're going to be able to do at the root is say the DV info for this um file is no longer up to date. I'm going to write
[22:19] out a new paret file with new um puffin is the format that DVs are stored in with new DV information. And then at read time we can do a um zip join. So we this
[22:33] file this DV entry now applies to this file. So it's cheaper in the general sense than having to kind of do the hash joint that I mentioned for um when you're doing v3.
[22:49] And then um so mentioned before we're moving away from avo for um manifest files. Instead we're going to in parquet and what that gives us is kind of the same benefits that exist for um delta checkpoints. We can the organization's a
[23:02] little bit different but essentially we have a strct inside that strct we have strcts for each column and then inside of that we have the columns for the min and max values for each of those columns and the other statistics that iceberg
[23:15] stores. So that gives us um the benefits that we get from delta in iceberg where you can project just the statistic statistics you need when you're applying predicates.
[23:29] Uh so to kind of summarize what we're seeing uh for AMT improvements for iceberg uh we can use stats for printing manifests that um that's another advantage where now that we have a manifest at the root we can store aggregate statistics for each of those
[23:41] manifests. So we never need to open up a manifest that can uh certainly not contain it if like the max value is above uh a min value that we asked for. um we have lower read amplication for
[23:54] retrieving statistics because it's stored in a columner or parquet file and uh at read time we're avoiding the um kind of more expensive uh general join of data with their uh deletion vectors.
[24:07] Um and then we also get lower right amplification for removes and DML operations because we can uh store things both we can store things in the root and um when we want to just update
[24:19] remove things from the leaf we can uh update bit maps instead of having to rewrite them at any given time. Okay. So when we think about use cases and like operations that we're going to
[24:31] perform on the table, what this ends up looking like is smaller pens will uh can be collected in the root. So you get a single file, right? So they're nice and quick. They don't ne they scale kind of with the size of the commit you're doing. And then as the root grows too
[24:44] big to kind of be manageable as you um to like read and rewrite each time, you can flush things to the leaf in a more stable state. And then we maintain the existing benefits of iceberg where larger appends you can directly write a
[24:56] leaf node and you can even write those in parallel so that you um can uh not have serial rights to the object storage. um you know for data expiration and bulk deletes we have a nice property
[25:09] of it can be a root only update so we don't have to uh rewrite a bunch of files and then for DML for small DMLs uh we can do um we can remove uh we we can
[25:24] use manifest DVs to remove things at the leaf and
[25:37] com level updates to avoid the uh right amplification. So this brings us to um what this means for delta and how do we get the benefits of the AMT in delta. And the main the
[25:52] main thing that gets introduced for delta is um previously when you wanted to add or remove content metadata um you had to have individual add and remove actions in the actual delta log. But now
[26:05] what we'll have is a new action called a checkpoint action. And so this makes checkpoints a synchronous process when we're committing but that allows us to point directly to an AMT. So here we see we have the root manifest in parquet and
[26:18] the existing leaf manifests and then on an update we can go ahead and just update the AMT and have a new pointer uh checkpoint pointer there. Um again we
[26:30] have the benefits of manifest TV for not rewriting and then we still maintain can still maintain the benefits of delta where we don't need to update the AMT every time. we can still do traditional delta log
[26:45] writes. And so this process continues um with another delta log write and then replay stays largely the same except the checkpoint is now pointed to directly in the log versus as a um kind of sidecar
[26:58] pointer in the delta format. And kind of the key thing to reiterate here is this AMT is identical to the iceberg AMT. So that means that when we're going to we want to go ahead and
[27:11] um share data between readers and writers uh either side can understand this format.
[27:26] So uh when we talk about compatibility options um there we're thinking about having two options in Delta. There's just an AMT enabled option which is what we showed on the last slide where you maintain the ability to have both delta log files and um uh checkpoint uh
[27:40] actions that point to um new AMTs and then an iceberg native V4 option which is similar to the existing um like iceberg compat options in Delta but in this case it will force updates to the
[27:54] AMT on every commit. So that means for things like uniform you always have up-to-date table state that can just be shared at uh zero cost with um iceberg.
[28:09] So the use cases look very similar for delta um the main differences and end end up here. And to reinforce the points is uh for small pens we keep the ability to use delta files um or which are still a little bit cheaper than rewriting the
[28:21] AMT route each time. we especially for small ply depends you don't need to um ever read the root you can just add those as a delta file and then the other big thing that this gives for um delta is when you're doing bulk deletes and
[28:34] updates um as new pointed out you don't need to write a big JSON file that has the list of all the files that you're deleting that becomes an a metadata operation with the root
[28:50] so in summary what we're looking for for delta v5 um is we're going to unify um the content metadata for compatibility. Um this kind of builds on top of the unification of the data layer um for
[29:02] compatibility that was done in iceberg v3 and delta v4. And we kind of get strictly better um table formats for both iceberg and delta where we get metadata skipping for like aggregate
[29:15] stats in the root. Um less log replay, better caching of the checkpoints. Um and then uh better write performance uh for some cases or no degradation right
[29:27] performance depending on what mode you're trying to use. Um so now I'm going to talk a little about uh how we're thinking about go going ahead and implementing some of these changes especially on the delta side. On
[29:40] the iceberg side, um we're working with community and like I said, we've already opened up some poll requests to start implementing the AMT. Discussions are still go like nailing down all the final details of the MT, but it's in a good enough state that um we can start
[29:53] building it there. So from a Delta perspective um you know historically uh Delta has been very tightly coupled to Spark and one of the things that um some of our colleagues have been working on and also members of the open source community is uh what we
[30:07] call delta kernels and the idea behind it delta kernels is they take very few dependencies or almost zero dependencies on any particular engine. So if you start thinking about how engines um
[30:20] understand table formats, you kind of have the engine sing on top and all the engines have some sort of uh pluggable interface for how you actually read data. So in Spark, what this looks like is, you know, Spark is the engine and
[30:33] then Spark has actually gone through two different iterations. There's data source v1 and now data source v2 um that allow you to uh integrate with external file formats. And so in this model um
[30:45] DSV2 you can write a new DSV2 connector that would make use of kernel and we're actually working on that right now um in uh the the the delta community. So that
[30:57] um you know the spark specific bits that do the bridge between tsv2 live um in spark and the delta kernel is really just this core notion of what the semantics of a um delta table look like.
[31:11] Uh there's currently Java and Rust implementations for Delta kernel and so for instance Apache Flink uses the Java implementation uh for doing its connection that that's something that uh was recently released
[31:23] and then uh other databases like click house and ductb have gone ahead and used the rust kernel for their integration and um for doing that and then um one of the things we're even trying to simplify further so that we're not duplicating
[31:35] code and like discovering bugs because implementations don't match is we're considering ing uh having Java just bind directly to the Rust implementation for uh making this work.
[31:49] And so what this means for V5 is um you know there's some substantial changes uh to the metadata layer but we're uh keeping the APIs on the kernel
[32:01] identical from a uh connector perspective when it's wrapping the kernel. So that means that especially for reads and to some extent writes, if you're using kernel already, you'll just be able to go ahead and use the same
[32:14] APIs. And if the table happens to be using the AMT, you'll um get those things for more or less free. Um because the MT is more complex, there might be some things that you want to tweak and control more directly. We're also going
[32:26] to be adding some more advanced APIs when you are working with AMT. So the key takeaways here is please if you're trying to integrate with Delta Lake um use kernel. It will help a lot with any of these migration scenarios. And then
[32:39] um you know we're we're doing this ourselves with the SV2 which is going to start using uh kernel. So now it's time for a little bit of demo to like make some of this stuff a little bit more concrete. And I am not
[32:52] brave enough to do a live demo. So I pre-recorded uh this demo and uh going to walk through it um a little bit slowly. So um here hopefully everybody's familiar with uh you know a data bricks notebook. We um the main thing to notice
[33:06] here is we're going to run SQL and like when we create the table we're creating an iceberg table and we can say it's format version v4. And so if we start playing this will go ahead and run and then we're going to insert a couple of
[33:18] more values into the table that that succeeds and then we're going to start kicking off some queries. Wait for them to return. We see we get back the expected result. Um get 300 and we
[33:32] have 102 entries because we did create table as select and then insert another couple of values. And now here um ex you'll have to excuse my uh poor shell script. uh code. But um this is the
[33:45] story for like if you have other engines that you want to use. In this case, we're using Spark um that uh you know with open source uh Delta extension the
[33:57] key thing here is we're loading the Delta Spark extension. Now we're configuring the catalog to use the Delta catalog and um this is going to be talking through UC in uh DBR, but any catalog that you want to use with Delta
[34:11] should work. Um and then similarly uh we um have similar code for iceberg. And here again we're just using the um
[34:24] vanilla iceberg uh extension. So we're loading up the iceberg spark session catalog extensions. And then we have um using the iceberg spark catalog plugin. And then I created a quick shell script
[34:37] to do um some probing of the table and some inserts. So, and the the use delta variable indicates whether we're going to use the delta writer and reader or the um iceberg writer and reader.
[34:50] And so, when we kick that off, we can see this uh go through its process. It finds the same 300 we got back on the previous screen. It inserted four values
[35:02] and when we try read back, we get 10,000. And then we go back to DBR and it just works. We get see the 106 values that are now there and we see the updated values that were inserted from
[35:14] uh iceberg.
[35:29] And now so to kind of summarize um and we we keep on uh repeating the same points but I think the important thing to take away here is that uh we are not compromising on like technical improvements to either Delta Lake or iceberg. We're making strict
[35:41] improvements to both formats and um at the end of the day this gets us much closer to having um very few differences between Delta Lake and Iceberg for anybody that cares about the uh
[35:54] distinction. Uh one last note before we do Q&A. Uh please if um you can fill out feedback for us um on the talks, it would be greatly appreciated. Uh we value all the
[36:06] feedback so we can shape uh talks for future sessions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.