Skip to main content

The Road to Delta 5.0

Summary

  • Delta Lake launched as an internal Databricks project in 2017 under the codename Project Tahoe and has grown over nine years to 40M+ monthly downloads while powering hundreds of exabytes of data processing daily.
  • Delta 5.0 introduces a catalog-first architecture that places Unity Catalog directly in the commit loop and modernizes Spark integration through Data Source V2 APIs for native filter pushdown and predicate support.
  • The path to Delta 5.0 converges the Delta and Iceberg formats under a unified commit and checkpoint structure while harmonizing the Delta Kernel across Java and Rust implementations for broad engine compatibility.

The Road to Delta 5.0

Watch: The Road to Delta 5.0
Marking its ninth anniversary this year, Delta Lake has grown significantly since its 2017 debut, spanning four major versions and 33 individual releases. As a premier ACID-compliant open table format globally, it sees 40M+ downloads per month and powers the daily processing of hundreds of exabytes of data. While the current roadmap includes the recent launch of Delta 4.3, the project is undergoing foundational shifts as it moves toward Delta 5.0. this video explores these paradigm shifts through two primary lenses: Evolution and Unification. Key topics will include: The transition of Delta into a catalog-first table format. Modernizing Delta on Spark by leveraging Data Source V2 APIs. The convergence of the Delta and Iceberg formats. Harmonizing the Delta Kernel across Java and Rust implementations.
Talk By: Nishith Agarwal, Sr Manager, Engineering, Databricks ; Rahul Potharaju, Director of Engineering, Storage, Databricks ; Tathagata Das, Sr. Staff Software Engineer, Databricks ;

Chapters

FAQs

What is Delta Lake 5.0 and what major changes does it introduce?

Delta 5.0 is the next major version of the Delta Lake open table format, introducing a catalog-first architecture that integrates Unity Catalog directly into the commit loop. It also modernizes Spark integration through Data Source V2 APIs and aligns the Delta and Iceberg formats under a shared commit and checkpoint structure.

How does the catalog-first architecture change how Delta Lake works?

In the catalog-first approach, Unity Catalog becomes part of every Delta commit, ensuring that metadata governance is enforced at the storage layer rather than as an external overlay. This moves catalog awareness from an application-level concern into the core of the table format itself.

What is the Delta Kernel and why does it matter for engine compatibility?

The Delta Kernel is a shared library that abstracts protocol details so that any query engine can read and write Delta tables without re-implementing the full specification. Delta 5.0 advances the Kernel with implementations in both Java and Rust, maximizing cross-language and cross-engine compatibility.

How are Delta Lake and Apache Iceberg converging in version 5.0?

Delta 5.0 moves toward a unified commit and checkpoint structure that aligns with Apache Iceberg, enabling the two formats to share metadata representations. This convergence reduces fragmentation in the open table format ecosystem and simplifies multi-engine deployments.

Full transcript

[00:08] Thank you for coming to our talk here today. Nishit and I are going to talk about what is happen going to happen in Delta Lake over the next 1 year or so. So, let's just jump right in. Let me start by introducing myself. I am TD. Everyone
[00:22] calls me TD. Uh I've been having fun with Spark and Delta. Uh I've been involved in as a core developer of RDDs, DStreams, Spark Structured Streaming, Spark Streaming, Delta Lake, Unity Catalog Oasis. It's
[00:36] been a wild. I've having fun for and still continue to have fun even after 16 years in the Spark ecosystem. Uh and Nishit.
[00:50] >> I'm Nishit. Um yeah, I've been also working on data platforms for uh a long while. Uh love open source. Been an open source enthusiast and uh been a long-time contributor as well. And uh I'm a senior engineering manager at Databricks.
[01:06] >> All right. So, uh what are we going to talk about today? So, before we get into the actual meat of the talk, I just want to remind where how far Delta Lake has come. Delta Lake is actually now 9 years old this year.
[01:18] So, we started in 2017 as a uh crazy internal project in Databricks. It's called It used to be called Project Tahoe. And you'll see actually the remnants of that name Tahoe in the code base. There are things like Tahoe file
[01:30] index etc. that are still around. Uh and the reason it was called Lake Tahoe is that we wanted to build a gigantic data reservoir. And us being in California, why not Lake
[01:42] Tahoe? Lake Tahoe is actually one of the deepest alpine or largest by volume alpine lake in US. And if you take the water that is uh in Lake Tahoe and you spread it all over California, it'll be
[01:54] 14 ft deep. It's that much water. So, yes, we wanted to make it gigantic data reservoir, and that's why Project Tahoe. And then in 2019, we actually open sourced it, which was
[02:06] amazing. We have come a long way since then. We have made lots of releases, gone through four major versions, Delta 1.0 where we were the first one to introduce SQL DML
[02:18] operations um in Delta even before Spark supported it. We actually we didn't have SQL support, so we actually went out and built Scala and Java and Python APIs for doing
[02:30] delete, update, merge because Spark did not support it in SQL. So, we did those kind of crazy stuff. We supported schema evolution. We introduced all of
[02:44] We introduced some of those
[02:56] 2.0, we introduced column mapping, separation of logical and physical names. We introduced compaction and Z-ordering as way to cluster data. We introduced change data feed. 3.0 was
[03:09] another level of crazy. We introduced deletion vectors, again inspired by Hoodie but done in a slightly different way than Hoodie's MOR. We introduced Uniform, which is again the first time anybody could transform
[03:21] one table format to the other from Delta to Iceberg. And we also introduced liquid clustering, the next iteration over Z-ordering and stuff. And last year we introduced Delta 4.0, which we introduced we introduced variant, again some of the
[03:34] first systems that supported variant all the way between Delta and Spark. We introduced Delta Connect. We introduced type ordering, all again groundbreaking stuff.
[03:50] As of yesterday, we released Delta 4.3. So, what's in Delta 4.3? We have uh Unity Catalog Managed Tables. We're going to talk in more details about that in the rest of my talk, so I'm not elaborating that here. We introduced geospatial support, which is
[04:03] again pretty groundbreaking. This we you can directly do geometrical and geographical operations using Spark and Delta as a first-class citizen. Uh you introduced proper support for schema
[04:15] evolution in the SQL syntax itself. Insert instead of no more setting uh SQL configurations or session configurations that affects everything that running the sessions. Here there's direct support at the query
[04:27] level. We also did dynamic data override, so more variations of DML operations uh in Spark. And so there's a bunch of other features in Delta kernel and stuff as well. I'm not going to go into that
[04:40] level of details because this is not a Delta 4.3 talk. What rather I'm going to focus we are going to focus on is where are we going from here?
[04:54] We are 9 years old. Next year is going to be 10 years. So we are thinking ahead that there are Next year we want to actually release a Delta 5.0. Nice timing and all. Um but looking zooming out, there are a bunch
[05:06] of themes that are going on in terms of foundational changes in Delta. Even after being here for 9 years, there's still foundational changes happening in Delta. And two themes that I'm going to
[05:18] talk we are going to talk about is evolution and unification. Let me start with the first step, the evolution. The two themes two sub-themes on the evolution. Let me start with one. So, evolution one, Delta is now catalog
[05:32] managed. Since the beginning of Delta, since 2017, Delta has been a file system managed Delta table. Where you Delta always committed using file
[05:45] systems put if absent. For those of you all remember, there was a long time where Azure supported for put of absent GCP GCS supported put of absent S3 not supported put of absent until couple of years ago. We had to
[05:59] find other ways of support figuring out put of absent file system to make it work with S3. We use DynamoDB as a a commit coordinator and stuff like that. So, we had but it was purely a file
[06:11] system managed Delta tables. Even if you had a catalog that actually um That doesn't show. Even if you had a catalog that actually just simply pointed to a Delta table
[06:24] path catalog had no other role to play other than just having that pointer to the path and then everything happened at the file system level. So, what was the result of that?
[06:37] So, all since all updates to the table were happening at the file system level, the catalog was never in the loop. So, the catalog did not know what is the current state of the table is.
[06:49] For example, if you you went ahead and change the schema of the table, you just committed to the file system catalog is out of the loop. Yes, some connector may choose to update the catalog lazily after the update has happened. But
[07:03] there's no guarantee. So, catalog information or the what are the what is the schema of the table, what's the properties of the table may be stale. You cannot trust the catalog. Which means that
[07:17] if catalog doesn't know the source of truth, you cannot have the catalog be the governance mechanism in a well enforced way. Because things like uh A back fine A A back policies, fine
[07:29] grained access control, data masking, these sort of things requires the catalog to know exactly what is in the table and be up-to-date be the source of truth. That was not possible.
[07:44] So, over the last couple of years, we redesigned the core principles in Delta and made Delta a first-class catalog managed table. So, in which basically the Delta connector, instead
[07:56] of committing to the file system, will ask the catalog, whichever is owning the Delta table, ask the catalog to do the commit for it.
[08:08] So, the catalog is always in the commit loop, commit path, which means the catalog is always up-to-date on what is going on. Either the catalog agrees to make the changes to the table
[08:20] and updates itself, keeps the schema properties, whatever the catalog stores up-to-date, or the catalog says that I'm not going to commit, so nothing has changed neither on the file system nor in the catalog.
[08:33] So, catalog is the source of truth then. Now, how is this actually built underneath? The way we have designed this is in a pretty flexible way. They're two sides of the same coin. One side is from the Delta protocol
[08:45] perspective. For those of you who don't know, Delta protocol spec is defines what is stored on the file system, what is persisted in the file system. So, we have introduced this new feature
[08:57] in the Delta protocol, which marks the table as this table is managed by the catalog, do not commit to it via the file system. So, all connectors that respect the protocol spec needs to know that, "Oh, I
[09:10] cannot commit directly to this file system. I need to find out what the owning catalog is." Which incidentally, if it is actually a table that is pointed to by the catalog, the connector already has done the work of going to
[09:22] the catalog, figuring, "Okay, here's the table name, select star from table name." And figured out the path from the catalog, so it knows what catalog it's talking to. And so, it knows how to talk to that
[09:34] catalog to actually do the commit. So, so that kind of naturally fits in with the catalog. It's just the role of the catalog becomes more important. But, from the Delta protocol perspective,
[09:46] what's on the file system, Delta does not care which catalog it is. So, there's a flexibility that you can any catalog as long it supports committing to Delta tables can become the managing catalog for
[10:00] Delta tables. So, Delta is flexible in that way. The other side of the coin is that means the catalog's responsibility, a particular catalog who has to support these sort of managed Delta tables need to provide Delta specific APIs for Delta
[10:14] connectors to do the commit. Incidentally, with Unity Catalog open source, we you you Unity Catalog both open source and Databricks Unity Catalog as of yesterday supports these special UC
[10:28] Delta APIs that we've introduced. So, for those of you who are familiar with Iceberg, this is very similar in the veins with IR Iceberg rest catalog. We actually took that inspiration from them. The few things that Iceberg actually got right in terms of
[10:41] managing things in catalog. We're all learning from each other. So, yes, we took inspiration. We designed this UC Delta APIs that is specific to the Delta protocol and we made Unity Catalog the first
[10:54] catalog that supports this. So, what this means that UC managed tables is a concept in UC. It has a separate spec of its own. You can go to the Unity Catalog repository and read that spec.
[11:06] And that UC managed tables are Delta tables underneath. And this is how both the Delta side and the catalog side works together to form a catalog managed table
[11:19] specific to a catalog. So, you can try this out right now. As of yesterday, we released Delta 4.3 and UC 0.5. So, there's a UC 0.5 server which supports this UC Delta API.
[11:31] The UC Spark connector and Delta Spark connector all work together to get the catalog managed tables working seamlessly with Spark just like any other type of table just like path based
[11:43] tables and stuff. Try it out. It's fully compatible with Databricks UC. Um and the next call to action for all Delta connectors out there with the developers of Delta connect out there
[11:55] that beyond Spark we want all engines which currently support Delta but on path based tables or just simple UC external tables where there is no this sort of commit management by UC to
[12:07] build support for these new UC Delta APIs cuz that's the future where Delta is heading towards. Uh Delta and UC together is heading towards.
[12:20] That was evolution one. Clearly very foundational for last 7 8 years Delta was on the file system. Now it is getting catalog first. Second, we're going to talk specifically
[12:32] about how Delta interacts with Apache Spark. So again for the last 8 9 years Delta's architecture on Spark has been a very specific style. So if you
[12:44] dial really time travel back to 2017 when we started Apache Spark had this data source API called well data source V1 or DSV1 for short which was a very simple interface
[12:57] for connectors to say here is how I can read and write data using Spark. Uh and there the since it was the first situation it was a very simple API there was no
[13:10] concept of data skipping. Every time you have to read the data it'll read all the data and that was pretty simple. Which led to a huge proliferation of the number of connectors back in 2016 2017.
[13:22] Everyone started implementing connectors on Spark because of the simplicity of that API. But Delta, which started simple as well, it got asset compliance, reads and writes, etc. on it.
[13:35] Uh but Delta wanted to do something more. Delta wanted to do introduce advanced data skipping using column min maxes and stuff. Which DSV1 did not have first class support for it.
[13:48] So, there was no direct way of telling of Delta telling Spark that give me the query filters using which I want to I can skip data. Delta always So, if you
[14:01] use the basic DSV1, you will always read the entire table. So, what Delta did is to solve this challenge, what Delta did is it went outside the DSV1 API, kind of like a side channel,
[14:13] and it introduced special rules in the Spark uh analyzer to identify Spark plans, extract the query filter from the Spark
[14:26] plans, and use that query filter to do data skipping using Delta's column stats and stuff. Which is really going like outside uh outside the DSV1 APIs and stuff, and the
[14:39] the the the side effect of that is Well, there's upside of that you get awesome data skipping, and that's why we have had great performance Delta in production for since the beginning. But the downside of that is that Delta is
[14:51] hooking into the internal details of Spark, like internal Spark plans, etc. It's hooking into that. There is a lot of tight coupling. Which is not great. It's like it violates all sorts of basic software
[15:04] engineering abstractions and stuff, but that's was the only way possible before. And but we So, we did it at the cost of our maintenance burden, because this constantly broke, and it prevented uh
[15:16] things to evolve well. For example, another example is that we implemented merge as I was telling a little bit earlier. Like we implemented merge um in crazy ways even before Spark supported merge
[15:28] uh your your officially. We had to do introduce custom rules, custom commands, etc. for making it work, which is awesome for end users, but downside is more tight coupling hooking into the internals of Spark. Which means that
[15:41] more chances of ossification that Spark can't evolve well and neither can Delta. So, what Spark did is that Okay, we in in back in 2017 we
[15:55] started to be w- Delta to be much smarter than what Spark natively supported, which was great. But Spark evolved. Over the last 9 years Spark has evolved. Spark has built a DSV2 API, which is smarter,
[16:10] allows more s- connectors to do smarter stuff. But Delta has been stuck on the DSV1 API. So, and now we're the last couple of years we're hitting the point that Spark's Spark's DSV2 API and what Spark
[16:22] natively supports using this D- DSV2 API is getting smarter than where Delta is. So, Delta needs to evolve. And that's what we are doing. So, what earlier, since we were having to bypass the DSV1 APIs to do fun stuff,
[16:37] now everything is natively supported in DSV2 and more. And so we have as we're evolving this, let's look at how certain things how we did in the past changes. So, you will think about data skipping
[16:49] once again. We're going outside the DSV1 APIs for to extract to read the entire query plan of Spark trying to pattern match that okay, here is the query filter, etc. outside the DSV1 API,
[17:02] right? Instead, with DSV2, Spark does its own job of it knows what the plan looks like, its own plan. It knows how to extract its own filters. It just gives
[17:16] those filters to the connector through the DSV 2 APIs. And the connector, which is Delta, the our V2 connector does not need to hook into understanding internals of Spark. It just says, "Okay, here is a filter. I
[17:29] just need to use this filter." Clear separation of responsibility, clear separation of what the storage cares about is the storage, and what the engine cares about is stay stays in the engine. No mixing
[17:42] of blue and red boxes. So, similarly, there's lots of stuff in DSV 2 APIs, direct support for Delta delete update merge, change data change data feed, transaction
[17:55] management, a bunch of stuff continuously happening. Spark 4.2 is in I think RC2 or RC3 right now. So, hopefully in the next couple of few weeks Spark 4.2 will be released, which will have a lot more DSV 2 stuff. It's a
[18:08] continuous evolution, and Delta is co-evolving with Spark to build all the functionality in the best possible way.
[18:22] And the way this is controlled is that there is a SQL configuration that is now introduced in Delta. You can try it out yourself. By default, it is the SQL config controls whether it goes through the V for every operation it whether it goes through the V1 stack or the V2
[18:36] stack. And so, by default the SQL configuration is in auto mode. So, Spark Delta will intelligently decide which one is ready which operation is ready to go through the V2 stack. For example, Delta 4.0
[18:49] that came out yesterday, streaming reads on UC managed tables uh goes through the new V2 stack. Uh and as we are building making sure this new V2 stack is feature and
[19:02] performance parity with the V1 stack, we'll through the releases we're going to allow more and more stuff to go through in V2. And the goal is that by Delta 5.0 everything should be in V2.
[19:16] That's the goal. Um so yeah, you can try it out yourself. Um there's more stuff going to happen on every Delta 4.x release and by Delta 5.0
[19:29] hopefully everything is V2. Now, here's the next question. See, basically we're rebuilding the entire Delta connector on Spark. If you're rebuilding the entire connector Spark, there's an opportunity of for
[19:42] fixing a bunch of other stuff. What else can we fix? Nishith. >> Fix stuff. >> Thank you, TD. I have the clicker. All right.
[19:58] So, I think TD touched upon how we're evolving the stack. Um let's talk about uh how we would want to unify the stack as well. And as part of that, I want to talk to you about how we're using kernel to evolve the Delta ecosystem and evolve
[20:11] the support and integrations that Delta has. So, let's look at a little bit of the journey um of the Delta protocol. So, it started with a authoritative
[20:23] implementation in Spark. And it continued that way for a while until, you know, other engines and integrations came about. As as they came, more in implementations emerged.
[20:35] And as part of that, uh there was a Delta standalone library that was built in order to serve as a common library to be able to integrate the Delta protocol with other engines. Sadly, the abstractions were hard to
[20:47] build and it ended up leaking a bunch of protocol details. And thereby it made it hard for engines to actually integrate into the uh into the standalone uh library.
[20:59] So, that didn't survive too for too long. So, we said, "Let's continue this journey and let's figure out and reimagine a way to still build in narrow ways a set of table APIs
[21:12] that hides all of the protocol details, right? Um and still allows engines to be able to be flexible. So, engines can do whatever they do best, but protocol is hidden from the engines.
[21:24] And that's what we did uh with kernel. Right? It's a library that hides all of the protocol details and allows you to plug in so that you get instant Delta support.
[21:40] So, as we've been on this journey, um obviously, you know, we want to support the entire ecosystem. And it turns out there's obviously a bunch of connectors in the Java ecosystem. And then there's a bunch of engines and connectors in native and Python ecosystem.
[21:52] So, naturally, we ended up building two sets of libraries, one in Rust and one in Java to support all of the ecosystem.
[22:06] The current state The great thing is um you know, the the kernel has actually helped solve a lot of the connector problems on you know, how to integrate with Delta. If you look at the ecosystem, Flink, uh Confluent, DuckDB, ClickHouse, Delta RS,
[22:20] um many popular libraries integrate with kernel to add Delta support. Within Databricks, we've seen massive adoption. Kernel is actually the de facto library to go in for any service within
[22:33] Databricks to go speak Delta. Actually, over the last couple of days, many of the keynote announcements that you heard uh the products are powered by kernel. Redeon is powered by kernel. Zero versus powered by kernel. AI
[22:46] Gateway is powered by kernel. Uh open sharing powered by kernel. Um so, there's already massive adoption and quite a lot of hardening that has happened over the years.
[23:00] Um so, now we have this dichotomy. You know, Delta Delta on Spark used to be the authoritative implementation, the most trusted reference, which is what people would go and say, "Hey, am I following the Delta protocol or not?"
[23:13] Um, and now we have this new implementation, Kernel, which is becoming the de facto implementation. Um, what do we make as the source of truth? Uh, this goes back to what TD was talking about. We're rebuilding Delta's
[23:27] you know, Delta Spark and building all of that on the connector. And what we want to do is build a connector on Kernel. And that's what we've been doing. The the new DSV2 connector that now is
[23:39] available in 4.3 uh, is powered by Kernel Java. That's great. We're able to centralize some of all of this stuff under Kernel. But we still have a problem. Now we have a Java implementation and we
[23:52] have a Rust implementation. You know, these are independent projects, evolve independently over time. Protocol implementations can differ. And if you've dealt with if you as a developer if you dealt with Java and
[24:05] Rust the way to develop is very different. So the APIs, the the nature of like thinking about APIs and abstractions is different. And that leads to a challenge of you know, not having the same feature parity across the libraries. And we want
[24:18] to solve for this. The good news is there is a solution in the works. So we already have Kernel Rust, right? It powers it's the core that powers a
[24:30] bunch of Rust integrations, but also powers a lot of native engines through FFI. We want to use the same core to power our Java ecosystem. We want to build on all of the amazing
[24:42] solid foundations in JDK such as JNI and FFM and provide a Java layer that directly builds on Kernel Rust. And with this unified Kernel
[24:54] you'll have one central Kernel implementation living in Rust and all the connectors can enjoy all of the features that come as part of kernel Rust. It actually drastically solves a lot of the pain points and overhead of maintaining two libraries,
[25:07] but at the same time, it enhances feature development speed, which is awesome.
[25:19] And that's that's kind of like, you know, going back to the evolution and ecosystem merger, right? Like we have Delta Spark on DSV2, which plugs in a wrapper over kernel Rust. That'd be great to have for streamlining
[25:32] our ecosystem, and that's our goal with Delta 5. Now, let's let's talk about like another aspect of unification,
[25:44] um which is unifying formats. Right? There's Delta, there's Iceberg. Where are we headed, and how do we unify the formats?
[25:56] I think it's important to understand why do we care about unification. Um and and for that, we need to look at what's the state of the interoperability today. There's a unified data layer. We've been able to have both the formats speak the same
[26:10] file format, Parquet, speak the same on-disk representation of deletion vectors, row tracking, and other data types like variant. But then the metadata seems to still be fragmented, right? Metadata about
[26:23] deletion vectors, metadata about your Parquet files, you know, stats about your columns, uh those are not unified. And that seems to be the final hurdle in uh you know, truly unifying the format for greater operability.
[26:39] Um and there are some reasons, right? Like there are some tradeoffs that were made for uh adopting um such different styles of metadata, right? You know, Delta Lake uh you know, is write-optimized. It's great for fast writes, fast commits, and has a lot of
[26:51] other goodies, like columnar checkpoints. But as a result of that, uh there are some trade-offs, right? You run into read latencies because you have to compact these Delta logs and do a log replay.
[27:05] Checkpoints in Delta are not incremental, so they're expensive to write. They're large checkpoints, and it's expensive to write. And finally, you have to keep translating between Iceberg and Delta. This continues to be the challenge.
[27:17] While in the Iceberg side, you know, it's great. It's read optimized, you know. Order of one planning. It supports all of the other goodies, too, but then the other trade-offs are slower commits due to maybe metadata write amplification.
[27:30] The three-level metadata structure of Iceberg could be sometimes complex and not very streaming friendly. So, while we unify the formats, we have an opportunity to also solve for these trade-offs.
[27:44] So, let's take a quick look at where the Delta is today, right? So, you have, you know, a commit uh at version 200. You write out a metadata file, 200.json.
[27:56] It's great. It's a transaction log. You can quickly write out this metadata file, have a quick commit, and that points to all of your data files. And this continues to be, you know, um possible for all your future commits.
[28:14] Once enough commits accumulate, Delta writes a checkpoint, right? And this is to bound all of the replay cost, right? Once the checkpoint is written, um Delta can basically use this as the baseline and replay any logs on top of
[28:27] it. But here's the problem, you know, each checkpoint is a freshly written new checkpoint, different from the previous checkpoint, right? For large checkpoints, you you have
[28:39] shards, which points to other checkpoints, but then overall, checkpoints are expensive to create. And as a result of that, you can't create
[28:52] checkpoints often. Which means you have to have a certain number of logs accumulate before you write a checkpoint. And as a result of that, you incur read latency.
[29:06] But there's a new proposal in the Iceberg community. Um and it's to evolve the Iceberg metadata structure to something which is an adaptive metadata tree. Right? You know, the root the structure looks as follows, right?
[29:18] There's a root manifest that has direct references to files. Once the root is large enough, it's flushed to a leaf manifest, which contains the references to files, but also is linked back to the root
[29:30] manifest. Right? And it's adaptive in a way that it rebalances itself as in when enough data is written to the root and the leaf. And it's incremental. Right?
[29:43] So, the good thing is it retains some of the If you look at it, it retains some of the read side planning semantics, but also makes things incremental. It keeps uh you know, lots of things latency bound.
[29:59] So, this is how it looks side by side, right? You have Delta on the left. There's you know, the checkpoint. There's a bunch of logs that you replay on top of it. And then you have the adaptive metadata tree. I think you can already see something
[30:11] here. I think there are ways where the adaptive metadata tree incorporates some of Delta's strengths, but also solves for Delta's weaknesses.
[30:25] And that's the goal. And here's what it looks like. You know, here the two commits live together. At version 200 that you see here, we have a manifest commit, right? It materializes the new root manifest.
[30:38] And the metadata uh tree is part of the commit, not as separate file. If you want to write a quick change, which is version 201, we just do we we utilize
[30:51] the fast nature of writing a small JSON log. The readers can start with the tree at 200 and then just reapply the JSON on top of it. And this is where we get the best of both worlds, right?
[31:03] Incremental tree for large writes and cheap JSON for small ones. No translation.
[31:16] So, this is where we want to end some of the tradeoffs that we had before, right? With with the choices on different metadatas. There's zero translation overhead, eliminates all of the uniform translation costs, right? Combines Delta's fast write
[31:29] capability with Iceberg's order of one planning read capability in a log tree architecture. And then it's catalog agnostic, right? Like your stats schema management across all catalog interfaces.
[31:47] So, what does the stack look like? The evolved unified stack, that's our goal, right? At the bottom, you have your parquet deletion vectors, your data files.
[31:59] The metadata is built on the the adaptive metadata tree in Iceberg V4. We talked about unifying everything on Delta kernel rust with a wrapper on top, which can then integrate into the data source V2 connector.
[32:11] And you can enjoy all of that on the Unity catalog.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.