Skip to main content

Your Guide to Open Table Formats — Delta, Iceberg, Best Practices, and What’s Next!

Summary

  • Delta Lake and Apache Iceberg are co-evolving rather than competing, with Delta 5.0 and the Iceberg v4 Adaptive Metadata Tree bringing the two formats closer together at the metadata layer for high-performance multi-engine workloads.
  • The Unified Delta Kernel, now generally available and written in Rust, powers Delta reads across Spark, Flink, and DuckDB, while Catalog-Managed Commits enable safer multi-engine writes, stronger governance, and multi-table transactions.
  • This video covers best practices for working across open table formats today, including Auto CDF for incremental pipelines and new AI-workload features such as the FILE data type, VARIANT shredding, and native lakehouse indexing.

Your Guide to Open Table Formats — Delta, Iceberg, Best Practices, and What’s Next!

Watch: Your Guide to Open Table Formats — Delta, Iceberg, Best Practices, and What’s Next!
Open table formats had a breakthrough year—and now the focus shifts to how to use them effectively. In this video, we’ll break down what’s new across Delta Lake and Apache Iceberg, including an early look at Delta 5.0 and the Iceberg v4 Adaptive Metadata Tree, which brings the two formats closer together at the metadata layer for high-performance, multi-engine workloads. More importantly, we’ll share best practices for working across formats today. Learn how teams are using the Unified Delta Kernel (now GA, written in Rust) across Spark, Flink, and DuckDB, and how Catalog-Managed Commits enable safer multi-engine writes, stronger governance, and multi-table transactions. We’ll also cover Auto CDF for building efficient incremental pipelines. Finally, we’ll highlight innovations for AI workloads, including the FILE data type and VARIANT shredding. Walk away with practical guidance to build, optimize, and future-proof your open data architecture.
Talk By: Benjamin Mathew, Product Manager, Databricks ; Scott Sandre, Sr Software Engineer, Databricks ;

Chapters

FAQs

Should I choose Delta Lake or Apache Iceberg for my lakehouse?

The narrative has shifted from choosing between Delta and Iceberg to leveraging them as co-evolving formats that share innovations at the metadata layer. Enterprises can benefit from lakehouse storage improvements regardless of which format they use, and tools like the Unified Delta Kernel and Catalog-Managed Commits work across both formats.

What are Catalog-Managed Commits in Delta Lake?

Catalog-Managed Commits make Delta catalog-oriented like Iceberg, enabling stronger governance, safer multi-engine writes, and multi-table transactions. They provide a catalog-coordinated commit protocol that improves correctness when multiple engines write to the same table.

What is the Unified Delta Kernel and what does it power?

The Unified Delta Kernel is a generally available, Rust-based implementation that provides a single Delta Lake reader used consistently across engines including Spark, Flink, and DuckDB. Having one implementation reduces fragmentation and ensures consistent read behavior across the full open lakehouse ecosystem.

How are open table formats evolving to support AI workloads?

Open table formats are adding AI-specific capabilities such as the FILE data type for multimodal data, VARIANT shredding for semi-structured data, native lakehouse indexing including vector and full-text search, and column append operations for efficient model output storage. This video covers what is generally available, in preview, and on the roadmap.

Full transcript

[00:08] Okay, hello everyone. Welcome to Data and AI Summit. Uh in this session, we'll be talking about all the innovations that have happened over the past year across Delta and Iceberg, and also what's coming next for these open lakehouse ecosystems. Um
[00:20] This Oh, my name is Ben. I'm a product manager here at Databricks. I work on open table formats and interoperability across uh across engines and catalogs. And if you want some recommendations on the best Thai food in San Francisco, I will
[00:33] definitely talk your ear off. >> Hi, my name is Nishith. I'm a senior engineering manager here at Databricks. Um I am a long-time contributor to open table formats. Uh love solving problems
[00:46] in data infrastructure. And Thai food sounds great. And then if along with that, if you want to talk sports, I'm here. >>
[00:59] make sure to fill out your surveys after this session. But to to give some context on this session, in the past, we've heard or enterprises often ask us, "Should I be using Delta or Iceberg?" That narrative is changing now. The
[01:12] reality is uh Delta and Iceberg, these these ecosystems are co-evolving so that enterprises can leverage lakehouse storage innovations regardless of the format that they use. Engines are also adapting based on these
[01:24] innovations, and we're even seeing workloads are changing now with AI. And here at Databricks, our fundamental thesis is that the open lakehouse architecture ends this trade-off between performance and ecosystem interoperability.
[01:37] So with that, over the next 40 minutes, we'll cover five main things. First, we'll give an overview of the open lakehouse architecture. Then we'll talk about how these open table formats are co-evolving. Then we'll talk about how engines
[01:49] themselves are being modernized and adapting with these storage level innovations. Then we'll talk about how open table formats themselves are evolving with AI. And then finally, we'll cover what's coming next uh for these open lakehouse
[02:02] ecosystems. Awesome. So, to start, we'll talk about what is the open lakehouse architecture? And to do that, we'll begin with open table formats. You see, when organizations wanted to start running massive amounts of
[02:15] analytics on their large-scale data sets, they looked to object storage as a place to cheaply store and operate on that data. However, by design, object storage didn't provide
[02:27] transactions. And then that's exactly why Delta and Iceberg, these open table formats, came to be. These open table formats did two things. First, they brought asset transactions to data lakes.
[02:39] And second, they allowed enterprises to interact with their large-scale data as if they were just simple SQL tables. And Delta and Iceberg, these open table formats, laid the foundation for the open lakehouse architecture.
[02:55] But as enterprises started building on the lakehouse architecture, their needs grew beyond just that of tables. They needed files, functions, ML models, and they needed a central place to govern and organize all these different kinds of assets.
[03:08] And that's exactly what we built in open source, Unity Catalog. Unity Catalog is this multi-modal catalog that allows enterprises to govern, discover, and audit all these different kinds of assets. And together, open open table formats
[03:22] and open catalogs form what we call the open the modern open lakehouse, where you can store your data in your storage in open table formats, and point any engine at a central governing open
[03:35] catalog to access that data in a governed manner. Unity Catalog governs over 17 exabytes of data in open table formats, which gives us here at Databricks a very
[03:47] unique view into how enterprises are leveraging their lakehouses, what they need from the those lakehouses, and how workloads are changing. And, you know, our overarching charter here here still remains the same with
[03:59] the open lakehouse, where enterprises should not have to think about the formats that they store their data in, but rather any innovation that comes to the storage layer is available to all enterprises. And the first key component of
[04:11] delivering on that is co-evolving these open table formats, Delta and Iceberg together. Delta and Iceberg releases over the past year have brought a lot of new features and functionalities to the ecosystem,
[04:23] but for the purpose of this session, we'll be covering three groups of innovations. First, I'll talk about how a new feature called catalog commits is making Delta a catalog-oriented table format very
[04:35] similar to Iceberg. Then, I'll talk about a series of innovations that are happening at the data layer that enable the data files for Delta and Iceberg tables to be represented in the same way on disk.
[04:48] And then finally, I'll introduce the work being done at the metadata layer to unify how the metadata structures of Delta and Iceberg are today, so that we can enable a more interoperable, performance, and
[05:01] future-proof set of open lakehouse tables. Awesome. So, I'll start with catalog commits. So, when Delta was designed, Delta was oriented around the file system, where if you wanted to interact
[05:13] with a Delta table, you would leverage file system APIs to perform those accesses. And that worked for a while, but eventually organizations started to run into some problem into some problems that were
[05:25] directly related to the fact that the file system was the commit coordinator, rather than the catalog. The first challenge that organizations were running into is what we call the multi-engine, multi-agent sprawl
[05:37] problem. You see, enterprises want to use different kinds of engines for their different workloads. And they expect to be able to define their policies once, and point any engine at those policies
[05:49] to respect those policies upon data access. So, those accesses themselves are governed. However, since Delta was a file system oriented table format, there was no way to actually have Delta clients reference these policies through a
[06:02] catalog to then access those tables. And as a result, third-party engines, if they wanted to access UC managed Delta tables, they had to do that by static paths. They couldn't go through the catalog to access those tables.
[06:14] And accessing uh these tables by static paths leads to three huge problems. First, of course, these accesses are not governed, which can be very risky for sensitive data.
[06:28] Second, since you since the catalog is no longer aware of the accesses that are being performed, there's no way to actually audit uh which clients and who is actually leveraging that data, which can be a nightmare from a compliance perspective.
[06:41] And then finally, without the governing catalog being aware of the kinds of operations that are being performed, you can run into a scenario where an external engine is performing a schema operation like dropping a column, and dropping that column may
[06:53] fundamentally break relationships between tables. This can lead to silent metadata drift and even downstream correctness issues. The second overarching challenge that organizations were facing have to do
[07:05] with transactions. You see, enterprises want to standardize on the lakehouse architecture, and that even means bringing their traditional data warehousing workloads to the lakehouse.
[07:17] But, a very common paradigm in traditional data warehousing is multi-table transactions. And since Delta has been a file system oriented table format, the only granularity at which transactions were possible were at the
[07:29] single table level. Organizations couldn't perform multi-table transactions. So, if organizations wanted to perform a multi-table update, this may result in some of these updates uh uh potentially
[07:41] failing, leaving downstream assets like a dashboard in some unreliable state. And as a result, bringing these multi-table transactions and these traditional data warehousing workloads have been pretty difficult for teams
[07:54] building their lakehouses on Delta. And if we take if we take a step back and look at how Iceberg addresses these challenges, it does so by leveraging the catalog as a commit coordinator. Meaning that all reads, writes, and accesses are
[08:08] done through the catalog. So, in an effort to address these challenges for Delta practitioners, and as part of the overarching charter to co-evolve these table formats, we worked with the community very
[08:20] closely to build catalog commits. Catalog commits is this open standard by which Delta tables can leverage a catalog as a commit coordinator. Meaning the catalog is now responsible for
[08:34] facilitating all reads, writes, table discovery, and table authorization. Unity Catalog managed tables provide a reference implementation for Delta tables, which leverage the catalog as
[08:47] its commit coordinator. And now catalog commits are GA on Databricks and deliver so many benefits to enterprises leveraging UC managed tables. First, external engines can now write to
[09:00] UC managed Delta tables in a fully governed manner. Second, catalog commits enables consistent governance, attribute-based access control, and auditability across all engines.
[09:13] Third, catalog commits enables multi-statement and multi-table transactions on the lakehouse, allowing enterprises to bring these traditional data warehousing workloads to the lakehouse. Fourth, Unity Catalog can further
[09:26] optimize the data layout of these UC managed tables because now it has full visibility into all engines that are accessing these tables. And finally, catalog commits standardizes how tables across Delta and
[09:40] Iceberg are discovered, uh accessed, and authorized. Ultimately, Project Commit represents this next major evolution of Delta, and we're very, very excited about all the different benefits it brings to
[09:52] enterprises. Awesome. Now, we'll talk about how Delta and Iceberg are being unified at the data layer. And this is super important because having a common representation of how
[10:05] these tables are on disk means that we're one step closer to a state where any storage innovation that comes to the storage layer is available to both formats. Some of the key innovations that we'll
[10:17] be talking about are row lineage, deletion vectors, variant, and the geospatial data types. So, to start, row lineage and deletion vectors are critical to making incremental processing possible.
[10:30] These were two features that were introduced to Delta and very recently brought to the Iceberg community through Iceberg V3. Before V3, incrementally processing Iceberg tables was very expensive. There
[10:43] was no native way to detect changes to an Iceberg table across table scans, and deleting files resulted in writing lots of metadata. Row lineage and deletion vectors change
[10:55] this. Row lineage is able to indicate the rows that have changed uh but since the last table was across table updates, and deletion vectors represents deletes in a
[11:07] highly compact bitmap. Second, Databricks developed variants as a data type for storing semi-structured data, and brought it to Parquet, Delta, and Iceberg.
[11:20] For context, historically, working with, processing, and storing semi-structured data has been very difficult. Enterprises generally had two options. First, they could model this
[11:32] semi-structured data with a rigid schema, which gave performance, but as soon as that payload would evolve, these pipelines would need to be evolved as well, otherwise they would just break. And the other option was that they could
[11:45] persist these payloads as JSON strings, uh which provided the flexibility of JSON, but at the end of the day, when it came to parsing and querying these payloads, this introduced a lot of complex logic and regex parsing um on
[11:59] that read path. Variant stores payloads in a binary representation that maintains types and nesting, meaning that engines can go and extract the data that they need in a
[12:11] highly performant way. And also, variant shredding materializes common fields across these payloads as columns in the Parquet files, meaning that now with shredding we can enable predicate pushdown and column
[12:24] pruning on the semi-structured data. Together, variant and shredding enable efficient analytics over semi-structured data over semi-structured data in the open lakehouse.
[12:39] And lastly, Databricks helped introduce geospatial data types uh to the open lakehouse across Parquet, Delta, and Iceberg to unlock geospatial analytics directly on the lakehouse. Geospatial data has historically been
[12:51] treated as a specialized workload requiring dedicated tools uh and even other systems outside of the lakehouse. Location data has typically been represented as strings or even in custom
[13:04] binary formats, and performing operations on this geospatial data uh such as distance calculations and boundary checks usually required complicated UDFs um or again, even using
[13:16] external systems. The geometry and geography types now enable geospatial analytics directly on the open lakehouse.
[13:31] So, leveraging the catalog as the commit coordinator, deletion vectors, row lineage, variant, and geospatial data types are now all available on Databricks across Delta and Iceberg. We're super excited to continue to work
[13:43] with the Parquet, Delta, and Iceberg communities so that enterprises can continue to benefit from the innovations that are happening at the storage layer. The next step in this broader unification is focusing on the next
[13:55] layer up, unifying the metadata structures across these formats. The metadata layer is the frontier because once these metadata structures are unified across formats, enterprises can truly benefit from innovations
[14:08] across Delta and Iceberg. To give some background here, Delta was built to be optimized for fast writes, whereas Iceberg was built to be optimized for fast reads. And as a result, each of these metadata
[14:21] structures have evolved in their own ways. But now, we're driving collaboration across the Delta and Iceberg ecosystems to bring and define a unified metadata structure that brings the best of both
[14:35] worlds to these formats. The unified metadata structure will be available for preview on Databricks in Q3, which we're we're super super excited about. There are a lot of people who are deep
[14:48] in the weeds working on this broader unification story across these communities. We don't have enough time in this session to go into those details, but if you would like to learn more, I'd highly encourage that you check out Anoop and Mike's session tomorrow, uh sorry, on Thursday, uh to
[15:02] learn more there. Ryan and Anoop's session already happened earlier today, but I would highly recommend that you go watch that recording as well. Now that we've talked about unification uh at the storage layer, I'm super
[15:15] excited to hand off to Nishith, who will talk through the future of engines and also workloads uh now with AI. >> Thank you, Ben. So, I think it's important to now talk
[15:28] about, you know, how do we modernize engines, um which will deliver some of the use cases and features that Ben talked about. And I want to talk to you about Delta Kernel. Uh so so what is Delta Kernel? Um
[15:41] you know, there's a growing um ecosystem of Delta integrations and connectors, many of which are across different languages, uh different memory formats, and it'd be great if you had a set of
[15:53] table APIs that connectors can use to just read and write data. And it abstracts away all your Delta protocol details, right? This way, engine APIs can basically use
[16:05] Delta Kernel to integrate with this abstracted framework, but still keep the power that engines need to read and write actual data. And that's exactly what Delta Kernel is. It's an open-source library written in both Java and Rust that provides these
[16:18] narrow set of APIs. It abstracts and delivers exactly on the promise of making sure that you plug in once and you get it get protocol everywhere, and it's very extensible for all the performance reasons of what however
[16:31] engines would want to integrate. And this is how it looks like. So, you have the Delta protocol abstraction. You expose them to APIs,
[16:43] but then you also provide enough flexibility to engines so that they can do what they're best at at reading lots and lots of data.
[16:56] And you know, we've seen a lot of success with this architecture. Uh many of our open-source uh partners and integrators, DuckDB, ClickHouse, many of these partners have integrated with Delta Kernel, and it's made their lives very
[17:09] easy to add Delta support. And we're happy to continue working with them. So, where is Delta Kernel being used? You know, what's the scale of it? Uh across the ecosystem, we see uh a lot
[17:22] of adoption with our partners, uh a lot of open source engines, uh you know, connectors such as Delta RS, um Flink, Spark. Um so, it's widely adopted uh across the open source
[17:34] ecosystem. Within Databricks, we see massive adoption. Actually, you probably use some of these products or you probably heard about them in the keynote earlier today. Zero bus, uh which lets you do real-time
[17:47] ingest, Delta Kernel powers it. Delta Sharing, which you use to share tables, Kernel powers it. Lakehouse, uh you heard about this the new real-time CDC,
[17:59] Kernel powers it. And a lot of other services. One thing that I haven't mentioned here, which you heard about today in the keynote, Photon, uh which delivers real-time Lakehouse, powered by Kernel. Um so, we definitely see Kernel being
[18:12] used as the de facto uh central place for anybody to go connect and understand Delta. Um so, I want to talk about, you know, what's what's been happening over the
[18:24] last year, uh specifically in Kernel Rust. Um you know, we've been delivering a lot of user-facing features, uh and along with that, a lot of performance improvements. Um you know, over the last year, we've
[18:36] shipped close to 20 versions. We've added catalog commits, which Ben covered, which lets you talk to catalog manage tables. You can do your favorite DDLs, you can you you know, create tables, alter
[18:49] tables. Um and, you know, we've been adding all of the support uh that Delta customers enjoy today, right? Clustering, row tracking, >>
[19:01] um and, you know, Iceberg support. On the performance side, um there are some cool innovations happening as well. Right? We we delivered incremental CRC APIs that are super important uh for
[19:14] near instant table reads. Something that written uses. You know, along with that, you know, you have statistics which will power all the pruning and make sure we read the least amount of
[19:26] data that we should. Um And then, you know, a bunch of other other cool cool improvements such as the incremental scan APIs, sidecar checkpoints, you know, making checkpoints themselves which tend to be also growing more
[19:40] performant. So, the great thing about adding all of these features is you add it once in kernel and all of the ecosystem, you know, internally within Databricks as well as all our open source ecosystems
[19:52] just get them virtually for free. Right? And that's the goal of the kernel, right? And that's the architecture that we want to validate. But still there are some challenges, right? And I I want to talk to you about
[20:04] one of the major challenges. You probably see it already. There's two implementations of kernel. One in Java, one in Rust. For the native ecosystem and for the JVM ecosystem.
[20:16] Right? You know, both of these projects evolved independently and they don't do run a risk of divergence. You know, some features are supported in kernel Rust, some features are supported in kernel Java. And now you're looking for parity across
[20:28] both these libraries. And if you've dabbled with Java if you've dabbled with Java and Rust, you'll appreciate there are differences. You know, how Rust code is written, how APIs are designed different from how
[20:40] Java works. And that leads to, you know, more maintenance overhead, you know, not just as developers, but you know, release processes, quality testing, all of that has overhead.
[20:52] And this is what we what we want to solve. The good news is, you know, we have a solution in the works. Right? And this is what it looks like. You have kernel Rust which is at the
[21:04] core. You know, it powers uh this is, you know, it's adding all the latest and greatest features. It'll power all of the new incoming engines that want to connect to it. You know, the the native engines which
[21:16] are building on Rust directly consume kernel Rust. If you are other if you write other native code such as C++, you go through a binding layer, right?
[21:29] And then we're extending that further to say, "Hey, like why don't we build a Java kernel layer, uh which is also built on these bindings, uh through the awesome technologies that JDK provides such as, you know,
[21:41] uh JNI and FFM. And finally, the hope is all our the Java ecosystem, the connectors start to use uh this kernel layer.
[21:54] What we get with that is, you know, you build once in kernel Rust, and you actually can power the entire ecosystem, whether it's your native ecosystem or your Java ecosystem. And this is this is right now happening in the community.
[22:10] Some side effects that maybe consumers of this uh kernel Rust library also might see is performance. You know, Rust is typically more performant than Java. And you know, more connectors who actually start to integrate with uh kernel Rust might see that as a side effect as well.
[22:23] And we you know, obviously get faster feature development. Um now, you know, obviously we've built this, you know, architecture, uh and we want to set us up uh
[22:36] set us up for some other future innovations that are supposed to happen in the AI world, right? And I want to talk to you about uh you know, how the open table formats are evolving for AI use cases. Um and I want to cover how workloads are
[22:49] evolving, uh especially as enterprises are leaning into AI, and then how open table formats are going to address some of the workload challenges.
[23:02] So, I think overall, uh you know, the let's talk about workload changing for AI. There there are four major shifts happening. The first one I want to talk talk to you about is semi-structured data, right? As
[23:14] more and more LLM applications are being used across the industry, uh they're generating a lot of semi-structured data. Second,
[23:26] you know, enterprises are dramatically scaling their usage of multi model data. You know, documents, images, videos. Third, data itself is becoming more dynamic.
[23:40] You know, as enterprises iterate on their models, columns are changing faster than the datasets themselves. You want to be able to add a column without rewriting your entire dataset.
[23:56] And finally, you know, enterprises um expect, you know, faster retrievals um on collections of documents, images, and such. So, now we'll walk through what is happening in the community to address
[24:08] many of these workloads. So, we briefly talked about variant and shredding when we talked about Iceberg V3. You know, the broader problem we're
[24:20] trying to address with variant is in the past semi-structured data has been expensive to store and process. Uh you know, variant and shredding unlocks semi-structured objects in a
[24:32] typed manner, so they can they shouldn't be processed as strings. But, to build end-to-end use cases, it becomes challenging. You know, it makes it practical to build end-to-end use cases where agents,
[24:45] models, and pipelines are consuming these semi-structured objects. We drove variant development across Parquet and Delta and Iceberg. It's been awesome to see the ecosystem benefit from this awesome data type.
[24:58] You know, engines can navigate nested fields without just reading them as raw JSON, which is awesome. I think multimodal data is is more challenging.
[25:10] You know, beyond just semi-structured data, it's a whole class of unstructured data that once that has challenges on how, uh, you know, enterprises want to leverage and govern this. You know, the the the classic problem
[25:23] has been there's there's not been a good way to represent this unstructured data in Iceberg or in Delta. We You know, enterprises have had two options. You know, one, store this as binary, right? Um, which has its own challenges
[25:37] on how you read them and, you know, causes kind of memory issues and query performance issues. Um, or and you store them differently from where where your table is. The You know, your your actual table is
[25:49] stored, which creates problems in how you govern the data, you know, how do you man- manage the life cycle of this and such. For example, organizations may choose to store it in an object store, while your table which
[26:02] actually references that is separate. And we're working with the community now to build something called a file type, uh, which stores a type reference to your actual multimodal data with
[26:14] metadata around it, right? And this will sit in the same Delta table or the Iceberg table that you have. And this makes it easy to govern, you know, control life cycle, um, you you know, you want to delete them, etc.
[26:31] And then it just becomes a lakehouse table. The third one, um, is embeddings, right? You know, since columns typically store prediction values, it's very common, uh, for column values to be added, just like
[26:44] new prediction values. If you updated, um, the table, this would typically result in rewriting a bunch of data. This is quite expensive, especially
[26:56] in the AI world where tables columns are moving faster than the tables. So, we're working in the community to build a feature called column appends. The idea is pretty simple. Um
[27:08] we currently have something called sidecars, which lets you associate a piece of data with the actual table. And we want to evolve that to make it a column append-friendly table. So, what would happen is you add a
[27:21] column, it adds it into a sidecar, and then it registers it back to the table, just like you added a column. No rewriting of data data, cheap. That's exactly what we want.
[27:34] And uh you know, a lot of uh you know, use cases like, you know, embeddings and wide columns that are uh wide schemas that are used for ML workloads all would benefit from this.
[27:49] Um and finally, indexing. You know, um indexing is a powerful tool uh used widely again in AI workloads these days. Uh use it for uh you know, vector indexes uh for rag applications. Um some
[28:01] use cases around full-text searches, some uses use cases around point lookups. Um and what typically organizations tend to do is disperse this data outside of the lakehouse into a proprietary system or an open
[28:15] system, but it now diverges from where your actual lakehouse table is. You may want to put this data in your vector DB or in a text search type of database, and now you have this dichotomy of like maintaining two
[28:28] different sources of data. And our goal is to create a common standard so that you can actually create indexes on open table formats. And there's a lot of work happening um in the community um along this.
[28:40] And there's some examples here, um but the goal is, you know, you create your index on your lakehouse table, and you don't have to actually disperse your data anywhere else.
[28:54] So, to summarize, you know, these are the key innovations happening to address some of the AI workloads. You know, variant and shredding, which we talked about, already part of iceberg V3. Uh file data type, column up hands indexing actively
[29:06] happening in the community. Uh both the communities coming together uh to build a solution that works for everybody. With that, I will hand it over to Ben to talk about all the stuff that's coming
[29:18] up. >> Awesome. Uh yeah, to close this out, let's talk through a roadmap of all these innovations that we've covered and uh like what's available and when things will be available.
[29:30] So, first with regards to the co-evolution of Delta and Iceberg, Delta catalog commits are now GA on Databricks, um and this aligns Delta with the core catalog-oriented model that was pioneered by Iceberg.
[29:44] Also, deletion vectors, row lineage, variants, and geospatial data types uh all align how Delta and Iceberg are represented on disk as data files. And these have been GA'd in data in Delta
[29:56] and are now GA on Iceberg through uh Iceberg V3 on Databricks. Second, engines are being modernized. We're building a Delta Spark connector on DSV2, uh and we're also building one
[30:09] unified kernel as Nashat mentioned. And finally, Delta and Iceberg are evolving for AI. Variant and shredding are now GA on Databricks, and the file data type is in private preview.
[30:21] Um and the proposal for the file data type is also in Parquet. The column up hand proposal is also in Parquet, um and full text search indexing is beta on Databricks.
[30:38] As we've been mentioning here at Databricks, we are heavily invested in evolving these open lake house for the changing needs of enterprises. And we're super excited to continue to work with the parquet, delta, and iceberg communities to make this a
[30:51] reality. So, with that, thank you so much for attending our session. Um and we will be around to answer some questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.