Open Table Formats for Dummies

Summary

Delta Lake, Apache Hudi, and Apache Iceberg each add a transaction and metadata layer on top of Parquet to solve core limitations including lack of ACID compliance, schema drift, and small-file accumulation.
Open table formats differ in their metadata storage approach — catalog-based versus log-on-storage — and in how they handle compaction, clustering, deletion vectors, and merge-on-read, which directly affects operational complexity and engine compatibility.
Choosing the right table format requires evaluating ecosystem fit, roadmap direction, and specific feature trade-offs, with Delta Lake deeply integrated into the Databricks Data and AI platform and Hudi and Iceberg having strong communities in other ecosystems.

Open Table Formats for Dummies

With the enormous amount of discussion about open storage formats between nerds and even not-nerds, it can be hard to keep track of who’s doing what and how this actually makes any impact on day to day data projects. Let's take a closer look at the three big projects in this space; Delta, Hudi and Iceberg. They’re all trying to solve for similar data problems and have tackled the various challenges in different ways. This talk with start with the very basics of how we got here, what the history is before diving deep into the underlying tech, their roadmaps, and their impacts on the data landscape as a whole.
Talk By: Holly Smith, DevRel, Databricks ;

Chapters

FAQs

What are open table formats and why do they matter?

Open table formats like Delta Lake, Apache Hudi, and Apache Iceberg add a transaction log and metadata layer on top of file formats like Parquet, enabling ACID compliance, time travel, and schema evolution on data lakes. They address the core limitations of raw Parquet, which offers efficient columnar storage but lacks native mechanisms for handling concurrent writes, deletes, and schema changes safely. This video explains how each format approaches these challenges from different architectural starting points.

What are the main limitations of Parquet that open table formats solve?

Parquet excels at columnar storage with row groups, metadata filtering, and bloom filters, but it has no ACID compliance, no native handling for schema drift, and accumulates many small files over time as data is written incrementally. Open table formats wrap Parquet with a transaction log or metadata catalog that enables consistent reads and writes, manages schema evolution, and supports compaction to consolidate small files. This makes Parquet-backed data lakes production-ready for workloads that demand reliability and correctness.

How do Delta Lake, Hudi, and Iceberg differ in metadata storage?

A key architectural distinction covered in this video is metadata storage: some formats store the transaction log directly on the storage layer alongside data files, while others rely on an external catalog. This design choice affects catalog dependencies, multi-engine access patterns, and operational complexity at scale. The video compares both approaches and explains how each format's choice shapes its ecosystem integrations and governance model.

How do I choose between Delta Lake, Hudi, and Iceberg for my project?

Choosing a table format depends on your existing engine ecosystem, the specific features you need — such as merge-on-read, clustering, deletion vectors, or multi-table transactions — and your team's operational preferences. Delta Lake has deep native integration with the Databricks Data and AI platform, while Hudi and Iceberg each have strong communities in other engine ecosystems. This video walks through the overlaps, trade-offs, and roadmaps of all three formats to help guide the decision.

Full transcript

[00:09] Uh all right everyone uh first of all thank you so much for staying around so late in the day and second of all thank you for coming to this nonAI talk um
[00:26] uh so uh yeah so this instead is a talk about uh data storage and this is confirming the rumors that yes, we are in fact all giant nerds in this room. Um, so why storage? I mean, like conceptually as humans, we get storage,
[00:38] we take stuff, we put it in a place, and like bam, it's stored. Um, and I'll be honest, for my first decade, uh, of career in data, I never really thought about it too much, but then everyone
[00:50] started arguing about it online. Um, and I thought it was maybe worth a bit of a closer look. And so, who better to take you through this than me, Holly Smith, uh, chief dummy?
[01:03] Who am I to have the audacity to stand up here and talk about table formats? Um, and I do have a few things going for me in terms of qualifications. In my defense, I find experts really hard to
[01:17] understand. I know I work with a lot of them and by that logic I'm not an expert and that means I'm going to be easy to understand. Right?
[01:29] all I don't have a CS degree. I never thought I'd be ashamed of my master degree but here we are. Um I have worked at data bricks uh for seven years now. Um I did a lot of time kind of doing and working with customers uh five years as
[01:42] a senior architect um in their consulting wing. I will say I spent a lot more time using Delta than I did iceberg and hoodie, but I have shown this to a bunch of iceberg and hoodie people as well. Um, I'm also not an open source contributor and I have great
[01:55] respect for those who are. This talk is only going to focus on open source. I do not have the time to go through every single proprietary implementation that is available. Um, but despite this, I do have a few
[02:08] things going for me. And for those of you who know my work very well, you will know that I can read. I read the Delta books. I read the iceberg books. I read the one house
[02:21] early release of the hoodie book. Uh I also read the documentation um novel as that is uh to really get to the bottom of some of this. Um I also secondly working at data bicks I have access to
[02:33] the um the experts. I also have access to people in the communities who are really strong members of each community. um and I can uh annoy the hell out of them to help understand it better.
[02:45] Now, this is an introduction session, so uh like there's some really advanced tech in here that I will have to gloss over some of it for the sake of time. Um if I had my way, I'd make this talk six hours, but unfortunately I don't. Um and
[02:57] finally, my advantage, um I have no shame. I can just keep asking the dumbest questions and then just keep on going. Um, and we are going to be going over some complex topics today. And I know we're all coming to the table uh
[03:09] with different levels of knowledge. Um, so to benchmark just how complex something is going to be, uh, we will be using the industry standard, the Nando spicy chart. And no one is going to shame you if you can't manage an extra
[03:22] hot spicy Nandos. Just realize it's probably going to be over soon anyway because I'm not smart enough to talk in length about some of the really, really complicated topics. You'll get the salient points at the end anyway. Don't
[03:34] worry, but let's start off super easy. And how did we get into this scenario anyway? Um, first things first, uh, data is getting bigger. Uh, this is not going to be a shock to you. Of course, with the advent of AI, this is getting, you
[03:47] know, even more and more true. Um, it's also getting faster as well. Kind of like people analysts tolerate slight delays in data. Um, but when it comes to business applications and agentic blahy blahy blah, um, we need that data even
[03:59] faster. We also need to find it as well. So having it in ADLS is just a giant dumping ground of data is not good enough. We need to be able to find this data. And also um with the proliferation
[04:11] of tools, the network effects means that we're not everything is going to play well together. We need to have it usable as well. And something I want you to keep in mind throughout the history of data. These things have always been a
[04:23] problem and it is only going to get worse. And at this point I know you all have a burning question and that is what's wrong with Excel.
[04:35] Okay. Uh so let's talk about some fundamentals here like what is it that we need uh from a file format. And so the first thing is basically um how fast is it to read? Uh how fast is it to uh persist to disk when we write it? Uh in
[04:48] terms of the size I know storage is cheap um so it's like not a huge concern. Um but it's uh also once it's on desk as well uh you have the chance to make some changes without reading it in terms of the flexibility. So if you
[05:00] want to like add or remove some columns things like change the data type like widen it or if you've got numbers and strings and things like that how flexible is it once it's already written down. Uh and then finally how plugandplay is it? Uh what if you want
[05:12] to start using it with a new language? What if you wanted a new compute engine like what about a new meta store? Okay quick question show of hands. Who here has used parket? Pretty much everyone except the sound
[05:26] guy. Excellent. Uh it is prolific and it is support by supported by the three projects that I am going to talk about. So let's go into a bit of a deep dive really go into the detail of what parquet is. So uh when I originally
[05:38] thought of this talk I was like yeah I'm going to show what it looks like on disk and we'll like really start from first principles. Turns out on disk it looks like total gibberish. It's like this is still a representation. um everything is a representation of different features
[05:50] and so instead of excavating this I am going to have to use some representations uh spoilers it is all segments and metadata so if this is our table which I believe is the highly relevant table of Nando's branches it's
[06:04] very important table um the first way that we're going to divide our data is into row groups uh and so this is like horizontal sections of our table now you might be thinking like why bother doing
[06:16] anything of this at all. And that is a fair question. Um, as we start to subdivide our data, we can start to process it in parallel. Instead of having one big machine read one big table, why not have 10 smaller machines
[06:28] reading in parallel? And so to do that, you need to start subdividing. And this is where the first big subdivision is going to come in. And so the next layer down is to the column level. And this is where the phrase park is a column and a
[06:40] data data type comes from. The items within these columns are stored next to each other on disk for speedier retrieval. Um, now I've shown you this with five records in it. Realistically, it is going to be much bigger than this.
[06:52] And it's still not quite small enough for us to use. So, we're going to have to go one level deeper. And we are going to go to pages. And this is the smallest thing that we are going to. It's where the interesting things start to happen.
[07:06] And this is where we find our first piece of metadata. Associated with each one of these pages, there is metadata that contains summaries of what's in it. So it's like the size, when it's compressed and
[07:18] uncompressed, uh where everything is on disk, the statistics, the minimums, the maximums, the various binaries that you might have. You've also got dictionaries as well for repeated values. And there are some optional things in there around
[07:30] kind of like skipped columns and repeated values. And you will notice an entire uh trend throughout this entire field that when we have a data storage problem, we're going to answer it with
[07:43] metadata. So going to do a bit of audience interaction here. Uh when I say data storage problem, you're all going to shout metadata. And we're going to give this a go. So I have a data storage problem.
[07:56] >> You know what? That was actually pretty good. Give yourselves a round of applause. normally,
[08:10] not what it actually looks like as kind of like hexodimal values on disk that we saw earlier. Instead, it'll be some kind of mumbo jumbo that looks like this. And so the bits in gray are the actual data itself. And nestled in between our pages
[08:23] of data is the metadata. Now, does anyone see a problem with this? If you thought, hi, isn't that going to be a right faf to scan through all of this, the entire file to collect your metadata, you'd be right. So, I
[08:36] have a data storage problem. >> Excellent. Thank you very much. Uh, and at the end of the parket file, we're going to add in some metadata. Um, and we're going to have a lookup of where all of this data is. And this is called
[08:50] the page index. And if we think about this, like the smallest unit is called a page. We're going to index it. are we're going to make a list of where everything is hence page index and we can also start making this page index to do some
[09:02] clever things too. So, uh, whom here has ever heard of a bloom filter? Okay, a few people. Uh, so they are basically probabilistic features, uh, filters. So, they'll bring back everything you want
[09:14] plus a bit more. Um, and the idea is that it's kind of a fuzzy match. Um, and you'll get all of that information back plus a bit more, but it'll be much faster. So, if you're doing like needle in a haystack type queries where you
[09:26] want to return everything that might be, I don't know, like a suspicious account in like logs of 10 million, this is maybe a way that you want to do it. And so, um, there are some practical implications when you're using a bloom filter. You have to define it up front
[09:38] and you get to pick the tradeoff between how fast it returns something versus how accurate it is. The least accurate it is, the faster it's going to be to respond. You'll need to do some next steps obviously afterwards. Um, or if
[09:50] you want complete accuracy, well, that's not a bloom filter. That's just a normal filter. So, if I wanted branch ID uh to be my Bloom filter, um, it doesn't just store those values and keep them in the footer, it'll encode them. So think of
[10:02] it like a dictionary. So instead of the value 3 4 5 8 um it'll map them to a zero and then the next one will be mapped to a one. Um now this is like a number. So it's not particularly impressive but this if this was a text
[10:14] field um this reduces the search algorithm space. Like computers are really good searching through numbers. They're not very good at searching through text. Um but this is a bloom filter though. So it is not onetoone matching. It'll group them together so
[10:27] there's even fewer things to sort through. And all of this is stored in the metadata next to the page index. And if this doesn't make any sense, don't worry. It's a fairly niche feature um for needle in a haststack queries. Um
[10:40] now we have a lot of metadata about our tiny pages, but what about the whole file? Um and so this is the footer that is included in paret and it's more about like the table itself. So when we do our
[10:53] queries, we want to read as little as possible to reduce that I/IO and so we want to co have it colloccated and having the schema in there um is a way to check of what is in that table along
[11:05] with useful information like the number of rows but also kind of creators and version numbers as well. But this doesn't tell us about the columns and I have a data storage problem. >> You know what I feel like this is getting worse over time. It's fine. Me
[11:18] too, to be honest. So uh don't worry.
[11:30] at a so at a glance uh we do know what we've got in our table here. So in summary, uh metadata um
[11:42] out there. You might have heard of like OC or AVO and they are kind of similar um in terms of like the segmentation. Um, we've got like the different ways that we can segment our data. So whether we want to have it segmented by rows
[11:55] first or whether we want to have it by the number of rows or like the size of the rows in terms of how much they take up on space. Um, this does pretty much impact uh more wide data than it does any other type or if you've got super
[12:07] narrow IO problems. New projects are popping up like Lance and Nimble as well to do things. This again this is like really much uh about the implementation um because there are other ways that you can go down in terms of like what's the
[12:19] next level of segmentation um does it have you know what are the ways that it implements bloom filters um in terms of the way that you're storing uh the types of the storage between logical and physical how does it do that now this is
[12:31] not entirely relevant to this talk um but just in short other formats are available and they do make slightly different decisions when it comes to the way that they start breaking down their file formats Now, if we remember our original list of
[12:44] what we need a file format to do, let's assess how well parket does this. So, first up is the speed of retrieval. How is this going to make things faster? So, first up is column. Uh, which means that
[12:56] we can skip columns that aren't needed. Uh, this is very useful if there's lots of columns or if they're like big arrays that we really don't need. Um, it also means that when we process our data, it's all in the same type, which makes
[13:08] it very CPU friendly. And instead of uh blindly reading a file, we're going to check the metadata first to prove that it's in there. If there's a minimum and maximum in there, we know we're looking for a particular value and it's not
[13:20] between the minimum and max. We don't have to look at the file then, do we? And that's really nice. Uh, we also have row groups and they can be independently read by different nodes of a cluster as well. So, you can start to distribute it. Um and just like distributed reading
[13:32] with all these VMs, um some things can be said for the same rights as well. It's not quite the same level. It's one level up, but we also have some parallel writes as well. Uh and instead of those uh space consuming repeated values, uh we're
[13:45] going to make a dictionary to save on size as well. So in terms of the storage on disk, uh it is going to be much better. Um and there's different types of compression that you can work with too. And so to go one level further, uh
[13:57] we could actually hash these values as well. Um, interestingly, this doesn't happen at the entire table level. So, the same value might have multiple different hashes or different encodings. Uh, which is why actually sorting your
[14:09] data in park can sometimes save on space, which is, uh, quite a neat little trick. And so once a fine is written, uh, that's that's kind of it. Uh, but the engine can make some kind of forwards compatible, um, ideas. When we
[14:22] add like a new column or new files on there, we can just pretend it's secretly null in any of the old ones as well. Uh there we go. Sorry. Uh Adam's bits and pieces.
[14:35] Um all right then. So this is uh parket. And so if we think about this, so what is wrong with a database? This might be another way that you do some data things as well. Um and for me the glaring thing is normally they're not not great
[14:48] ecosystem players. You know, I know this is debatable as a lot more kind of open source things are coming out there. Um but for me in terms of you know how open source something is is like could you fork it and like add in your own connectors or like add your
[15:01] own data types. Um there are some things out there that you can there are some things when they talk about open source that's not the kind of open source that it is. Um you can't always choose the engine either as well. Um I know things like duct DB are coming out but again
[15:13] tends to tends to not be something so flexible. Uh again choosing your own metas store as well of what you need. This is like a if you've never heard of a metas store it's like a reference library of everything that's in your data. um they tend to be very dependent
[15:25] on SQL as well. So if you're trying to do something that's not SQL friendly that can be a little bit of a challenge as well. So in summary, how does park solve for all of our data wos? Uh well it does
[15:37] these things uh quite well and so I candidly for me this is why park has been one of the reigning champions in terms of open source projects for data for ages. So in summary I hope you all agree that for your data engineering the
[15:51] parket is much better than Excel. Um, but why stop there? Like we're about to talk about three different things that aren't parket and if park is open
[16:04] source like why didn't it go more? Why didn't it start to solve some of these other problems that we are going to talk about? Um there was a big push uh from users from kind of consumers of parket to have like data lakes that do start to
[16:16] behave a little bit more like databases and get those kind of features that they were used to. I could not find an exact quote from the Parket project of like why they didn't go on this journey. Um, but I did find uh a male from their
[16:29] archives in 2024. This idea that a parquet file should be like self-contained, whatever that means. Um, okay. So, where does a file format end and a new technology begin? And
[16:43] there's many differing names for the things that we're about to talk about. Um, but I'm going to refer to them as table formats. And the problem with having uh and there's still a few problems that we have when it comes to parquet. Um one of the things is kind of
[16:58] acid transactions and rollbacks. Uh one of the problems of having your footer right at the end of your file means that if you fail halfway through with a right you're kind of screwed because you need that footer for everything and if you don't have it you just got a bunch of
[17:10] corrupt data. Uh the next thing is schema enforcement. A park is incredibly gullible. you can just lie and be like here's a time stamp and give it an array and it'll be like cool excellent thank you um and this is
[17:22] going to cause havoc with any optimizations from like a CPU perspective but also from an end user perspective as well like this makes it really really difficult uh we also have a bit of a small files problem as well uh this tends to happen if you're doing
[17:35] things with streaming or you've got like little tiny transactions that are happening and you end up with more metadata than you do actual data which means you spend all this time chasing around all of metadata when in actual fact you didn't even need that much
[17:48] because it was probably better off uh in one big file. So you have a small files problem. And then finally uh again it's not so great at schema evolution. So we have a data storage problem.
[18:02] Yay. There we go. I think we've got like one more. Don't worry. So how could table formats help? So before we get into like with the specifics of each table, I'm just going
[18:14] to talk about generally how file formats can help. Sorry, table formats, how table formats can help. So the first thing that we're going to do is we're going to version our tables. I've got my one table here, and it's made up of
[18:26] these three different paret files that we've got here. So one table, but it's got three files underneath. Now, let's say that we go ahead and append some data. So each file represents a country where our Nandos are located. If I go
[18:38] ahead and I append South Africa to our table, I now have version two. And all I had to do was add that file in there. Um, the same is true for deletion as well. Uh, so if I wanted to delete
[18:51] something, sorry Australia, maybe it's uh not there anymore. Um, I can just update this version of my table to say actually version three only contains 0,2 and three. Now, um, now what if there
[19:03] was some data in just one of these files that we wanted to update? Let's say that's South Africa data. We want to update it. Um, if we update it, we can just make changes to that one file that we've got there instead of having to
[19:15] read in all the other stuff that wasn't relevant and write it out again. And we've got this new definition that's contained all of these things here.
[19:27] to read in an entire table and then rewrite it all out again when we want to make our changes like with the South Africa example. There's just one you know a few values in there or we can just switch the new version. We don't
[19:39] need to touch all the other data as well. Um no files were actually deleted when we talked about this. I know we talked about kind of deleting files. We just kind of ignored them. The files stayed there. We just didn't include them in the definition of the table
[19:52] anymore. Um, and something else which means uh is that we can roll back to previous versions as well. So if I accidentally deleted something, the data is not actually gone somewhere. We can just go back to that old definition of a
[20:04] table. Uh, and finally the mapping of where these files go to, which version it needs to be stored somewhere. And that's going to come on to some of our
[20:16] first decisions that we are making. But um so for now, let's say uh we're going to say we're going to store all of this on disk. Um and there are some other things that we're going to add in there as well. So things like the versions that we've got in there and the files.
[20:29] We could also add in things like what is the operation that made that new version. Um and we could also pull out some stats from paret as well like that. That's got those lying around. Let's use them and put them in this file as well. Um, and because we pulled out all the
[20:42] paret stats, we can also be now more even more aggressive um in our file skipping and do that much earlier on in the read as well. There we go. Better better skipping. Uh,
[20:54] now because we have all of these files flying around, um, it might be prudent to reorganize them, especially if they're too small. Um, and this is what we call uh compaction. So, if we've got too many small files, why not just
[21:07] compact them and have them as one big one? And we know that in future it's going to be much faster to read from. Now this does involve some like meaty compute. You do technically have to read it all in again and then write it out again. It's not a trivial operation, but
[21:19] if you've got a table that is being constantly used by hundreds of people, it is well worth it. Um and whilst we're doing this, maybe we want to sort of order the data. If we're going to read it all in and write it out
[21:31] anyway, why not try and sort of colllocate the data so that it's much faster to go through? And lastly, um if we have files that are no longer used, especially if we've got loads of files, um this can start
[21:43] affecting read time. So if they're not needed, we can just go ahead and delete them. And remember, uh this is uh we can get rid of these ones. This is not the
[21:55] same as like deleting the actual data itself. So let's say I don't know, let's say the analytics team is just like one dude that works and uh I don't know, doesn't do a lot of work. likes to go out surfing all day. Uh he never touches
[22:08] the data. It's safe to say actually he can come back to, you know, all of the tables even though it's been kind of cleaned up. It's still going to be involved in that last table and it's still all going to be there. So even if you're not reading from that particular
[22:20] file, it's still going to be in there as well. It is the versions of the table, not necessarily that. Okay. Um so let's stop dancing around the theory and let's start comparing some of these projects. So, uh, Hoodie was the first
[22:34] one out of the gate and it was created by the people in Uber back in, uh, 20, uh, 2016, um, when they were struggling to process huge amounts of their company data. Uh, they open sourced it in 2017
[22:47] and donated it to the Patri Foundation in 2019 and at 2020 it became a top level project. Um, Iceborn, Iceberg was the second one that was off the gate uh, with their project back in 2017 created by the people at Netflix. uh similar to
[23:00] Hoodie, it went to Apache later in uh 2018 and it saw a real explosion in community contributions in 2021 with all new file types and processing engines. And lastly, but not least, uh Delta,
[23:12] which is made by Data Bricks, the people we're here to see this week. Um and it was made back in 2017 and it was open sourced to the Linux Foundation in 2019, which is the year I joined. I don't think the two are linked because I had nothing to do it. Um data was a little
[23:26] bit sneaky in their open sourcing. Um, and they open only originally open sourced part of it, but in 2022 they kind of came to their senses and open sourced all of it and gave uh the community the good stuff. Um, it turns
[23:39] out people want to contribute much more to a project when they can get their hands on the good stuff. Uh, so let's start dancing around and time to kick the beehive and compare all three of them. Um, the first thing that I w I did
[23:51] earlier is I kind of cheekily described where we store the metadata. And I technically described it as this, which implies that it's stored next to the data, whether that's like S3 or ADLS or GCS or or whatever, just chuck it in
[24:03] some JSON files and call it a day. But that is not the only way. Now whilst this is the choice that Delta and originally uh sorry Hoodie and originally Delta weighed, it is not what the iceberg project chose and they went
[24:17] with a catalog. And so now we have to acknowledge that a new technology exists today and that is uh cataloges exist. Cataloges are sometimes called um uh meta stores because they store meta
[24:30] data. Uh and the basics kind of like map files to table names. So when you do like select star from Nando stores, it knows which files to go hunting for. There are new generations that do so
[24:42] much more uh with metadata like discovery like more than just tables and there are some things that you might have heard of in here as well in terms of like hivelu obviously unity and polaris as well. These are a lot of popular open source ones at the moment.
[24:54] So uh let's talk about iceberg metadata and if you've ever come across the idea of semantic satiation uh that's what this is. Uh, we're going to store the metadata and the list of metadata with the other versions of metadata and then
[25:07] store it in a meta store. Got it? Loads of metadata.
[25:19] Technically, Iceberg does work with more than just paret, but for now, let's just pretend that's our parquet files. On top of that, uh you then have a manifest file and that contains like the file locations, the partitions, some statistics in there as well like some
[25:32] min and maximums and some counts. And then after that, you've then got the mapping of uh which file maps to which snapshot version that we have here. And then finally, we've got like a snapshot version definition as well, which has
[25:44] got like schema and partition in it as well. This is the rough outline for iceberg in terms of the catalog. So uh where do we store all of this metadata? Uh so we've got uh iceberg
[25:56] storing it in a catalog rather than on storage in terms of how they do it. You know they've got a slightly different way whereas hoodie goes for a timeline approach uh and Delta goes for logs uh in terms of the format of those files as
[26:08] well. But Hoodie does actually have a little bit more flexibility in terms of what you can do with it as well. And in terms of the pro of like where to store this metadata, there are a few interesting consequences. So, uh, the storage side tends to make it like a
[26:21] little bit faster when it comes to reading, especially writing. Um, but from the iceberg perspective, um, it does allow for cool things like multi-table transaction. So, this is the ability to make changes to two tables at
[26:33] once, which is a really popular thing that, uh, people want from their data. There are some cons as well. Uh, so you can't do catalog levels commits for iceberg and what once upon a time was uh, sorry for hoodie and what was was
[26:46] delta. Um, and it does mean that iceberg is also dependent on your implementing a catalog as well. So if you just wanted something like super quick and lightweight, maybe you don't want to spend time setting up a catalog, then maybe iceberg is not for you. Um, all
[27:00] right. So um, now Delta has actually recently taken the direction to adopt a catalog too. Again, turns out multi-table transactions are things that uh, you as customers uh, have been
[27:12] putting as highly demand on there. And uh this means that if you want to do those coordinated commits, you need a place to store all of those uh metadata bits and pieces. And it turns out the best place to do that is in a catalog
[27:24] and where we coordinate all of that from. Uh all right. So in terms of these implications as well, um the way that this is done, it does make them acid compliant, but they do get there in
[27:36] slightly different ways. Now acid compliance, if you've never come across this is um this is remember failing properly. So, we're not going to have any partial fails. We're not going to have any half rights. Uh, we're not going to have like reading something
[27:48] before it's even done elsewhere. It's also kind of a bit of a marketing term as well, and it is open to a bit of interpretation. Now, there was a real ding-dong along a while ago between uh
[28:00] Hoodi and Iceberg about just how acid compliant each other's projects actually were. But in my view, pretty much 99% of the population will say uh yes, that they are acid compliant. Um I do think
[28:14] that argument did prove under extreme circumstances that yes, there are some considerations. Uh but if you are pushing it to the extreme, um you're I don't know why you're sat here listening to me. you could probably give stand up
[28:27] here and talk do this talk instead of me. Um, another impact of the implementation is like the ability to go back in time and do things like time travel as well and see older versions of the data. Um, and because iceberg has
[28:40] this at the catalog level, it can do like full catalog multi-table roll backs as well, which is a pretty sweet feature. And so next up is checkpointing. Uh this is a way of kind of combining all of the
[28:52] previous logs that we have so that we could speedrun all of our file history and go through it rather than going through one by one. And this makes for much faster reads. Um Icebug doesn't technically need this because it weigh the way that it organizes the metadata
[29:04] in the catalog. Um but for Delta and Hoody um they do it in parket and they do it in slightly different ways. Uh we also again don't need to hold on to all of this history forever. Um, and for this the approach varies in terms of how
[29:17] you're going to clean up that old data, how you're going to vacuum it. Uh, Hoodie has a cleaner that works kind of on a per commit level. Um, Iceberg gives those files an expiration date and so once it goes past them, it'll go and clean them up. And for Delta, you can
[29:31] give it like a specific timeline and say, "Hey, hold on to this for seven days." Um, but you can also do it manually as well. Um, okay. in terms of uh how to actually sort your data. So all of them have this
[29:44] ability to get rid of the small files problem by reading it all in writing it out in much sense more sensible sizes and that is called compaction. Um and so this is how we are going to fix that. But again given that we're about to read
[29:57] everything in and then write it out again as one big file maybe it makes sense to organize that data in a way that it is going to be much more efficient to read. And to do that we have different types of search algorithms. Uh we've got uh zed ordering
[30:10] and we've got Hbert curves as well. Um they are in both like both kind of like fractals that zoom in in different ways. By the way, when you go home to your families today and or tomorrow and you
[30:22] get asked what you learned about, I really hope with uh we learned about the importance of search algorithms within metadata storage. Um
[30:46] will try to sort the groups um of new stuff without having to rewrite the old stuff. I thought okay that might be slightly wrong but never mind. Um all right next up is uh deletion vectors. This is a
[30:59] highly requested feature. Um and this is uh the idea that we can basically pretend that some data doesn't exist anymore. or if I've got a monster table, I just want to delete one little record. Uh what I can do is I can implement
[31:11] something called deletion vectors where we just pretend that data doesn't exist anymore. We just tombstone it. Um and that is in deletion vectors. And finally, I want to talk about merge on read and copy on write. Um now hoodie
[31:24] has a lot more functionality here and technically iceberg and delta do this a little bit because that's technically what um uh deletion vectors are. Um I do want to give a bit of an example for this because it can be quite fundamental
[31:37] if you are not used to using hoodie. So if you've never come across these before right this is what copy on writing is two very different ways of reading your data. This is the most intuitive to understand. So we'll start here. Um each
[31:50] one of these colors represents like a base file. So let's say parket for our example. Um and when we read from it uh we're just going to read. But let's say we go ahead and we make a change to our data. So, we're going to go we're going
[32:02] to sorry, we're going to read from our data. Very straightforward. Um, but let's say uh we want to go ahead and we do an upsert and we're going to make some changes um to those uh base files that we've got there. And the next thing
[32:14] that we do when we read it, we just read from the most uh recent version. There's no tricks. This is like very intuitive like this is this is how it works for copy on write. It is not the only way.
[32:26] So, merge on read is where it gets really interesting. So let's say we start with our files and then we do like three appends in a row and we have three lots of logs appearing um in our file groups and we're going to keep going
[32:38] with our appends. Now we have two options. Um option one is a read optimized table. Um and that is only going to read from our green column and
[32:50] base file because it's going to be much faster. So we're going to speed up our read but it does mean that we're going to miss out on our recent appends. And you might be thinking like why would you want faster outofdate data? Well, for
[33:02] some use cases if you are constantly appending absolutely all the time but actually what you want is a you know a decent enough read um this might be something that you are doing as well. But then there's also a slightly different version of this table as well.
[33:14] Um and if you wanted the real time table um then actually then you do start combining those logs but it is a slightly slower read that we have here. Uh, all right. And then also, uh, we
[33:26] also have indexes as well. Hoodie has the option to bring a few different options that you've got for you as well. Um, which Iceberg and Delta don't. And you might be thinking at this point, Holly, I just wanted to store some data
[33:38] and you're giving me so much stuff that I need to think about. Um, you're right, candidly. Um, I do want to highlight this point quite intensely. Normally, when I give kind of talks like this, someone's like, "Oh, but I need like a
[33:50] very specific feature." And it's like, "Well, good for you." But for the vast majority of you, these are really niche features that you might not necessarily need. All right, so how do we head how do we go ahead and manage a schema? I have a
[34:04] data storage problem.
[34:16] enforced on write and there are types of different things that you can do here in terms of drop and rename. You can do it with Hoodie. There are some slight limitations uh because of the way that it stores metadata which is based on like a name and a position rather than
[34:29] like a canonical ID like what happens with iceberg and delta. Okay. Uh design choice five. I say choice in air quotes um are the things that you can have from an ecosystem when things like this are being developed as
[34:42] well. It is not just kind of like the storage that you're using whether this is I don't know S3 ADLS you can even use this on like on premises things as well. Um you also have choices when it comes to the engines and the integrations. If
[34:54] you don't like this list of engines there's nothing stopping you from building it yourself. Uh in terms of the catalog as well all of these have integrations with cataloges that you might want to use. Again it just keeps going in terms of like orchestration and also the platforms that you might be
[35:07] using as well. Now when it comes to ecosystem I will say um it tends to cause a lot of arguments. Um in my opinion like open-source projects everyone can argue about just how open
[35:20] source a project is whether this is the number of contributors the diversity of contributors the velocity of which they're rated like you're releasing features. One thing that I will say is that people like to argue about this but the people who argue aren't normally the
[35:33] contributors. Uh so I feel like this is kind of some unattainable thing that not everyone is going to agree on but there we go. All right. Um so let's have a look to see what did we go to set out and solve.
[35:46] Um and what do we have here? So the first thing that we wanted to solve for was acid transactions and rollbacks. That is absolutely something that you can do with all three of these projects. Uh in terms of the schema enforcement um
[35:59] because we've got schema on right for all of these things that is now going to happen. Um we can also solve for our small files problems as well with things like compaction and collocation. Um we also have advanced schema evolution with the way that we handle our metadata too.
[36:14] All right here is my opinion at the end. Um this is a means to an end when it comes to picking the table format that you want to use. Um you want to be able to solve for the projects that you are
[36:27] doing. There is a big overlap in the capabilities of all three projects. Now, the vin diagram is not a complete circle, but it's kind of close. And I will also say that if you have a rubbish project that is not going particularly
[36:40] well at the moment, the odds that it is going to be solved by a table format are pretty slim to none. Solve for the problems you have. Uh something else that is happening in this area as well is there is emergence
[36:52] of multiiformats as well. Again, 99% of the file format is still that paret stuff underneath. Like the clever parts are all in the metadata. And there's nothing saying that you can't replicate this metadata elsewhere. So, X table,
[37:06] this uh goes ahead and it makes all different three sets of metadata at once. Uh we've got uniform as well. Huh. Have we changed that name? We might have changed that name. We like to do that, don't we? Um or you can take a delta
[37:19] table and kind of read it as whatever. um but then also having something at a catalog level as well to kind of abstract away from that as well. Here are some things that are coming out um in this ecosystem.
[37:31] All right. So finally my opinion who has won I think uh it's you you the community you get all three projects for free that are heavily contributed to more than any other software enterprise
[37:45] enterprise software that we all probably use. Thank you very much. Have a wonderful summit. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.