Delta Kernel: Catalog-Managed Tables

Summary

This video explains how Delta Kernel — a Java and Rust library that provides a unified Delta table abstraction for connectors — has been extended to support Catalog-Managed Tables, which designate Unity Catalog as the authoritative broker for Delta commits rather than the underlying filesystem.
Catalog-managed commits address two key problems: engines bypassing the catalog and causing schema drift, and the inability to coordinate multi-table transactions without a trusted catalog acting as the coordinator.
The design keeps Delta Kernel catalog-agnostic by abstracting catalog interactions behind a unified API, enabling deep integrations for DuckDB, Delta-Spark, and Delta-Flink while shielding those connectors from the complexity of catalog-specific protocols.

Delta Kernel: Catalog-Managed Tables

With the introduction of Catalog-Managed Tables, Delta Lake has unlocked unified governance, enforceable constraints, and a path towards multi-table updates. As a result, Delta connectors must adapt to properly load, read, and write Delta Tables, as the Catalog is now the authoritative broker of commits, not the filesystem. In this talk, learn how we designed Catalog Managed Tables support in Delta Kernel—the one right Delta abstraction to rule them all. We’ve built a unified catalog-agnostic API against which connectors can develop, enabled deep and powerful engine integrations for DuckDB, Delta-Spark, and Delta-Flink, all while keeping the complexity of catalog support out of Kernel.
Talk By: Scott Sandre, Sr Software Engineer, Databricks ;

Chapters

FAQs

What is Delta Kernel and why does it exist?

Delta Kernel is a Java and Rust library that provides a simplified, unified interface for connectors and engines to read and write Delta tables without needing to understand all the internal details of the Delta protocol. Its goal is to make it easy for any query engine to integrate with Delta and automatically benefit from new Delta features as they are added, without each connector having to track protocol changes independently.

What are Catalog-Managed Tables in Delta?

Catalog-Managed Tables are a Delta protocol extension that makes the catalog — such as Unity Catalog — the authoritative broker of commits, rather than having the filesystem serve as the source of truth for what has been committed. All writes go through the catalog, which can enforce governance, maintain consistent metadata, and enable multi-table transaction coordination.

How does the write path change with Catalog-Managed Tables?

Without catalog management, a Delta write uses a put-if-absent transaction directly against cloud storage to claim a commit. With Catalog-Managed Tables, the engine creates a staged commit and then submits it to Unity Catalog to ratify — the catalog decides whether to accept and register the commit, enforcing consistency and enabling coordination across multiple tables in a single transaction.

How does Delta Kernel stay catalog-agnostic while supporting catalog-managed commits?

Kernel exposes a unified, catalog-agnostic API that connector developers code against, hiding the specifics of how different catalogs handle commits internally. This means connectors for DuckDB, Delta-Spark, and Delta-Flink can all benefit from catalog-managed table support without each needing to implement catalog-specific logic — Kernel handles that complexity on their behalf.

Full transcript

[00:08] All right. Uh, well, my clock just started counting down. Can folks hear me? Yeah, everyone hear me? Okay, cool. It's very weird that you can all just hear me, but that's cool. Um, uh, cool. I'm going to be talking about Delta kernel and catalog manage tables. Uh,
[00:20] I'm not going to be I'll um tell a little bit about what I'm going to talk about. I'm going to assume that people have a little bit of familiarity with uh what delta kernel is, but I will tell you uh briefly what kernel is and then
[00:32] if you have specific questions about kernel rather than kernel plus catalog manage tables, I'm happy to answer those as well towards the end. I will be leaving some time for questions. Um also, sorry, I'm going to end up looking at the screen a bunch because I don't
[00:44] get the slides over here. Um but yeah, I'll talk a little bit about what kernel is. Then I'll do some uh talking about what catalog managed tables are and why they're useful. And then I'll talk about how we integrated kernel with catalog
[00:57] managed tables. Um cool. So what is delta kernel? Uh it's a Java and a Rust library uh currently implemented in the two languages. We're working on unifying that. Um and it provides sort of narrow
[01:09] table APIs. The goal of which is to make it really easy for a connector uh something that wants some kind of engine that processes data to access Delta tables, read the data, not have to know
[01:21] the gory details of all of how Delta itself works and just have a nice easy interface to to work with those tables. Um but also to get all the latest and greatest features that Delta can provide
[01:33] as we upgrade and add new things um easily. Um yeah and then we also uh because engines care a lot about how data is read, how memory is managed, things like that, we also have a set of
[01:46] engine APIs which are ways that engines can control how Delta reads files, writes file, uh sorry, how kernel reads files, writes files, things like that. Um, so the goal is that engines can really seamlessly integrate with Delta,
[01:59] have everything happen sort of in the way that the engine wants it to, and uh at the same time uh not have to worry about exactly how Delta works. Um, so, uh, here's a partially madeup quote,
[02:12] although it's not really fully made up, but a bunch of different people I'll show on the next slide, but you know, we've heard from a number of people that, uh, when they used Delta kernel, they actually found it a lot easier to integrate with Delta than, for example,
[02:24] when they wanted to build their own, uh, iceberg integration from scratch. Not surprising, right? Like building something from scratch, it's harder, but if you really want to leverage all the capabilities of your engine, you kind of need to do that. Um, so yeah. Uh so it
[02:37] it's being used quite a lot now. I think this is the third year I'm speaking about kernel at DIS. So the first year I think I came up and said like it's really just duct DB had started testing it out. Um now we've got a whole bunch.
[02:49] So across the ecosystem uh you probably recognize a bunch of these. These are a combination of uh native connectors uh like delta RS and written in in Rust or Click House written in C++ or duct DB in
[03:02] C++. Plank, styox, confluent, these are all Java systems. Delta.net is another integration. So we have a wide set of integrations across all the ecosystems over uh FFI for CC code and so on. Uh
[03:16] also it's been really taken up within data bricks which is a a sort of a great victory for us because it uh sort of validates that people both inside and outside want to use it. So you've probably heard of zero bus at this point um do streaming ingest, delta sharing is
[03:30] using it and a bunch of other things. um who are all using Delta kernel internally. So it's really becoming sort of a a narrow waste of ways that people access Delta and it's going to be uh so
[03:42] you can feel confident if you're using kernel that it it's being used by lots of different connectors and should support all the things that you really care about. Um so what does it look like? Uh basically I mean this is not a super interesting diagram but you know
[03:54] the idea is basically your engine or connector can come along ask delta kernel hey I want to read uh some store some stuff. Delta kernel talks to cloud storage, gives you back some of the data that you care about and you're welcome to go off and answer your query. Um,
[04:07] this is a simplified version uh because the engine can kind of get involved again in the reading and writing. I'm not going to again not going to go into a ton of detail about that here. If you're interested in like the gory
[04:19] details of how a kernel works, we do have a user guide. So if you search for the Delta kernel user guide, this is something I said in my talk last year. Hey, we need a user guide. So that happened. Um, so you can find that online. Um, and that'll explain more about how sort of what we call the
[04:32] engine APIs, which are the things that would get involved between kernel and cloud storage in order to allow the engine to do its own data format, uh, reading and stuff, uh, are involved. Um, okay. So, that was what kernel is. Let's
[04:44] talk about like what's the problem if your if your table is just living over here in cloud storage and there's no no catalog involved, what's the problem, right? Um, so one problem is that you can have different systems that have
[04:57] different views of the data. So in this picture what we're showing is that say somebody's using Flink to access the data in cloud storage and they've dropped a column from a table right but Unity they didn't tell Unity about it.
[05:09] So now people coming in to the Unity catalog through the Unity catalog ext uh way of accessing the table uh are going to see that they think that this table has an old metric and a new metric column whereas in actual fact when they
[05:22] finally get there there's only the new metric column. So this can cause kind of like schema mismatches uh things like that. Um you know Delta still protects you from it still keeps a transactional view of the data. You won't get
[05:35] corrupted data but you know you don't have um you don't have the governance rules. You don't have the auditability that you get when you use the catalog that can track everything that's happening in the catalog. And you can get metadata drift so that like you can
[05:48] have the catalog view of the table be out of date from what's actually up in the in the in the files on disk or in cloud storage telling you what it is. Um so this is one problem. Uh another
[06:00] problem that you get is that we don't you can't support multi-table transactions. Um so delta is a sort of single table protocol. You use put if absent to to make commits to it. that doesn't work across multiple tables
[06:13] because you have to put two files and there's no put these two files if absent uh in cloud storage. So um you know but a lot of uh multi-table transactions are key in a lot of data warehousing
[06:25] workloads. So uh this makes it difficult obviously to migrate those workloads uh to to the lakehouse. Um so we want to solve these things. We want to put Unity
[06:37] catalog or any catalog really uh in the path of doing commits um and and reads um and so how do we do that but still sort of maintain the promise of uh
[06:50] kernel. So Unity catalog now supports what we call catalog commits. This is a a new addition to the Delta protocol. Uh you can go read about the gory details of of all the rules about it in the Delta spec if you're interested. But uh
[07:04] Unity catalog now supports uh catalog managed commits uh on catalog managed tables. Um and so what was the goal as we were looking at integrating this into kernel? Uh what we wanted to do kernel
[07:18] should make this as easy as possible. Again the promise of you don't have to think super hard about how delta works, how catalogs work if you don't want to. But at the same time we want to maintain the kernel philosophy which is we don't
[07:30] want to lock you into a particular catalog or a particular way of integrating. Maybe you have your own ways of dealing with these things as a connector. So we need to keep that flexibility. Um we wrote the spec um the
[07:43] delta spec to be independent of which particular catalog you're using. It has rules about where you need to write the data before you commit it and things like that. But it doesn't say you have to use Unity catalog. It just says you have to use a catalog. You're free. You
[07:56] can take your data wherever you want. You can use whatever catalog you want. As long as you abide by the rules of the spec, it's a valid delta table. Um, we also wanted no Unity catalog specific code in kernel. Again, wanting to
[08:09] maintain this independence. Um, and we so we shipped this UC integration as a separate create in the Rust ecosystem jar in the in the Java ecosystem so that you if you just want to integrate with
[08:21] Unity, you know, take the nice easy path, all that code is there for you. It plugs together. But if you uh want to do things your own way, you're more than welcome to. So, um I'm going to talk a little bit now about sort of the how
[08:34] does this work? Like what does it look like to do a read and a write if you have a cat without a catalog and with a catalog? Um and then I'll I'll be done basically after that and we'll have time for questions. So, let's say I want to
[08:46] to do a read uh and I don't have a catalog. Well, what happens? We've got our engine coming in. It wants to do some kind of read query. Um the first thing it needs to do, it needs to ask a snap for a snapshot. This is one of the
[08:58] APIs that kernel provides. So when you ask for a snapshot, you're going to tell it, hey, here's the path of the table. Uh here's where you can find this table in cloud storage. So kernel is going to go off. It's going to read that cloud
[09:10] storage. Uh getting a little bit into delta details here for those of you who know, delta is a series of uh JSON files that are written out in in a in a special location that record what's happening to the table. So, the colonel
[09:23] is going to go look in cloud storage. It's going to see, okay, here's the files that make up the the log. It's going to read those files. It's going to process them and figure out what what data is in the table. Uh, and then it's going to build either a snapshot or a
[09:36] scan depending on what you're trying to do. This is all the the the kernel logic that's abstracting the details of the of the Delta protocol away from you. It's basically going to give back to the engine and say here's a snapshot that you can get things like table versions
[09:49] or schemas from or here's a scan these are the files that are involved in the table you should read these files if you want to do this so this is sort of pre-C catalog uh as we saw in the in the previous slide right this is going
[10:01] directly to cloud storage we're bypassing Unity or any other catalog that you might have you got no no governance here you're just going straight to cloud storage kernel would need credentials to talk to this cloud
[10:13] storage bucket etc. Um, so how does this look like now if we introduce a catalog? So as we said, we don't want kernel to have to be in the way of you using your catalog. So
[10:25] the first thing you actually do is hit the catalog. And usually when you talk to a catalog, what you need is some sort of table ID, some way of identifying your table. Um, this could be, again, here I'm showing Unity, it could be a
[10:38] different catalog. All of the format here is specified in the spec. Um, so, uh, but we're just going to use Unity as an example. So, um, due to way the way the slides laid out, you have to read the arrows kind of backwards, but the UC
[10:50] client, which you could, this could be the one that we provide or your own one, um, calls a load table to Unity. And then Unity comes back and it tells you a bunch of information about the table, including the table path, but also
[11:02] importantly, it tells you something called ratified commits. It says, "Hey, some people may have committed to this table, and those files are not yet in the normal place. they're living somewhere else and here's where they are. Um, and we haven't done this yet,
[11:14] but you can even do imagine some like cool optimizations like actually here's some inline commits. Here's the data of a commit that hasn't even been written to cloud storage yet. This is only in Unity so far. So, there's all kinds of fun optimizations that that become
[11:27] available once you have cataloges involved. Um, and then you move on to the the next stage, which is going to look similar to what we did before, but you're gonna instead of passing only the table path, you're going to pass what we call the log tail, which is the end of
[11:41] the log that Unity just told you about. And what we see here is kernel goes to cloud storage and it notices that there's this ratified commit, which according to the spec has to live in this underscore staged commits directory
[11:53] um that it was told about. And so now Colonel knows, okay, I got to read the regular log zero and one, but also this new 002 uh file that's in stage commits. And there could be a bunch of other junk in stage commits which were commits that
[12:06] didn't succeed or were aborted or whatever. But based on what Unity told it or what the catalog told it, Colonel knows which of those files are actually included in the table and which aren't. Um, so you just pass that through and
[12:19] now it looks the same as before, right? So like you didn't really have to change too much. We managed to keep uh Unity out of kernel except for introducing this notion of a log tail and that is included in the spec. Ratified commits
[12:31] are part of the spec. So we're like okay that's good. That's delta specific not catalog specific. Um and you end up either with your snapshot or your files. Uh so that's the read path. Um and now
[12:44] I'll briefly go through the right path and then I can take questions from there. Um so what does a transaction look like without a catalog? So transactions are a little bit more complicated. Um, so the first thing you're going to need to do is get a
[12:57] snapshot. That's what we just went through. So you do the normal snapshot path that tells you sort of the latest state of the table. Um, and then uh you ask kernel, hey, I want to start a transaction. And kernel says, cool, here's a write context. The write
[13:10] context is basically all the information that you need to know about the shape of the data that you need to write. And remember, kernel wants to be out of the way of the engine. So the engine can actually go write the data files that it needs to go create. Colonel's not
[13:23] involved in that necessarily. It can help you if you want, but you can just go write your data files, but you do have to pay attention to this write context. It tells you a little bit about the shape of the data and it tells you sort of what metadata you need to give
[13:35] back to kernel. Now when once that data file write has completed, the the engine passes some file metadata uh to back to the transaction and says, "Hey, I'm ready to commit this transaction."
[13:47] kernel can look at that file metadata, kind of validate that everything's okay, uh, build it, build up what the commit's going to look like, and what it does is it just generates a bunch of actions, right? So, again, we're getting a little bit into Delta protocol details. Don't
[14:00] worry if you don't know what those things are, but basically kernel says, cool, this is the information that needs to be written to the delta log in order to finish out this transaction. And then it's going to call uh there's going to
[14:12] be a put if absent call. Uh this might go back through the engine, might not depending on how you've configured your integration with kernel, but it's really just telling cloud storage like hey I've had got these actions I know how to serialize them. Please put them into a
[14:25] JSON file and call a put if absent into the delta log. Um and if somebody else already got there first and tried to commit to the table, this is where your transactional integrity gets maintained. Um you'll get a failure. if you were the
[14:37] first one to try to commit commit, you know, version number two, you succeed and your table now has 0, one, and two in it. Okay, so this is no cataloges. Um, but as we'll see, we don't have to do too much uh when we start to
[14:50] introduce transactions with a catalog. So again, uh if you do have a catalog, we're going to do the same kind of thing to get a snapshot, right? We're going to load the table and get the snapshot the same way that we did before. Uh this
[15:03] time we do have to go through UC again. We're catalog managed. we have to do everything through the catalog. Um, but the rest kind of looks very similar for most of the query time, right? We're going to get the transaction. We're going to get the right context. We're
[15:16] going to write our data. All of this is happening just as it was before. We're going to give back the metadata. We're going to generate all of our actions again. But now, instead of calling put if absent, we can't do that. We're going to just take that that metadata or those
[15:30] uh actions that we produced and we're going to give them back to the engine via uh an interface that we've specified and say like hey here's the data that you engine need to commit. However you want to talk to your catalog I don't
[15:43] care what you want to do with these you can write them out to wherever you want. You just need to figure out how this this data makes it into the catalog such that it can maintain the contract that's specified in the delta protocol. Um,
[15:55] obviously our UC client that we provide does this correctly. So what you can do is you can just pass that straight into the UC client. Um, which we'll call commit uh, and then we'll commit to UC catalog and the it'll it'll write out.
[16:09] So I guess I didn't do that final bit, but the UC client itself will write out the stage commit file and then commit it to you to to Unity. And then you're you're basically done with that commit. And you can imagine that if you were
[16:22] doing a multi-table transaction, you would just replicate all of this this part down here onto another table and then they can both go uh I just didn't want that was that was too much to fit on one slide, but like it it's pretty much the same flow. Um so now without
[16:37] having to put bunch of information into kernel about how the catalog works uh or anything like that we've we've enabled doing transactions via kernel and you
[16:49] can plug in this the existing UC client if you want you can use your own ductb for instance already had written a bunch of UC code so they use their own client but a lot of most of our internal consumers and others are using the uh UC
[17:02] client that we provide. Um, yeah, and I think I'll probably stop there and and go to questions since we only have a couple minutes left. So, thanks everyone and uh thanks

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.