Using Delta-rs and Delta-Kernel-rs to Serve CDC Feeds

Summary

• Serverless CDC delivery: Delta-rs and Delta-kernel-rs open-source projects enable reading Delta tables and change data feeds in Rust or Python, bringing CDC into serverless environments without Apache Spark.
• Change data feed applications: CDFs are a common tool for synchronizing changes between tables and scaling data processing, now accessible through serverless platforms like Lakestore.
• Databricks ecosystem integration: These tools work with Delta tables and Unity Catalog, enabling governed change data feed processing outside traditional Spark-based architectures.

Using Delta-rs and Delta-Kernel-rs to Serve CDC Feeds

Talk by Oussama Saoudi, Software Engineer, Databricks ; Stephen Carman, Senior Resident Solutions Architect, Databricks
Change data feeds are a common tool for synchronizing changes between tables and performing data processing in a scalable fashion. Serverless architectures offer a compelling solution for organizations looking to avoid the complexity of managing infrastructure. But how can you bring CDFs into a serverless environment? In this video, we'll explore how to integrate Change Data Feeds into serverless architectures using Delta-rs and Delta-kernel-rs—open-source projects that allow you to read Delta tables and their change data feeds in Rust or Python. We’ll demonstrate how to use these tools with Lakestore’s serverless platform to easily stream and process changes. You’ll learn how to: Leverage Delta tables and CDFs in serverless environments Utilize Databricks and Unity Catalog without needing Apache Spark
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What are Delta-rs and Delta-Kernel-rs?

These are open-source projects that allow you to read Delta tables and their change data feeds in Rust or Python without requiring Apache Spark.

What are change data feeds used for?

CDFs are a common tool for synchronizing changes between tables and performing data processing in a scalable fashion across systems.

How do these tools work with Databricks?

Delta-rs and Delta-Kernel-rs integrate with Databricks and Unity Catalog, enabling change data feed processing while maintaining governance.

What is Lakestore in this context?

Lakestore is mentioned as a serverless platform where these tools are used to easily stream and process changes from Delta tables.

Full transcript

Good evening and welcome to our final presentation of the day using Delta RS and Delta Kernel RS to serve CDC feeds. I will now pass the presentation over to your presenters and. Hello everyone. Thank you for coming out. Uh as our uh presenter had just said earlier, this is using Delta RS and Delta kernel to serve CDC feeds. And so some paperwork to get out of the way. Just forward-looking statements as well as uh please go and fill out your surveys.
I love statistics. So if you do, I will definitely see anything that is put there. So please go and do that if you get a chance. We appreciate it. And so let's start off by going through what we're going to do for an agenda real quick. Um so is going to start us off by talking a little bit about what CDF is like what its application is and you know the data space of things. Um he's also going to talk a little bit about Delta Kernel.
If you haven't gone to any other talks this week about kernel we'll talk a little bit about what that is, why you should use it, why it's cool, things of that nature. Um and then the challenges that we had when we were implementing CDF for Delta specifically. Um there is some nuance to that we'll get into. Um and then I will take back over uh and talk about Delta RS and its CDF implementation and how that is leveraged through kernel.
Um the problems that we had with the initial implementation um how the kernel fixes those problems and then how we actually did the integration. Um, and then finally, we'll have kind of a combined demo and questions going. At the end, I'll uh I'll just show a little bit of um of using some Sparkless uh some Sparkless um Delta RS to do some analyzation of some CDC feeds. And yeah, and I think we'll get started. So, without further ado, uh Uh I guess that was obvious.
Um, and so, uh, I'm a senior RSA here. So, I work in the profs serve. Um, I've been a core contributor of Delta RS for I think it's been two years now, something like that. Um, I'm the implement of a bunch of stuff in it. Check constraints, CDC, um, Unity catalog integration, things of that nature. I've been around Rust for about eightish years, something like that. I time flies when you're having fun, I guess. Um, and then here you go.
Hey folks, um, I am a software engineer at data bicks. I work on the delta kernel rust. So this is like sort of the core of this delta protocol implementation that we'll be talking about today. I'm the author of the CDF and incomit timestamp implementations uh, in kernel. And finally, I am a systems and rust enjoyer. So if you want to talk databases, come on up and talk to me after this talk. All right. So I want to introduce to you like what is change data feed and why do you want to use this thing.
So change data feed represents all of the changes that have happened at each commit on your table. So you could take a look at you know at version one I added a column or added a row at version two I added another one. Version three I deleted one and added a new one. And so you can think of this as sort of the deltas that have happened at each version of your table. And if you take the integral of that, if you bring it all together, you have the final state of your table.
So what does this get you? Well, one application is this idea of incremental materialized views. So let's say I want to maintain this idea of, you know, the total price of all of my products. And this is a bit of a dummy example, but it gets the point across. And suppose I want to keep track of that for every uh day. You know, I want to see what's the total price at all times. If you had to run this query repeatedly, that becomes really expensive, right?
So, how can we make this more incremental? How can we pay less on our compute using CDF? Well, so instead, let's keep track of like at each version what changed. You know, over here, I added two new items. Okay. Well, I only have to increase the value and my like total revenue by just that amount instead of doing a full table scan and reading the whole table and figuring out all of my total revenue. Right? So, this becomes a much cheaper query compared to the full table scan.
Another one is ETL. Let's say you're bringing in a bunch of data, maybe these JSON blobs, and it's unstructured, it's not cleaned, and maybe you want to have a cleaned up version, your golden table. Your ETL pipeline can consume that change data feed, see what are all the changes that have come into this rough JSON blob table and refine it only the things that have been updated. You don't want to repeatedly update and refine the things that uh have come in previously.
Now comes in the delta ecosystem, right? Okay. So this delta table is has the ability to showcase these uh change data feeds. But for every connector that you have in your ecosystem in your data platform, they have to reimplement that over and over and over again. So this is where kernel comes in, right? You want to do your the CDC and you want to consume it from all these different places. The idea is for kernel to bring this all together to one spot so that not only can you do change data feed but you can also do type widening and look at identity columns and so on.
The way that I like to pitch kernel is a small trade offer, right? As long as an engine or a connector is able to give us the ability to read JSON and read parquet, we give them the full delta protocol. And so with that, any engine that is able to read parquet or JSON is able to write read these change data feeds. That becomes pretty powerful, right? You can now use duck DB. You can connect to it through umnet or you can connect to it through uh flink or trino and so on.
Now implementing CDF in Delta kernel was a little bit tricky right at a high high level. You need to get a log segment. You need to go through that log. You need to figure out the paths of the of the data files that have been added and removed. We also have to consider deletion vectors and that's tricky. Okay. So we need to resolve those. And finally, we read the data files and tell you what has changed in your data so that you can go ahead and keep that incremental materialized view.
So you can do that ETL pipeline. So let's take a look at an example. I create my delta table. First commit. I added some data. Okay. Well, let's say I want to remove some data. Okay. Well, we added a deletion vector. Logically, what we did was we removed the old data and added it back again with a deletion vector. That's how it looks like in the delta log. And let's say I update it again. I remove another row. This is what it looks like, right?
Logically, we remove that old path. Remove that old data and DV. We add a new one, right? So, this is sort of what we're given from the delta protocol. And this is what we need to wrangle. Now let's take another look at a different example. Right? Let's say I have some more data coming in and I did an update and logically inside the delta table what happened was we updated but physically what's going on is we removed an old data file and added a new one.
So we have to resolve that and sort of figure out what has been changed in each update. And so here's an example of us resolving these deletion vectors and figuring out what's been added, what's been removed at each version, right? And showing you some of the complicated deletion vector logic. you know if it's if it was previously removed and then afterwards it was not removed then has been added right so we have to do all of this tricky uh sort of logic in delta kernel in order to surface that to all the connectors in the ecosystem let's take a look at uh this example over here where we have a CDC file and once again it's delta log operations that the connector ecosystem would have to go through.
They they would have to go and figure out, oh, I need to go read the CDC file and figure out what's the pre-image and what's the post image. This is stuff that we've all that we've implemented in delta kernel rust. That's not all. There's also time stamps. Now, you're able to do time travel over your table. You're also able to do CDC over time stamps in your table. I want what have been the changes between Monday and Tuesday.
Colonel has to go figure that out so that you don't have to in your ecosystem. Let's say I want to find the version at 190. We are able to use file modification timestamps and just go straight away we found it. There's trickier cases with more advanced delta features in incomit timestamps where the time stamp is inside the commit. You have to do a read. So it's a little bit trickier to find. Let's say I want to find the first version that happens after time stamp 220.
Well, we know it happens after the incomit time stamp. So, we have to go read it. How do we do this? We go do a binary search. Okay, let's start at the middle of our range. 300. Are we there yet? Uh, I don't know. Let's go to the left. 200. That's too low. 225. Perfect spot. And why am I showing you this? This is again complexity in the protocol that we've abstracted away from connectors so that the entire ecosystem can do it and not just spark.
So delta kernel in short implements all that just once for the entire ecosystem. I'll pass it off to tell us about Delta RS. All right. Uh my mic works right or am I not yelling into it enough? Um okay. Okay. So, let's then talk about Delta RS, which is um the Rust implementation of the Delta protocol. And so, this is a project that happened uh that's been around a little bit longer than the kernel. Um but it's, as I said, an open source implementation there.
And so, we implement the majority of table operations in in Delta today. If you use Delta in data bricks, you probably will notice there are a couple things that you can't do in Delta RS that you can do in data data bricks. Um, but we get the majority of stuff. If you want to merge, insert, update, things of that nature, then you're good. But, um, if you want some extra features like say deletion vectors or uniform or liquid clustering, those are not available right now in Delta RS.
It doesn't mean that we won't get there eventually, but they're not there today. By the way, if you want to contribute, by all means, we'd we'd love to have uh additional people come and help us with that. Um but yeah, we've been working on this for about five years now. Um and so um the majority of the contributions from this are actually um non-data bicks uh contributors as well. Um, I'm actually I well, PAC was not always a data bricks employee when he contributed, but I believe I I was the only data bricks contributor for quite some time, but the majority of this is is owned and driven forward by uh non-data bicks um people.
So, you know, it really is like a open-source like you know, out there in the wild. So, anyone can contribute, come and talk to us about it. Um, you know, we'd love to have extra people coming and help. And so let's look at a brief example. I know this is not really a talk so much about code, but let's just give you a brief example of what reading CDF from a delta table looks like. So this is a example taken just right from the documentation.
Um you can copy pasta this into a uh a file and compile it and run it. Uh, Granite, you have something at that's uh at um the uh I went one too forward. Granite, you have a table actually at that location there where my first arrow is pointing. Um you know, you will be able to just read a Delta table through that. Um from that we then create a data fusion session and associate um dataf fusion with that um with that table and we then load CDF.
CDF uses a common builder pattern for it. So you specify in this case I just specified an end version um and so in this case it would use the default version of zero to start. So this would be a a read from version zero to version four of said table. And then you just you know uh build it. And so after that there's a little bit of housekeeping for partitioning in order to kind of distribute this. And then we just collect it and print it out.
And so this is what like um 10ish lines of code. Uh hope if I can't count right please don't uh hold that against me too much. But um this is a very um this is a very low low amount of code in order to get like a CDF read. And just to be clear here, there's no spark involved in any of this. Uh there's no JVM here. There's nothing like if you were to build this and give someone else the executable. Granted, they also have a table at said location.
This would just work. No JVM, nothing else. They don't even need Rust installed. It's a statically linked binary that you can just pass around to anyone. And so now that we've looked a little bit about that, let's talk about the initial implementation of CDF in in Delta RS. Um, as the author of it, I also know where all the skeletons are buried for that implementation. And pretty much every mistake I made along the way in in doing this.
And so, um, the first thing and the most obvious thing was there's no deletion vector support. So, if you had a table that had deletion vectors enabled on it, uh, it's just not going to work. Um, and as had mentioned a little bit here, that's mostly because we don't have deletion vector support anywhere in Delta RS right now. It's something that we'll eventually get from kernel integration, but um, it was really it was it's very difficult to actually figure it out.
Assama spent a lot of time doing it for kernel. Um, but at the time I was just lazy, I I guess is the best way. I don't actually remember why I I didn't implement it at the time, but it would have forced me to basically implement it everywhere in kernel or not kernel in delta RS and we didn't have it at the time. So I wanted to get CDF in. I didn't want to implement deletion vectors in literally everywhere and so we just sided with no deletion vector support for that.
And surprisingly that actually worked out because we eventually wanted to get it through kernel anyway. And so in in an opposite case, if we had implemented it, we would have had to rip out all of that logic in order to let kernel do the deletion vector stuff and we would have had a bunch of code that we would have eventually just gotten rid of anyway. Um, another thing is um is that if you try so there was a time where if you go and you install an old version of delta RS and you try to read a table with CDF and deletion vectors enabled, you'll get something back.
Whether it's correct, uh, I don't know. Um, that was a bug that we eventually fixed in a in a subsequent version where if you have deletion vectors enabled, since we don't support it, you just can't read the table anymore. So it's no longer possible to get entirely incorrect who knows what it is but for a time that was also possible. Um additionally timestamp support um both in kernel as well as in delta rs has been um I guess inconsistent at best is maybe the right way to describe it. um delta kern or delta kernel um always had uh file modifications as well as like the JVM implementation also uses file modification times for doing some of these operations sometimes eventually another feature in Delta came along called incommit timestamps which remediated the issue of needing to look at the file modification time turns out not every file system actually gives you good file mod time so relying on this didn't always work correct me if I'm wrong there Um yeah, another problem that uh we tried to solve with increment time stamps was that it was not monotonic and so you could go up and up and up and then back five days and then forward 10 days.
Uh it was the wild west in the old mod um file modification time stamp. So in commit time stamp insured we only go forward in time. Yeah. And you know if you've ever written timestamp time zone code in literally any language, it's always a pain. I always get it wrong personally. So you know um this was especially difficult in an initial in the initial implementation I just used file modification. So there was no ICT um awareness at all.
So like had said if you had some wacky delta log that went forward backwards you know up down left right whatever then sometimes that would be confusing because um while my example only showed off reading versions you could also use time stamp ranges to also read versions. So, uh, if the commits messages themselves were all messed up, then timestamp reading was also kind of messed up, too. Um, and so, as I had mentioned, I know I always get ahead of my slides before, but it would lead to basically inconsistencies in the reads based on the time stamp.
And so, what we did was um, Osana and I have been talking about this for many months, but we eventually led to replacing the read path of Delta RS with the kernel's version of it. And so, um, this actually was really easy. Um, I actually just deleted a lot of code and replaced it with the kernel version of it. Um, we're currently in the process of doing a lot of kernelization in Delta RS right now. So, um, you know, there are a lot of moving parts in here that could change and, you know, go in different directions.
Whoops, I was not supposed to show that just yet. Um, but uh, you know, there's still some testing that we have to do for this, but for the most part it it kind of came out to basically like a net 10,000 lines of code like removed. And so um, now when it breaks, go to. Don't my problem now. Don't come to me. Well, you can still come to me, but I'll probably just send you to And so just, uh, as a little PR message, too, uh, we haven't merged this quite yet, so that number may change.
Don't don't hold me to that one. Um but so the end result of this integration was that like um now you get deletion vector support. You can read it with a you can read it with a table with deletion vectors enabled. You will get correct consistent results. Um you can you can if you're using an older version, you now get correct results. If you're using a newer version, you now can actually read CDF. Um, you also can have whatever wacky timestamp time zone versioning like weirdness and the timestamps will resolve correctly.
Um, because now we have incomit timestamp, but if that's not available, it falls back to file modification time like what it was previously. So, um, results among that again consistent, correct, everything like that. And, um, I don't have to manage like a large swath of code. Um, I think I've only done actually two or three bug fix PRs and releases on the CDC path. So, um, while I normally don't write very good code, this is actually exceeded my expectations for amount of weird mistakes I've made throughout the time writing it.
And so now now it's less of a maintenance obligation for Delta RS. We have enough code we have to rip out and kernelization. This was one of the easier ones to kind of get through. And so um but uh that's not to say that there isn't future stuff that we could do that would you know help us make this uh better. And this is both work on the kernel side as well as on Delta RS's uh side of things as well. Um probably the first one would be more data fusion integration.
Um, Delta RS uses data fusion all over the place to do a lot of things with um with merging, inserting, like essentially anything that does like a SQL oriented thing, check constraints, they all require an actual execution engine which data fusion provides for us. And so, um, CDC currently from Colonel actually, um, returns these things called record batches, which if you've gone to any of the 3,000 talks about Arrow across the week, you'll probably hear about record batches somewhere in there.
They're the general abstraction for kind of a blob of data in Arrow. Sorry, any AROP PMC's in the in the crowd that was totally wrong. Sorry. Um but essentially um it currently returns record batches out of it and preferably we would like it to return just files to read because that would allow us to wire it more into data fusion to get better um like metadata statistics things of that nature. basically build like a more SQL oriented like logical plan instead of what we do right now which is basically just take the record batches put them in a data source execution in data fusion and hand it back to you.
So the ability for like the data fusion planner and optimizer to do stuff reorder things like look at it uh is essentially none right now but um you know that's something that we can definitely do down the line. Um, and this would allow us to do things like I had said. So we could separate like just scanning metadata, just figuring out what we have to read versus actually going and reading it. Currently kernel reads both both things.
It does both sides of it. And normally what we would do for kernel is we would plug in um we would let the engine actually do the reading of the data path. The kernel would just tell you what to read. And so, um, but currently we're we're kind of breaking our own contract a little bit here. Uh, just look the other way for now. Um, and so, uh, and so, yeah, but again, like this is something that we're going to have to uh, you know, there's a lot of open discussion about where and what needs to go where for this.
So, it's kind of an open and involving topic. Um, again, if you'd like to contribute to any of these things, you're more than welcome to join the conversation. We're always happy to have more people um in it. All right, so uh let's hope that I can do this considering I'm kind of navigating from a window that I can't see on my end. And so um so I've just taken the time to create a very small, not very good um lakehouse app that is deployed on data bricks.
And so um you can see my URL up there, datab bricks apps. This runs on AWS. And um this just reads a very uh a very small CDC table that sits in data bricks. It's a managed Unity catalog table. And so I have the versions that you can actually read from uh or that or that. And so since I don't um specify a version here for my my neat little app, it actually gives me a rundown of all of the subsequent versions and what has actually happened in them.
So we can see like for instance in commit version 10 I inserted uh three rows you know so on and so forth. Update updates happened in um version 4. talked a little bit about the update pre-image uh update post image type thing. Um and you can see I've done a bunch of deletes and a couple inserts across the line. And so again this is a uh this is a Unity catalog managed table. this uh delta RS actually gets Unity catalog vended credentials from this.
So this doesn't have AWS keys signed up to it. It doesn't have any idea where this table is. It actually goes to the Unity catalog API to go and grab these credentials and go and know where to read this from. So um this has no credentials essentially given to it or associated with it. It gets all that stuff through Unity catalog. So, uh, I know the talk's not about Unity catalog APIs and credential vending and stuff, but it does make doing a lot of this stuff a lot easier as well.
And so, if we go up here and thank god this has been easier than I thought and say like we'll change it to four and then we'll go to say uh eight. Why not? And so, we can then reread and we will get, you know, just the things that happened in four and eight. So just to show that this is not a pre-anned kind of thing here. But if we scroll down a little bit, I have also taken the time to um do a little bit of this will generate some very mild code for you that you could potentially run in data bricks with some light modifications here.
Um and this actually looks at the schema of the table that delta RS gets and will build say your DBSQL statement or your Python code to dynamically build this. So this uh this table actually has a schema of name, age, salary, birthday. I just made it up. Um and so it's all dynamic based around that. And so if you plug in a different table, it will obviously build a different merge statement based on what the schema is here.
And so with a little, like I said, with a little bit of modification, you can copy pasta this into either notebook or DBSQL uh console and run it. And it'll probably break the first time. You'll probably have to massage it a little bit. I can't guarantee this works 100% of the time, but um you know, all all the basic building blocks are there. The only reason I didn't um actually do the right back with Delta RS, which totally could happen here, is because um Unity catalog vended credentials for rights uh hadn't come out yet.
Um, and so when that is available, you'll be able to do all of this through Delta RS. Maybe in a lakehouse app if you can somehow do it in six gigs of memory. Um, that's still kind of a small amount of memory for anything, but um, but yeah, so I just wanted to give you an example of something that you could do that, you know, uses all this. There's no Spark involved in any of this. And yeah, I think um, do we have any other questions or anything?
If not, I think we're uh I think we're good. Um this will also be um on GitHub later. I have to open up the repo. It's private right now. And five five minutes to spare. Not bad. Yeah, we have five minutes. Um feel free to stick around and ask questions if you so desire. Thank you very much. Thank you. Uh, oh, I have to also say do your surveys again if if please if you want. I won't be offended if you don't.

Learn more about the Databricks Data and AI platform.