Delta Kernel for Rust and Java

Summary

• Simplified Engine Integration: Delta Kernel makes it easy for diverse query engines—DuckDB, ClickHouse, Spice AI, and others—to read and write Delta tables without protocol complexity.
• High-Performance Library Design: The kernel streamlines APIs to let engines integrate the way they want while handling Delta protocol details, enabling consistent performance.
• Expanded Capabilities: Recent additions include write support and Change Data Feed (CDF) scans, expanding Delta Kernel's functionality for operational use cases.

Delta Kernel for Rust and Java

Talk by Nick Lanham, Code Wrangler, Databricks
Delta Kernel makes it easy for engines and connectors to read and write Delta tables. It supports many Delta features and robust connectors, including DuckDB, Clickhouse, Spice AI and delta-dotnet. In this video, we'll cover lessons learned about how to build a high-performance library that lets engines integrate the way they want, while not having to worry about the details of the Delta protocol. We'll talk through how we streamlined the API as well as its changes and underlying motivations. We'll discuss some new highlight features like write support, and the ability to do CDF scans. Finally we'll cover the future roadmap for the Kernel project and what you can expect from the project over the coming year.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Delta Kernel and why is it important for query engine developers?

Delta Kernel is a high-performance library that simplifies integration with Delta Lake for query engines and connectors. It handles the complex Delta protocol details, allowing developers to focus on their engine's strengths rather than reimplementing protocol logic repeatedly.

Which query engines can use Delta Kernel?

Delta Kernel supports multiple query engines and connectors including DuckDB, ClickHouse, Spice AI, Confluent, and delta-dotnet. The kernel is designed to enable easy integration across diverse data processing systems while maintaining consistency.

Best database for AI agents that need to read and write to the same table.

AI agents often need concurrent access to tables—reading data for context while simultaneously writing new results or state updates. This video covers Delta Kernel, which provides a unified interface for reading and writing Delta Lake tables efficiently. Delta Lake's ACID transactions ensure that agents can safely read and write concurrently without data conflicts, making it ideal for AI agent state management and result storage. Delta Kernel optimizes this access pattern with language-specific bindings for Rust and Java, enabling high-performance agent workloads.

What are the new features in Delta Kernel like write support?

Recent enhancements to Delta Kernel include write support, enabling engines to modify Delta tables, and Change Data Feed (CDF) scan capabilities for capturing data changes. These features expand Delta Kernel's use cases beyond read-only operations.

What is the roadmap for the Delta Kernel project?

The Delta Kernel project continues to evolve with planned enhancements over the coming year, focusing on additional features, performance improvements, and broader engine support. The team welcomes community contributions to accelerate development.

Full transcript

I think we're going to go ahead and get started. So, I'd like to introduce Lick Nick Lanam who's going to present the Delta kernel for Rust in Java. All right. Yeah, thanks for coming everybody. You can you guys can hear me, right? Um, cool. Uh, yeah. Yeah. So, I'm going to be talking about the Delta kernel for Rust and Java. Um, it's going to kind of be more of like a highle view, some of the things we've been up to, some of the lessons we've learned, some of the differences that have happened between the kernels.
Um, I'm going to Oh, yeah. I have some slides that I'm got yelled at for not having in my deck initially. So, you have to all look at these for a second. Okay, there we go. Um, so, uh, I'm going to spend a little bit to talk about what is a delta kernel anyway. Uh, for people who are already familiar with the kernel, uh, this part might be a little bit slow, but I think the rest of the talk doesn't make a huge amount of sense if you don't really know what it is.
Um, there's also a sort of a more deep dive into how this works, uh, and like details about the kernel tomorrow at 12:40. Uh, and are presenting that. So if you find this interesting and you want to know more about like the gory details of how kernel works, um that's a good session to check out. Um yeah, what is it? Uh this is heavily paraphrased, but basically it's a way for other connectors uh like things like ductb, click house, spice, confluent, all these kinds of systems to be able to read data out of delta tables um and to make it really easy.
And in one way or another, they've all said that it made it really easy for them. Um, okay. So, if this was sort of a an easy thing, we might make it look like this. Uh, kind of your standard diagram of how a library works. You know, this is very similar maybe to uh how like a compression library or something might work where you sort of say, "Hey, I've got some stuff coming in. I know that you understand how to get this data out of cloud storage and back to me.
So, I'll just call into you. You'll do all the work." and to give it back to me. Um, so if it was easy, I probably wouldn't be up here talking to you and also we would have just done this. But this kind of blows up somewhere, right? So engines or connectors have really strong opinions about things like how I do IO, what format I read my data in, how I do expression evaluation, all these kinds of things. Do I distribute my computation?
Do I not distribute my computation? How do I distribute my computation? So this arrow is kind of like where all of those decisions get made. Uh because the kernel has to go read the data. So it does IO, it has to read it into something. So it does format stuff. It has to transform the data. So it does expressions. Um and so if we hide this arrow away from the engine or the connector, then I just have no no ability to control that as the engine.
So we can't do that. Um, so you know there's another option which would be to provide extremely low-level APIs uh that let the engine kind of hook into every little bit and piece and basically write their own version of a delta connector. Um, which is kind of how things were before the kernel. You would just go implement your own version, read the spec, do all the all the work that you need to do. That's really annoying.
Um, it means that you have to be on the treadmill of features that are showing up and having to implement new all the new features as they come out. We get like PE connectors that are behind. We can't like upgrade tables to use the new features because too many people aren't unable to understand those features. So, what we're trying to sort of do um we don't want this too high level. It'll be slow because we'll be converting in and out of engine formats or doing things that engines don't want.
We don't want to be too low. That's too many delta details. So, we're trying to find this kind of like Goldilocks zone um which is basically APIs that let the engine have enough control uh but not make all not make them understand the gory details of the Delta protocol. Um and so sort of what defines that zone, how do we think about what that zone is? We want to be able to have every engine kind of get the best performance as good as if it had written it all by itself just about.
Um we definitely don't want to force any particular data format on you. Uh, if you like to keep your in-memory data in your own crazy custom format, that's great. If you like to use arrow, that's great. If you like to use something else, that's great. Um, and we assume that engines or connectors already have a bunch of abilities because they already are engines. They already know how to do a lot of this stuff. So, they know how to do things like read and write data files, JSON, parquet, that kind of thing.
They know how to evaluate expressions. These are things that most data processing engines already need to know how to do. So we want to be able to let them leverage that existing stuff and get all the performance benefits that they get by all the work that they've put into those things. Um and so we come up with this kind of thing that we call the U architecture. So basically how this works is queries might come in uh from the users and this big blue U hence the U architecture um is the connector uh and they'll start they'll enter the kernel uh via these table APIs.
They'll say hey delta kernel I need to read some data or I need to read this table I want to scan it or I want to start a transaction on this table and do some work with it. Delta kernel will say, "Okay, cool." It'll start to figure out what it needs to do. And then as soon as it needs to do any actual real work like the IO data reading, expression evaluation, it'll come right back out to the engine via um an interface that we call the engine APIs.
So, uh for all the Java people in the audience, it's an interface. For all the Rust people in the audience, it's a trait. Um, and uh, this is just a set of things that are basically functions that allow kernel to read JSON, read paret, evaluate expressions, all the things I just said, right? So, we come right back into the engine to do that work. Uh, so the engine can go off and schedule that work however it wants, do all that, and then it can give that data back to the kernel, and kernel can can do more work.
And um I'll talk a little bit later about the data model that we have for representing this data that comes back. I think that's a bit of an interesting case where we actually diverged between Rust and Java because of the way that those two languages handle sort of data ownership issues and things like that. Um so yeah um so that might sound like a lot of work. uh you know implementing all of these APIs for reading JSON and parquet and evaluating expressions requires you to understand a fairly large surface area.
Um you know it it lets you leverage all of your nice good benefits of your engine. But you might sort of think well I don't really want to spend all this time hooking up my expression evaluation to kernel expressions understanding all of the like data model stuff about how I represent data to kernel. I just want some data. So, uh, we've done that hard work for you. Um, at least on the Rust side, um, uh, and on the Java side, but, uh, on the Java side, the data that you get back is more like it's going to implement an interface that you can pull out.
It won't be in your own engine format as much. Um, in on the Rust side, we use Apache Arrow, uh, which is a really cool, uh, in-memory format. It's sort of language independent and it lets you define how you lay out data in memory. Uh and you can do zero copy transfers between uh different different systems. Um so you could ask the default engine. So that's what we call this to go do all the work and you kind of get back to that original picture that I showed um where it's just like one big arrow easy just like a like a library.
You call it it gives you back data for the table. Um, and oh right, I was like, I should have put a slide in that had that picture on it. And there it is. Right. So, um, so if you use the Delta kernel plus default engine and you are committed to Apache Arrow, then you can get everything for free. Um, or not for free, but it becomes really easy to use this library. Um, cool. So, a bunch of people are using it. Um, it's it's pretty exciting.
Uh we've got integrations from uh Delta RS which is a a Rust data processing engine. Confluent has written a connector using the Java kernel. Duct DB has integrated it. Click House has integrated it. Spice AI has integrated it. Um we have a wonderful person here who is uh doing a net integration who I asked for a logo and he didn't send me one. So that's why it just saysnet. Maybe someday you'll send me one. Um but yeah, a bunch of different people are using it.
Um, this gives us relatively good confidence that the APIs are somewhat reasonable. Um, we on on neither Java nor Rust have we declared sort of like API stability yet. We're not willy-nilly breaking things on purpose, but we're still sort of reserving the right to say, well, these APIs ended up not being exactly what we wanted. We're changing them slightly. Um, but we don't anticipate huge changes given that we have a decent amount of people who are integrating it.
Um, and uh, none of them have come back and said, "Whoa, this totally doesn't fit our data model, or it just really doesn't fit the way that we do things." Um, back to that early quote slide, most people are like, "Oh, yeah, this makes sense. I can plug this in. It's it's pretty good." Um, cool. Uh, so before I move along, I'm going to now that was kind of like the what is kernel. I was aiming for 10 minutes and I'm almost exactly right.
So that's great. Um are are there any questions at this point in terms of not understanding kernel or things like that? Yeah. Sorry. Can you speak up a little bit? Or maybe there's a mic. Sorry. I I should have Let's Let's get your mic. I should have warned them that I might ask for questions in the middle. the U engine that you are talking about like know the query engine the U-shape architecture. So and what was the question?
So is the is there any specific uh software like know are you built into something is it part of the delta kernel? Ah you're saying this this blue part. Yeah. So the right so like the right hand side of this we are building that is this picture right. So you are wait sorry I'll go back and forth for a minute but like here the connector gets shoved all the way onto the left um and just sort of has queries coming in. So we don't have a sort of the kernel itself doesn't have a a whole engine that you can just take and use, right?
It's it's a library. It's not designed to be a standalone thing. Um it's a library that you can integrate. Uh we have done a lot of the work on the on sort of the right hand side of the U to handle paret reading, JSON reading, expression evaluation, all that kind of stuff. That's kind of honestly the much more work that you would have to do if you wanted to integrate at a very deep level. We've done all of that on top of Apache Arrow.
So if you just want to be able to call an API that says like, "Hey, scan a table for me and give me back the data," then that's what you get here, right? You just say like, "Hey, I I want to read this table at this location and it'll give you back Apache Arrow record batches that are the actual table data and you can go off and process that uh either by converting it into your own format or maybe you're already and for example, that's what um Spice AI does, right? like they're already an Apache Arrow shop, so they or a system, so they just do that directly.
Um, and that works great for them. Other systems might want to, you know, read it in their own custom format and then they have to do a little bit more work. I'm going to talk a little bit in a second about how we've made that even easier. Um, okay. Can you tell us the sample uh wherever you have engineer? Yeah, like what can you tell us like know with what we can integrate your delta kernel? Um for example, can I use it with plain JDK or like know JDBC connections or like know maybe yeah you can uh we don't have a JDBC but you can that's like we have a we have a rust sorry I probably should have made this a little bit more clear.
We have a Rust implementation and a Java implementation of this, right? Precisely because most the people who are on the JVM want to stay in the JVM. Most people who are on native code don't want to go into the JVM. Uh and so we have basically two implementations of this. So that if you're on the JDK, you can use the Java version of this and it all works inside Java. Uh if you're some kind of native connector, uh then you can use the Rust version.
So uh it works anywhere that for the most part and on the rust version um we have we we have a C bindings and C++ bindings. So like duct DB is a C++ engine, right? So they ClickHouse is a C++ engine. So they integrate with our what we call our FFI uh layer. Uh, so they can just they just get like a nice C header um that says all of the functions that we export to them and they can call all of those and you know it's it's somewhat complicated because of the kind of back and forth but we think we have a pretty nice model there.
I don't have time to go into like the regory details. Um I'm happy to talk more about this after. I kind of I want to move along a little bit. Um but I appreciate the question and if you have more like let's talk after for sure. Um cool. Uh so what have we been up to? Um, you know, Delta is a complex thing. As I said at the beginning, there's lots of new features always coming out. So, we've been adding lots of different uh support for different things.
Uh, we've added write support for to both of the kernels. Um, which is a big one. So, we can handle appending new data to tables. Um, also called blind append. Uh, we've got CDF support. this change data feed, you know, like kind of subscribe to a stream of what what updates or or not subscribe to a stream, but get a list of the updates that have happened over a particular range of versions. Uh we've added checkpoint v2 support.
Um for people who know a lot about Delta, that might be exciting. If you don't, don't worry about it. Um CRC support is another kind of like low-level Delta feature that can help accelerate uh and maintain data integrity. Very useful. Um we've added domain metadata support. This is a way for connectors or a particular engine to store uh information about a particular domain that's relevant to just their use case of the table.
Um so that can get embedded in the table and they can store and retrieve that kind of information. Um that's helpful. Uh log compaction another sort of accelerate feature. So we take you know delta kind of builds up these small changes deltas hence the name. uh and if there's a lot of them that can can slow down reads especially. So com we have compaction systems that kind of bring that size down a little bit. So we've added support for that.
Um we've added support for replacing tables, schema evolution, more things like this. So one of the big things we've been up to, I guess the point of this slide at a high level is there's a lot of surface area in kernel. We've been doing a lot of uh sorry in Delta and we've doing a lot of work to just kind of add all of that feature support as much as we can into the kernels. there's still a way to go, but um a lot of stuff is is making it in.
Um and you as an engine or connector developer get all of this mostly for free. You know, there might be an API or two that you have to call, but for the most part, uh we do all this work and then you get all these features. So that's nice as an engine developer. Um we've also got a bunch of internal use cases. Um so you know this is something where datab bricks is also investing internally into kernel. Uh and so you know if you are an external connector you don't have to worry about is this something that's going to disappear?
Is it going to go away? Data bricks is going to continue to support this because we're using it a lot internally. So we've got a um row ingestion service. Uh I don't know if that's the final name of it. You might see it under a different name. There were still some debate about the final name, but I think it's going to be called LakeFlow Connect row ingestion. Um, this is a a service we have where you'll be able to sort of send small like rows of data kind of IoT oriented.
I've got lots of little updates coming in. This will accept all of those and build a nice big delta table for you in the background. That's using the kernel um managed iceberg which you'll hear more about um will uh is is using this. Our delta sharing system is using actually both kernels. um they use the Java kernel uh on the on the uh server side and then they use the Rust kernel on the client side. Um so they're an interesting customer because they use both sides of it.
Um we're going to start working on a new Spark DSV2 connector um that is also going to be using kernel and there's more internal uh things that uh I'm not allowed to talk about yet but um that are are using kernel. So data bricks itself is pretty heavily invested in kernel as well. Um, and we want to basically become the narrow waste. Anybody who wants to read or write delta, you go through kernel. That's the way you do it.
New features get implemented in kernel. That's kind of the direction we're moving and we're last year I said the same thing and we're further along that path at this point. It's a great question. Um, that's an it's an early phase thing. I would say we're hoping on the order of a year, but um that's I can't I can't commit to that at this point. I didn't see who asked, but um right there. Yeah. Um we know that people want it.
Um we are we are starting to work on it. There's an open source issue that was recently created about doing this um where we're tracking that progress. So you can find that in the Delta uh GitHub repo. Uh you can ask for timelines and we'll give them as soon as we have them. But honestly, it's it's partially uh just about how much time we get to work on it. Um yeah, as of you know, we've been doing some work to support external integrations.
Um we we meet with lots of different people and um you know, you saw a few minutes ago that slide with all those ones and I just wanted to say thanks to all the people who contribute or or who work with us to to make those um connections, connectors and systems work. They've been super helpful both in having a reason for the kernel to exist and being able to say, "Hey, you know, this this this system can read Delta, which is really cool, uh, and in just giving us feedback on what the APIs should look like."
So, if you have a connection, a connector or an engine that you're building and you're interested in getting involved, I'll throw some stuff at the end for how you can find us. We have a weekly meeting that people can come through and tell us, "Hey, this API sucks," or, "Hey, this API is great," or whatever, right? like um those kind of things. So, thanks everybody who has participated. Um okay, and then I'll probably take about 10 minutes to talk about some things that we've learned and then I'll try to leave about 10 minutes for questions at the end.
Um so, probably the biggest thing that has affected API design for us I would say over the last year is getting out of the engine's way. Um, you know, we had this idea that we would come through in this U shape and uh we'd go back and forth between kernel and the engine and the engines are like that's cool but really you know like I don't want uh this U thing. So, you know, like forget about you. V is the new hotness, right?
Like that's where we want to be. And so what uh what is so remember this was the U thing that we talked about before. So V looks more like this. These might be small uh to read but I'll just kind of go through it then. So the idea is that um the engine or the connector goes to the left first into the kernel say for doing a scan and the kernel using the default engine will uh go do the metadata replay. This is the part of delta which goes and figures out like hey what are the actual data files that you need to read.
Um, and then from that we'll come back uh a list of the files that you need to read. Um, and we have like a an API where you can get that as like a callback or an iterator style thing that just says, hey, read file A, file B, file C. Um, and then an expression that says, hey, after you read that file, you need to apply the following expression. And that will transform the logical the physical data into the logical format uh that this table is actually represented by.
Um and this has been a really successful kind of model because now in the phase three um the connector doesn't need kernel anymore, right? It does its own it's it's on like its own uh path. It can do whatever it wants, which is I'll go read all the data. I can do that exactly however I want. I never need to come back to kernel and I can just apply this transformation expression at the end and then continue processing the query as if it was entirely within my own system.
Um so this has been uh this has necessitated a few changes in some of the APIs. Um one of the big ones was just this idea that um all of the information to transform a physical file into the logical format uh can be communicated in a single expression. Um so far we that has held true. We hope it will continue to if we site find some feature where it doesn't it'll be tricky but we're pretty sure that we can keep going that way.
Um yeah so this has been a big one uh help with performance and also just you know it lets people not have to implement that whole giant engine uh API. I probably should have put a slide with like how all the functions on the engine API because it's it's non-trivial. Um it lets them not implement that. they're like sure you use arrow under the covers and do the metadata replay and then I'll use my format on the other side and those two things don't have to join with each other.
So that works well. Um so that's one big learning. Um and then also as much as possible, this is kind of like in the same vein but slightly different is um you know we should punt to the engine whenever we can like if we have if the kernel this actually just came up today somebody said this um if there's ever something where we can come up with sort of two reason more than one reasonable way of doing something we probably shouldn't pick one of them right we should say hey the engine gets to pick this because there's probably use cases for both um so uh some examples of this the kernel doesn't cache anything, right?
It um we don't we won't cache data that you read. We won't cache uh work that we've done. Um that's up to the engine to cache it. And then the way that we allow you to uh take advantage of that cache is to give you incremental APIs. So if you've done a bunch of work to replay the delta log and uh get a snapshot of the table at a particular point in time and you've cached that snapshot and now you want to kind of go see what are there any more updates is there any sort of incremental new stuff we'll allow you to pass that back in and we'll incrementally update that um and so on for other other kinds of things.
So caching engine concern. There's more than one way to cache things. We're not going to make a decision about it. Um I think I already talked about the second point a little bit. You know, try to minimize engine kernel communication. Uh so you know, we don't we just communicate this final transformation. We don't go back and forth and say like, hey, do this, do that, do the other. Um and the other big one uh which is landing is in about to be done in the Rust kernel and is already done in the in the Java kernel are opaque expressions.
So different engines probably have different ideas about how you parse a time stamp, how you compare the two dates, how you truncate decimals, things like this. So um we're sort of narrowing down the expressions that the kernel needs just for very specific operations uh to support certain delta features and for all the other kinds of uh expressions uh those will become opaque and the kernel will just say oh you know you have a predicate that you want to apply over this table great just tell me uh how to call it back with uh the data and then you can evaluate it and tell me if this predicate it is true or false and then I know what to do with that.
So I can then do all the kind of fancy data skipping file pruning partition pruning stuff that delta does um without needing to know exactly how this expression is evaluated. So this will let us reduce our expression language which is helpful for um engines who want to implement it. They don't have to implement this huge expression language. Um, and also lets individual engines that may have really fancy expressions that their users like to use are still able to use kernel and get all of the goodness that those expressions might bring.
Um, so that's like that's a big one and that we'll be pushing forward in the near future. Um, cool. Java versus Rust. Um, this is an exciting topic. No, not like that. Um uh this is just some some differences that we found in in implementing uh the kernel in Java and in Rust. So in the in the general case, we really do try to keep the APIs aligned between these two. This is easier for us uh as kernel developers mostly for me because I'm like the one who goes between the two of them the most.
Um but also, you know, internally uh for people who want to work on it, we might have to implement features in both kernels if they look very different. Um that's that's that's tricky. But uh I think the interesting thing here is that sometimes uh the data model or or the way that we represent ownership in Java versus Rust really pushes you in a different direction. Um so we have this thing called engine data which uh or in actually in Java it's called columner batch and column vector.
This is the data that comes back when you do a read. Right? This is when when I ask you to read some JSON or read some parquet for me. what does the kernel get back uh to poke at this data? Uh in Java um you can you have this columner or batch which is just a bunch of column vectors and you whatever data you give me just needs to implement this interface and it's a pretty simple one. It says like hey I've got some vectors with names in them and uh you can call things like get string right with a with a row ID.
Um, and all the ownership here is handled by the JVM, right? So like I might have read this data. I give you this column vector, you call get string with some row. You, the kernel can hold on to that. And the engine can throw it away. The engine's like, "Cool. I did the work that you wanted me to do. I've let go of that data." But as long as the kernel is holding on to it, the GC is good and it's like, "Okay, cool. That string needs to stay in memory.
I won't get rid of it." So this is a a pretty nice natural interface in Java. um it doesn't work well in Rust at all. Right? Because in Rust if you say, "Hey, I read some data and then the kernel asks to get some data out of it." Um and that we don't want to force a copy always because we might not need a copy. We might just need to do a comparison or something on it. Um, but if we did want to hold on to it, uh, we can't just hold on to whatever the engine gave us because when, you know, once all this is going on and then the colonel needs it later, the engine may have freed that data and now I have a dangling pointer, right?
Rust won't even let you do that, but across the FFI in general, this is just a mess. Um, so we we actually do this more uh in a sort of visitor model in the Rust kernel, it's a little bit more complicated, but it it lets us kind of express this well. Um, we basically tell the engine, hey, you've given me some data. The kernel knows what data it read. So, it knows the schema and it can just tell the engine, I want the following leaf columns.
And then it's up to the engine to give me back some getters that let me get borrowed values out of that data. So, it might look a little bit something like this. You know, I may have some root node which has a whole bunch of fields in it. That's the dot dot dot on the right. But the kernel is like, I need the following four things, right? Okay. And I may have a strruct even embedded in here, but we can we can flatten this.
You can't see the numbers. It's under my speech uh thing. But you know uh the the r the kernel can index into these getters is like I need getter, you know, three. I know that's a string. Um and then it can say give me row seven there. And that's going to be a borrowed value. And then if the kernel needs to hold on to it, it can just copy it when it needs to. Um so this is a actually very different model than in Java.
And we've had a bunch of conversations about should we unify these somehow and we've really come to the conclusion that no thanks to having a GC uh there's really an argument for these being different across the two across the two kernels and there's a few places like that but on the whole we try to keep it uh similar. Um another thing is that Rust has been pretty great so far I have to say um we've been enjoying building in Rust.
Um I think altogether we've only had one data race issue and it was in the FFI of course because FFI is where you basically give up um uh those protections. Um we returned an iterator to the engine which was uh not protected in any way uh from multiple threads and then uh some engines wanted to access an iterator with multiple threads and things blew up. Um, so you do have to if you're if you're doing sort of FFI unsafe go across the boundary thing, you have to think pretty hard about that stuff.
Um, but internal other than that, we've actually had very very few issues. Um, we've done some pretty large refactors and the compiler is really helpful. Uh, it really kind of lets you do that nicely. Um, the one place where we're still fighting and I think you know most people who spend a lot of time with Rust will agree that like the sync async world is a little bit challenging. I see a few heads nodding out there.
Um, so currently the core kernel is all sync. Uh, because we didn't want to force any opinions about which async runtime you would use. We didn't want to force opinions that you had to understand async. Async is pretty hard to model over the FFI boundary as well. Um, I won't say impossible, but it definitely adds a bunch of complexity. Um so it's very difficult to kind of abstract away this difference between synchronous and asynchronous execution.
Uh what we're starting to realize at this point is that ends up penalizing async engines um which are actually in the Rust world sort of the vast majority of things, right? So um it works but we think that there's probably a space uh that we need to start investigating now which says all right we might need to have kind of like the async version of this that um we're hoping doesn't have to be its whole own thing but we're not entirely sure yet.
So that's a that's a big thing that's oncoming. If that's interesting to you, again, please join in. It's all happening in the open source um where we'd love to have people. Um and the other thing that's surprised us, I would say, is just how resourceful people are. Um some people have some of those integrations that I show that I put on the slide just kind of showed up and we're like, "Hey, we did it." Like, "Oh, cool.
Thank you." Um, and that's that's surprising because um, what I will say is that we need a user guide. Um, I think this is my final oh no, I have one more slide. Um, but I might skip it because of time. But um, I would say if you're getting excited and you want to get involved, that's awesome. We think that the Java and Rust docs are really good at this point. So like at the sort of code level understanding kind of thing, there's pretty good documentation because we've put a decent amount of work into making sure those are good.
Uh but what we're working on right now uh and hopefully we'll have in the nearish future time subject to time constraints is more of a high level picture user guide like hey you're just getting started with kernel what are the things you should know what are the big highle abstractions uh what's the what's the process all that kind of stuff so that is missing right now so uh if you're looking for it and wondering and thinking man that would be great we know we'll get there um we need this internally and externally right like we have internal people who we ask to help out with features and um they're like, "How do I do this?"
And then we have to meet with them and I don't have time to meet with every single person who does that. So um yeah um I think I'll Oh yeah, great. So here's the place to get involved or to integrate. So um there's a Delta users Slack. Um and there's a Delta kernel channel there. Um all of the people who are like regular developers on the kernel uh hangout there will see your messages. Um there's also these are the places where you can uh find the two different kernels.
They're both on the Delta IO organization. Um the Delta Java kernel, that's the top one. Probably should have labeled that is part of the overall Delta project, right? Um, so that has existing open source spark delta and various other things, but it lives under this kernel directory within there. So it's unfortunately a gross URL, but that's what it is. Colonel RS has its own repo. Um, because it's Rust and we didn't want to lump it in with all that Java code.
Um, cool. I think I had a little bit of stuff to say about FFI being tricky. Maybe I'll just shout out that like if you're doing any FFI development in Rust, I would highly suggest MIR. Um, it's very cool if you just write tests that execute your code. It can all be in Rust. You can have these like extern functions, but you test them from within your Rust unit tests. Um, and if you run myriad, it'll tell you that you screwed up your memory management in some way or another.
Uh, and it's it's super helpful. Um, and yeah, I'm not going to bother with the other bits. Things look good. So, yeah, thanks everybody. And I'll take questions for five minutes.

Learn more about the Databricks Data and AI platform.