Delta Lake on the Data Mesh

Summary

• Decentralized data product architecture: This video shows how to build data products using Delta Lake on a data mesh, enabling autonomous, decentralized approaches to sharing data across organizations.
• Cross-ecosystem access: Learn how users can access Delta tables using Spark and outside the Spark ecosystem using tools like delta-rs, Polars, and DuckDB while maintaining data product quality and governance.
• Nextdata OS integration: Discover how Nextdata OS data mesh can integrate with Databricks to enable autonomous data products that interact with Delta tables both inside and outside the lakehouse platform.

Delta Lake on the Data Mesh

Talk by KyJah Keys, Principal Engineer, Nextdata
Delta Lake has proven to be an excellent storage format. Coupled with the Databricks platform, the storage format has shined as a component of a distributed system on the lakehouse. The pairing of Delta and Spark provides an excellent platform, but users often struggle to perform comparable work outside of the Spark ecosystem. Tools such as delta-rs, Polars and DuckDb have brought access to users outside of Spark, but they are only building blocks of a larger system. In this 40-minute talk we will demonstrate how users can use data products on the Nextdata OS data mesh to interact with the Databricks platform to drive Delta Lake workflows. Additionally, we will show how users can build autonomous data products that interact with their Delta tables both inside and outside of the lakehouse platform. Attendees will learn how to integrate the Nextdata OS data mesh with the Databricks platform as both an external and integral component.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and agenda
• 1:50 Understanding data mesh
• 3:32 Data product principles
• 5:09 Nextdata product implementation
• 7:06 Manifests and configurations
• 9:25 Semantic models and validation
• 11:15 Orchestrating data products
• 12:09 Building with asset bundles
• 14:39 Lessons and best practices
• 15:58 Optimizing data product creation
• 18:26 Alternative approaches and Q&A

FAQs

What is a data mesh and why is Delta Lake important for it?

This video defines a data mesh as a decentralized, sociotechnical approach to sharing and managing data in complex environments. Delta Lake's proven storage format, combined with Databricks' platform capabilities, provides the foundation for building data products that can be shared reliably across teams and organizations.

How can users access Delta tables outside of Spark?

The video demonstrates that tools like delta-rs, Polars, and DuckDB enable access to Delta tables outside the Spark ecosystem. This flexibility allows data product consumers to use their preferred tools while benefiting from Delta's ACID guarantees and data governance features.

Is Delta Lake dead?

This video demonstrates that Delta Lake remains a vital, actively developed format widely used in production systems on the Databricks Data and AI platform. Continuous improvements and broad adoption ensure Delta's relevance and future. The video addresses misconceptions about Delta's status in comparison to newer formats.

How can product teams operationalize governed data products?

This video shows how Delta Lake enables treating data as products with clear contracts and versioning on the Databricks Data and AI platform. Governed data products become reusable assets across teams reducing duplicate work. The video emphasizes shifting from data as byproduct to data as strategic asset.

Is Delta Lake proprietary or locked to Databricks?

This video clarifies that Delta Lake is open-source with vendor-neutral governance enabling use across platforms. The Databricks Data and AI platform provides optimized implementations while customers retain flexibility. Delta's openness prevents lock-in while enabling ecosystem innovation.

Full transcript

0:00 Introduction and agenda

All right. Um, I think it's time to get started. So, first, hi. Thank you all for coming out to listen to me speak. Uh, seems like there was a lot of excitement and a lot of hype behind this. And let me apologize in advance because you're probably going to be disappointed because I'm an engineer and this talk is mainly for engineers. So, you're going to see a lot of screenshots from my IDE, but hopefully it will be um informative.
Uh with that said. I'm a principal engineer at Next Data. Um and I'm here to talk to you about using Delta Lake on the uh data mesh. So, it's just a forward-looking statement. Data Bricks told me I had to put it in here under the impression that absolutely nothing I say to you will be true in this talk. But, I mean, that's kind of an exaggeration. It will all be true, but it will probably change by the time you get your hands on any of these things.
So, by show of hands, how many people are familiar with the concept of the data mesh? Just excellent. There's be a lot less to talk about. So, in terms of what we're going to go through today, it's just going to be a quick I'm going to quickly cover, you know, what is the data mesh? um get into how Next Data implements data products and also provide an example of attempting to build some data products using um data bricks asset bundles and then maybe some other uh ways in which you can interact with Delta Lake while building using our autonomous data products.
I was going to have you save your questions until the end, but I think it'll be more productive if um you go ahead and ask questions as they come and then we'll try to pause and get to those. So, let's see uh let's see where this takes us. So the first thing is what is the data mesh? Uh so the founder of our company Jamaat Dagani she uh wrote the book called the data mesh and this is just a quote from the book itself that says data mesh is a decentralized sociote techchnical approach to share access and manage analytical data and complex and large scale environments within or across organizations.

1:50 Understanding data mesh

And those are a lot of words and they're pretty cool words and that's why I included the quote. But it does make it kind of hard to say that because any sentence is going to be an overly reductive description of data mesh. But the reality is data mesh is the data products, the communication between those data products, the policies and the governance that goes and goes on top of that. And that's really all it is in a nutshell.
But there are some principles that come into uh a data mesh. One of those is domain ownership. That means we want our people that are building data products to actually be able to build the data products without the impediments that you know generally come into that process like getting permission from this group or that group. Um we want our data platform to be self-s served. We want people to be able to iterate quickly.
We want them to be able to define their data products and build them. However, we still need some sort of governance and that's why federated computational governance is one of the principles involved in the data mesh because we we simply can't live in chaos. Um, and then the last one is data as a product because ultimately what is the point of a data mesh without its primary feature the data products that we want to share and data is the primary goal of this.
So that multimately ultimately means it has to be one of the principles of building a data mesh. So what is a data product? Similarly, it comes with a set of principles. It needs to be discoverable. It's absolutely useless if you can't find the data product. It's a common occurrence in large organizations. You know that there's data somewhere that you want to want to use, you need to use, but you can't find it. Um we want it to be addressable.

3:32 Data product principles

Obviously, once we found it, we don't want it to disappear. We want it to have some sort of long lived address at which we can find it. needs to be understandable. Nonsense helps no one. It's useless. So data products need to describe what they are, their requirements, their constraints, um what data they contain. They need to be trustworthy and truthful. That means when they execute, the data that they produce should be the data they say they're going to produce uh natively accessible.
So that means you don't need a proprietary new technology stack to access the data when you're using a data product. In an ideal world, you can continue to use the same tools that you've always used to access your uh your data products. Then we want them to be interoperable and composable. That's kind of useless if they're just independent data sets existing in the world. We want to put them together. We make new aggregates from different data sets.
Uh and we want it to be valuable on its own. We don't want to have to use random tools and plugins and other things to be able to derive value from our data sets. And obviously it needs to be secure. Most of us in enterprise scenarios will find that if it's not secure, we can't use it. So how's the next data implement data products? So the next data our solution the next data OS the solution that exists uh currently deployed on top of kubernetes and it's main responsibility is to deploy these aut autonomous data product containers.

5:09 Nextdata product implementation

So how are they discoverable once they're deployed they're actually registered with our data mesh and they show up in the UI. Uh, if you've seen any of our demos, there's that pretty constellation of data products, but additionally, you're able to search for them, and there are agents that will also help you discover them through their semantic links. They're addressable because they all have well-defined URLs on the internet.
They will always exist at whatever your data domain is slash that data product. Um, they're secure. We use TLS um communication and uh OOTH for access. Uh I'm not a security specialist, so I'm not going to go much deeper than that. However, uh they're trustworthy and truthful because they have both promises and expectations. Promises are declarations of the data having been output is the data you expect it to be. And expectations are actually the opposite.
Data coming in is verified against a set of expectations before your transformation runs. Also, our data products are natively accessible. All the transformations are just Python transformations or a SQL transformation. So, they're the tool sets with which you're currently familiar. Uh you don't have to learn anything new um other than how to define a data product in order to use them. And they are interoperable and composable.
Data products can use data products as as inputs. Um, and you can also see that relationship within the data mesh. So, I know I said it was valuable on its own and understandable and this screenshot is insane, but we'll break it down. Uh, because really this is all you need to make to um, declare a data product. You're going to specify the compute services that you use, the storage services that you use, the authorization rules, models that it expects to use from its upstream sources and the models that are performed along with the actual code for the transformation and any custom promises and expectations.

7:06 Manifests and configurations

So the next couple slides will be actual details on these things. So when we look here, you see that this is the compute section from our um from our previous slide. And what this is saying is we want to use data bricks compute to execute our transform. So the YAML below you see is actually the configuration that an infrastructure specialist or the data engineers themselves would uh would put into the data mesh in order to use this configuration.
Uh question. Yeah. You said you're allowed to, right? Yeah. What about the managed interfaces? I can't hear you. Your interfaces. The interface. Yeah. How do you manage those? So what is the rule about that? How do I have to expose the products? How do we expose the data products? Yeah. Uh so the data products are actually exposed on um the URLs where I said they're addressable or was there more I can answer for you?
Probably a little bit more. So, so you can have to me a lot of data products. You can have them in APIs, you can have them in SQL, you can have. So, what is the rules about how the interfaces are? So, the data products interface is if we go back to here and we look at this slide, the data product's primary interface is either its input or its output as you define it in the data product model. as a user if you wanted actual access to the data product outside of that there is a mechanism for that's done through the data mesh UI but that's out of scope for at least what I was going to talk about today which is just the process of building the data products okay I I was more thinking about the schemas so so to make sure that the interface are not changing to make that very stable okay yeah so stick with me as I move through here we should actually get to that um so as I said this was the declaration of compute services that your data product would use.
Uh moving to the next set is the storage services that your data product would use. Uh it's another thing that would be predefined before you actually get into the process of building data products. Now our autonomous data products have a set of life cycle hooks associated with them. One of those is provisioning. One is executing the transformation and then the other two are executing what we call the promises and expectations.

9:25 Semantic models and validation

So for someone to implement a storage service, the provisioning step is ultimately going to be optional, but it is useful in places where you're implementing a data product that has a schema. So I think this might be what you were asking about and this is how we define a model that we would use um or that a data product would expose. This could be this model could be used for either consumption or production. So if we go through this, you'll see that we've given it a name, a description, we've specified a schema, and we've added attributes.
And then additionally, we've added some semantic links between a different uh additional attributes. And then lastly, we have um some verifications on that'll either be executed as promises or expectations depending on how this model ultimately is used. The type system that we use when defining a semantic model is effectively just Aeros type system. So we found that's the most compatible set of types to use across all um storage services that we've encountered.
Uh just to check is that what you were looking for. Uh so expectations um and promises those are the additional things that you would define when you've defined your data product spec in this case we'd say I expect the model POS sellout when it does not match its type to cause the data product to fail and the other case here is when I've produced the model products if there was an error in my transform I would expect my data product to fail now being in a failed state means means that data products that consume from that original data product will stop.

11:15 Orchestrating data products

So this helps you avoid a cascading chain of failures when an upstream data product makes a change and you don't want that to trickle out into your further aggregates. So the fun part. So I was trying to build some data products uh just using data bricks access bundles and my scenario was it's very contrived uh the domains alpha beta and gamma. There's just some tables with some things in them and then a data product that consumes those two as aggregates.

12:09 Building with asset bundles

So this was my setup. I used data bricks asset bundles. If you have not used uh the data bricks VS code plugin, I recommend you do. It's pretty great. Um but there were a few tools that I used to make this a little simper simpler. There is a Python package called data contract CLI that allowed me to implement my promises. And um if we're going through this, what you'll see is my transform is in the main py and the data contract test notebook is actually responsible for executing my products.
So what I ultimately had to do um there we go. So ultimately what I could do, can you see that? Is that contrast working? Okay. Yeah. So this is if you're using the data contract CLI package, you can define your model basically in this YAML file with the header, model, and promises. And it turns out it's it's pretty useful for getting um the equivalent of a semantic model into a data asset bundle rather easily. So this was my transform.
It's pretty simple. generated some random data, inserted into a table, called it a day. This was my notebook that ran my promise after I did that. The most noteworthy step is the last uh or that step in the middle where it says run data contract test. that will take that YAML file, evaluate it against my delta table, tell me whether it's good or not. And then you can see my job. Um, didn't realize I made such a bad screenshot until I did my final review, but we were committed at that point.
So, that's where we are. Um anyway, there's a main task that runs my main transform and then there's a notebook task that runs my um that runs my promises. And this is the closest analog I could get to building an autonomous data product. So what I ultimately learned from this is authentication and authorization provided by Unity catalog are excellent tools. Having the manage compute is also a really great tool. Having the flexibility to use notebooks, wheels, and scripts was also a great development experience.

14:39 Lessons and best practices

The one problem I really ran into was I was very easily able to build the alpha and the beta data product because I could use cron jobs to schedule those. Triggering the gamma data product to run when both data products were ready involved a level of technical complexity I did not feel like implementing. So I ultimately didn't do that. But one of the nice things about using the asset bundles is if you can run it on Spark, you can run it.
Um, and then Unity catalog search, you know, suffices as a good discovery mechanism in this place. But this wouldn't be a fun talk if I didn't tell you how we could do it in a much cooler fashion. So if we're looking at this layout in my repository, I I broke it up into, you know, a model, a directory with my transform and a directory with my models and a directory with my spec, but it wasn't necessary, but it made it more readable.
So this was my data product declaration for the um the original data product, the one the transform I previously showed you. So if we're looking at this, you know, I went ahead and created a transform that generates code. I scheduled it using this expression, you know, with an hourly interval. I said I wanted to run it on data bricks. I said that my output was data bricks and I gave it access controls. And then my transform actually it really only changed by two lines of code for the most part.

15:58 Optimizing data product creation

If you look at line 14, you'll see that I added an annotation to specify that's the transformation function, which is a trigger for our autonomous data products to know which Python function to run when it's time to transform. And then in my write to table method, I got the table name from that output parameter that you can see on line 16 because our data products have knowledge of their underlying storage. so they can tell you those introspective details of how to use it.
It's also a useful facade if you're implementing something like a shallow clone, I'm sorry, like a promotional workflow. And then this was my model. Then my table only had three columns. It wasn't particularly interesting, but this was the declaration. So what was different? If you looked at my previous code, I actually provisioned the tables manually. That was a thing I had to do. Um, but using the next data autonomous data products.
Provisioning is more or less a thing you get for free. In most cases, it's just part of the driver itself. Uh, promises and expectations on whether your data is wellformed. Those are going to be implicit. You don't ever have to write any particular transformation for that. you don't have to say that's a thing that I want that is a thing that you get. Um and then the fun part I was able to orchestrate the behavior between the uh and the gamma transformation between the alpha and beta data products by using this when statement to say when alpha and beta are done I want to run and then our transform runs.

18:26 Alternative approaches and Q&A

So, so there are some other ways in which we can actually still continue to build data products. So, we don't have to use Spark compute. We can use local Python compute if we change the compute service that we specify um as our target. And in that case, the only change your transform would actually just be the removal of the Spark parameter. Another thing that we've seen a lot of people use is actually just using our object store drivers uh and then using off-the-shelf Delta Lake tools like uh Delta Lake Python or um Polers and then there's a driver that we have in the works based on Delta kernel RS to actually manage Delta Lake natively within a storage driver.
Uh we're currently working on implementing a proper promotion workflow. It's a thing that we've done with Snowflake where we use a shallow clone to um promote a table and we're trying to get the same thing to work in the kernel. Um but there are a couple uh uh pull requests we need to make to make that possible. And then really I'm open for questions at this point.

Learn more about the Databricks Data and AI platform.