Native Streaming-to-Lakehouse Governance with Unity Catalog

Summary

This video introduces a native integration between StreamNative's Kafka and Pulsar platform and Databricks Unity Catalog that uses catalog-based commits to write streaming topics directly as governed, managed Delta tables — eliminating the governance gaps that came with writing external Delta tables to cloud storage paths.
Catalog-based commits register and govern streaming data at commit time, enabling automatic access control, lineage tracking, query-readiness, and downstream auto-optimization without post-processing or manual metadata synchronization.
The integration delivers five key benefits: unified governance, transactional consistency, automatic storage optimization, improved discoverability, and a seamless path from event streams to AI-ready lakehouse tables — all within Unity Catalog.

Native Streaming-to-Lakehouse Governance with Unity Catalog

Real-time data powers modern AI and analytics yet governing high-velocity streaming data in the lakehouse remains a challenge. In this video, StreamNative and Databricks showcase a native integration that brings governance to streaming pipelines through Databricks Unity Catalog without sacrificing performance or simplicity. We introduce catalog-based commits, a new approach where streaming data written in Delta Lake and Apache Iceberg formats is registered and governed in Unity Catalog at commit time. This eliminates post-processing, manual metadata synchronization and governance gaps, while enabling consistent access control, lineage and auditing across streaming and batch workloads. Attendees will learn how this integration reduces operational complexity, accelerates AI and real-time analytics and delivers a seamless path from event streams to governed, AI-ready lakehouse tables.
Talk By: Benjamin Mathew, Product Manager, Databricks ; KUNDAN VYAS, Director,Product & Partnerships, StreamNative ;

Chapters

FAQs

What governance gap does this StreamNative and Databricks integration solve?

Previously, streaming platforms could write Kafka or Pulsar topics to cloud storage as external Delta tables, but this data was invisible inside Unity Catalog — it had no governance, lineage tracking, or access controls visible to the broader organization. The catalog-based commit integration resolves this by registering each streaming write directly in Unity Catalog at commit time.

What is a catalog-based commit and how does it work for streaming?

A catalog-based commit is an approach where streaming data written in Delta Lake format is registered and governed in Unity Catalog at the moment of commit, rather than being written to a storage path and later synchronized. This makes the data immediately query-ready and governance-compliant without any post-processing step or manual metadata synchronization.

What benefits does catalog-based commit integration provide?

The integration delivers five key benefits: unified governance with consistent access control, transactional consistency through ACID guarantees for streaming writes, automatic storage optimization through Unity Catalog's auto-compaction, improved data discoverability within the catalog, and a seamless path to AI-ready data for downstream ML and analytics workloads.

How does StreamNative's platform connect to Unity Catalog for this integration?

StreamNative's Ursa platform — which supports both Kafka and Apache Pulsar — writes topics as managed Delta tables in Unity Catalog using catalog-based commits. The video includes a demo showing the creation of a StreamNative cluster with Unity Catalog integration configured, demonstrating the complete setup from the StreamNative platform side.

Full transcript

[00:08] All right, perfect. Uh welcome everyone to our today's session. We are excited to talk to you about how to stream data at scale to a lakehouse and govern it using Unity Catalog. So, a quick forward-looking statement.
[00:21] Uh essentially, today's session is purely for your education purpose only. Please do not make any purchase decisions based on what we show you today because our uh roadmap is subject to change. So, uh yeah, just purely for
[00:33] education purpose only. Please take this survey. It helps with the feedback of the event. So, the quick motivation for the session, the core of what we're going to cover today is purely about uh
[00:45] you know, the challenges which most enterprises face in in terms of building this end-to-end pipeline from streaming platforms to like, you know, lakehouses, and how do they actually stream it at scale and govern it at scale?
[00:57] Uh so, that's really one of the biggest challenges which we're going to talk about and the solution for it. Now, historically speaking, this has been like, you know, done in many ways, and one of the ways was like, you know, to write topics as external tables,
[01:12] external Delta tables, and then which worked to a certain extent, but it had its own challenges because the data was only as far as, like, you know, the storage bucket was. It was not visible externally internally within a, you
[01:24] know, Unity Catalog or within a governance. So, the the broader visibility to most organizations was not there. It was not easy to discover the data which was written by the streaming systems, right? Um so, that was one of
[01:36] the biggest things, and then, of course, the other challenges with most enterprises face is where they actually have to write custom pipelines and things like that. So, one of the core solutions we're going to talk about Ben and I is purely about leveraging
[01:48] catalog-based commits to write topics as managed Delta tables in Unity Catalog, right? So, moving really fast. So, quick introduction. I'm Kundan Vyas, a lead product and partnerships at uh StreamNative, and then you can talk to
[02:01] me about streaming, Kafka, Pulsar, and of course building out fast derby cars. That's what I do with my son, 12-year-old. Ben, you want to go next? >> All right. Hello, everyone. Uh my name is Ben. I'm a product manager here at Databricks. I work on open table formats
[02:14] and interoperability across catalogs and engines. And if you want recommendations on the best Thai food in San Francisco, I will definitely talk your ear off. Um over the next 20 minutes, we'll be covering three things. First, I'll give an introduction into what catalogs
[02:27] commits are. Then I'll hand over to Kundan who will talk through how StreamNative is leveraging catalog commits. And then finally, we'll wrap up by talking about how you can get started with catalog commits today.
[02:40] Awesome. So, to start, let's dig into the main problems that we're aiming to solve with catalog commits. And in order to do that, we need to give some background on open table formats and open catalogs. Delta was introduced as an open table format to do two key things.
[02:54] First, it brought acid or acid transactions to data lakes. And second, it allowed enterprises to interact with their large amounts of data as if they were just simple SQL tables. And one of the key design decisions that
[03:06] Delta made early on was to leverage the file system as the commit coordinator. Meaning that if any engine wanted to interact with a Delta table, it would leverage these file system APIs to perform that access.
[03:18] And this is a very key decision that we'll come back to, but fundamentally, Delta laid the foundation for the lakehouse architecture. As organizations started to standardize on this lakehouse architecture, their
[03:30] needs grew beyond just tables. Organizations started to build different kinds of assets on their lakehouses, including files, functions, ML models. And they needed a central place to govern and organize all the access to
[03:42] these assets. And that's exactly why we built an open source Unity Catalog. Unity Catalog is this multimodal catalog that It enterprises to govern, discover, and audit all these different kinds of
[03:54] assets. And Delta and UniCatalog, they they worked well together, but over time enterprises started to face two key coordination issues. And these
[04:06] coordination issues were fundamentally because Delta was a file system oriented table format rather than being a catalog oriented format. So, we'll get into what those two key challenges are.
[04:18] The first is what we call the multi-engine, multi-agent sprawl problem. You see, organizations want to use different kinds of engines for their different workloads. And they expect to be able to define governance policies once and point all their engines at
[04:32] those policies to reference upon data access. However, there hasn't been a standard by which Delta clients can access tables through catalogs. And as a result, engines like StreamNative have had to
[04:45] access these UC managed Delta tables directly by path. And this static path-based access that completely sidesteps the governing catalog, this can lead to three key problems. The first, of course, is these accesses
[04:58] are not governed in the ways that you defined in your catalog. And, you know, this is especially risky for sensitive data. The second challenge is that the catalog is not aware of the different kinds of
[05:10] accesses that these engines are trying to make. And this, from an auditability perspective and from a compliance perspective, this is a nightmare. And then finally, the catalog is not aware of the kinds of
[05:22] operations that the engines want to use. And specifically, when we talk about schema changes, uh you can run into a scenario where an engine, for example, tries to drop a column. Um and that column is actually key to relationships between tables, and
[05:35] all of a sudden these relationships between tables can break. You can have silent metadata drift, and this can lead to downstream correctness uh downstream correctness issues. The second key challenge that organizations were facing relate to
[05:48] transactions. You know, as organizations wanted to standardize on the lakehouse architecture, they wanted to bring all different kinds of workloads to the lakehouse, including data warehousing workloads. And however, one of the key paradigms for data
[06:01] warehousing is this concept of multi-table transactions. And since Delta has been a a file system oriented table format, these transactions were only scoped to single tables. And as a result, if you wanted to perform these multi-table updates
[06:15] on Delta tables, some of these updates might fail, and this can lead to an a scenario where your downstream assets like dashboards, which rely on multiple tables, could be in an inconsistent, unreliable state. And, you know, this
[06:28] has fundamentally prevented organizations from bringing some of these multi-table transaction data warehousing workloads to the lakehouse. If we pull back and consider how Iceberg addresses these issues, it does so by
[06:41] design because Iceberg is inherently a catalog oriented table format. All accesses go through the catalog, and the catalog facilitates um all reads and writes. So, in an effort to address these core
[06:54] challenges that organizations were facing, and as part of this broader effort to simplify lakehouse storage and co-evolve these formats together, we introduced and we worked with the community closely to introduce catalog
[07:06] commits. Catalog commits is this open standard by which Delta Delta tables can leverage catalogs as their commit coordinators. So, the the catalog is responsible for facilitating all reads, writes, table
[07:20] discovery, and table authorization. And Unity Catalog managed tables offer a reference implementation for what it means for a table a Delta table to leverage a catalog as its commit coordinator.
[07:36] So, let's get into like how catalog commits actually work. You know, before catalog commits, Delta clients would write directly to the transaction log. Once the catalog commits Delta table features enabled, now Delta clients will
[07:49] stage rights to a directory within the transaction log, and it will also propose these rights to the catalog, specifically to a catalog commit API. And once the catalog ratifies these
[08:02] commits, it will flush these commits to the transaction log to then be persisted in that transaction log. And then upon read,
[08:14] Delta clients will look at both the transaction log for previously flushed ratified commits, and to the catalog for most recently ratified commits to then together resolve the latest state of the Delta table.
[08:27] By putting the catalog directly on all read and write paths, we lay the foundation for organizations to address these key coordination challenges that we had described before. And now, catalog commits is GA on
[08:39] Databricks, and we're super excited about all the major benefits that it brings to UC managed tables. First, it allows external engines to write to UC managed Delta tables in a fully governed manner.
[08:53] Second, it Through catalog commits, we are enabling consistent governance, attribute-based access control, and auditability across all of your engines.
[09:05] Third, catalog commits helps unlock multi-statement, multi-table transactions on the lakehouse, allowing enterprises to truly bring all these traditional data warehousing workloads to the lakehouse.
[09:18] Fourth, since the catalog now has observability into all engines accesses, it can better optimize the data layout of your tables for your workloads.
[09:30] And then finally, catalog commits standardizes the ways that tables are discovered and authorized across Delta and Iceberg, contributing to this broader unification effort and co- effort across these table formats. We're super excited about, you know,
[09:43] catalog commits being the next evolution of Delta and being this massive table upgrade for UC managed tables. So, with that, I'll hand it over to Kundan who will talk through how StreamNative is leveraging catalog
[09:55] commits. >> Awesome. Thanks a lot, Ben. >>
[10:07] managed Delta tables in Unity Catalog, right? So, quick introduction to StreamNative. We are a company created by the founders of Apache Pulsar, streaming message queuing technology. We also provide a native Kafka service in
[10:20] our cloud. So, we have a platform, single platform which is driving two native streaming services, which is powered by Ursa, which is our basically natively integrated storage with lake houses. So,
[10:34] this is how we stream data to different types of lake houses. You can choose your streaming technology, you can choose your storage, whether it's persistent storage or diskless storage based on the use case you want to drive.
[10:46] If it is basically a use case for ultra-low latency, you can use a disk-based persistent storage or if it is a you're okay with some relaxed latencies, then you can use a diskless storage to write topics directly as
[10:59] Delta tables to object storage, which will then be published into Unity Catalog. So, I'm going to talk more about that. So, that's a little bit about StreamNative. We are known for Apache Pulsar because obviously it's a company created by the founders of Pulsar, but we also launched
[11:12] a native Kafka service recently, which is powered by Ursa again. It's available on all three different clouds. Now, of course, there are all capabilities of Kafka for the people familiar with Kafka, connect Kafka brokers, Kafka connectivity, schema registry, but the
[11:25] unique thing here is that you have the choice of how you can, you know, create your cluster purely based on your latency requirements. If you are ultra low latency requirements then you can which is anyways a common requirement for Kafka you can go with late
[11:38] latency optimized clusters for persistent storage if you think that some of your use case are for analytics are okay with a little bit of latency then you can save all the cost of your persistent storage by skipping it
[11:50] completely writing it directly to the object storage so those are called cost optimized clusters. We also support Kafka queues so now we have essentially a new Kafka service in in the cloud so you can choose either of them. So I want
[12:03] to talk a little bit about Ursa which is basically how we stream data to different we are the storage to a different lake houses Ursa is definitely the work which we've done in the past where we realized that all the companies want to write
[12:15] data in open formats now so we definitely and our customers were waking up to that. So we really wanted a generic way to kind of stream all your data to your favorite you know lake
[12:27] house destination. So in this case Ursa is basically providing that choice whether you choose Kafka or Pulsar you can stream that all your topics to your favorite open table formats whether it's Delta or Iceberg based on use case and again
[12:40] storage wise you can choose either persistent storage or object storage. So I want to give you a quick sneak peek of the work we've been doing with you know Unity Cloud Catalog closely because we have a lot of customers who
[12:53] constantly ask for what are some of the new things coming up in in lake house space right so we started a long time back with support for generic Iceberg rest catalog which made it easy for us to support so many different catalogs but as you can see very early in 2025 in
[13:06] Jan we supported Delta tables already which means that you had the ability to write topics as Delta tables but here we had a bit of a challenge like you know this worked well but I think still had some challenges of where customers had
[13:18] to do their own let's say custom jobs doing table maintenance and all the other things which you know typically are not done when a table is not fully managed right? So, it worked well, but it had some challenges.
[13:30] We supported managed Iceberg tables in October 2025, which worked really well, going well, and we are constantly adding more capabilities. But today, here Ben and I here to talk about this particular solution for Delta tables, which is
[13:44] based on catalog commits, which Ben spoke about, right? So, now as an ISV, I'm talking about how we were basically leveraging catalog is commits to create a fully managed Delta table in Unity Catalog. Because that comes with all the benefits which Ben spoke about, right?
[13:57] So, that's a little bit on sneak peek on how we can evolve. Just a quick view on different other features we are adding over time. Iceberg V3 is big. So, like for streaming, whatever makes sense, we always keep an eye and, you know,
[14:11] we add that support, like variant support, data type support, schema evolution, row-based lineage. All these things are relevant for streaming. And also, there is variant type support in Delta Lake. So, all that is already
[14:23] there. Now, bring your own lineage integration is coming, you know, soon. Hopefully sooner than October. And we'll keep an eye on Iceberg V4. We don't you know, anything which is going to be relevant for our customers, for streaming, and making their lives
[14:36] easier, we'll definitely, you know, implement that. So, we'll keep you posted on that. So, now I want to quickly, in the interest of time, talk about really how we natively integrate StreamNative, which provides fully managed services for
[14:49] Kafka and Pulsar with Unity Catalog. And how do we stream all these topics as fully managed Delta tables, right? So, essentially, within StreamNative, we have Imagine this experience, you log in, you create a cluster. Now, instead
[15:02] of streaming all your data via a connector or a custom pipeline, you can just choose a native integration, which allows you to write all your topics to Unity Catalog, right? And what happens behind the scenes essentially is we are
[15:15] leveraging catalog-based commits. There's a Delta kernel SDK, which allows us to easily understand like a few important steps. Like if you want to write topic as table, you check whether the table exists or not. You know, what
[15:27] are the storage location of the table? All these things we had to kind of do a little more extra work before, but now the API provides us all these details, right? So, there's a lot of win-win for of course ISVs to easily implement it.
[15:39] We have less number of SDKs to implement it. But, the real win is here for customers who will actually end up getting a simple native integration where the data is landing as fully managed Delta tables. And of course, all the benefits of what Databricks is
[15:51] working on to make things life like you know, things better for people who are using managed Delta tables, right? So, um just one last comment here on this integration. Uh what really happens behind the scenes is
[16:04] whenever a topic is created and the data starts coming, essentially what we do is very quickly write that to a storage location. And essentially, because it's a streaming service, so we you can expect a lot of small files getting
[16:17] created initially. But then, we run our compaction service, which then converts all those files small files into larger files and parquet files. And we go through the whole life cycle of creating a Delta table, registering it, and or
[16:29] updating it. Well, the process is very similar for Iceberg. I'm not even talking too much about Iceberg, but that's a choice you make whether you want to write Delta or Iceberg. The experience is going to be very similar. So, that's what I like about the beauty
[16:41] of Unity Catalog and the vision it has. It allows you to sort of uh take a very similar consistent approach of how you stream data, right? So, I'm going to move on just in interest of time. Just a couple of slides so you get a sneak peek
[16:53] of what um this product looks like. Within StreamAnalytix Cloud, you can create a cluster. Sorry, the screenshots are a little not as clear, but essentially what this is showing is create your cluster, give it a name, and
[17:05] more importantly, choose your region. But, this is this is the important part. Choose your cluster profile. If you want, you know, persistent storage for ultra-low latency, you can choose latency-optimized profile. Or if you're okay to have some relaxed latency for
[17:19] analytics, you can skip that storage infrastructure, save that cost, and choose cost optimized, right? And then this is the most important part. Within the cluster, now you don't need connectors, you don't need all the custom pipelines. All you're doing is
[17:31] selecting your catalog, right? Which you've already registered, and you can say, I have already registered authenticated a card catalog, and I want you the this cluster to write data to that, right? So, this is what is
[17:43] the choice you're making. And once the data lands, the topics land in, you know, Unity Catalog, they'll land as tables. And I'm highlighting one part here, which is really important. This is landing as managed Delta table, right?
[17:57] So, that's a little bit of the flow on a very high level. You can just create your cluster, and, you know, wire up the native integration, which will stream all this data as managed Delta tables. So, I know it's a short short session,
[18:10] so we want to quickly summarize it here, and then give you some access to link which you can use. You can sign up for a trial of Stream Native Cloud if you want to get your hands on Pulsar or Kafka cluster, which can write natively to,
[18:22] you know, Databricks Unity Catalog. That's a blog Ben and I wrote together, so you can scan it and learn more about how it works behind the scenes, how catalog commits work behind the scenes. And Ben, you want to
[18:34] >> Yeah, and then on the Databricks side, so it's very easy to sign up for Databricks. You can do this You can do that right now. And then more importantly, once you're on Databricks, you can actually enable catalog commits by running a simple alter table statement. By running the statement now,
[18:48] this your this managed Delta table is leveraging Unity Catalog truly as its commit coordinator. And then finally, I highly encourage you to start using Stream Native with these catalog commit enabled tables.
[19:02] And you know, we'd love to hear your feedback. >> Awesome. We wrapped up actually 1 minute sooner. You want to talk about any of these sessions which they can attend. You can your >> Yeah, tomorrow we have a session what's
[19:15] new with open table formats and this kind of gives an overview of how Delta Iceberg are evolving together and what's coming next. I would highly encourage you guys to attend that. That's I believe around 4:10 p.m.
[19:28] So you can see that on on the schedule. And then anything else on the StreamNative side you wanted to >> Yeah, absolutely. Like if you guys have any other questions with us about how we have kind of worked together and what are some of the more curious cases you which are relevant here, we can hang
[19:40] right at the back because we have limited time and we also have a StreamNative booth the exhibition hall if you want to stop by there and then you know, catch up. So we'll we'll hang right there for questions if you have any.
[19:52] >> Awesome. Thank you folks. >> Cool. Thank you so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.