Open Streaming Meets Governed Lakehouse Running Apache Flink With Unity Catalog–Managed Delta Tables

Summary

This video introduces a new Flink Delta Connector built on Delta Kernel that enables Apache Flink workloads to write directly to Unity Catalog managed Delta tables, delivering exactly-once delivery guarantees, schema evolution support, and catalog-managed commits.
The existing path-only approach — where Flink wrote Parquet and metadata files directly to cloud storage — created governance gaps because data was invisible to Unity Catalog's lineage, access controls, and auto-optimization features; catalog-managed commits resolve this by registering every write at commit time.
The connector supports both the Delta Sink API and Flink SQL patterns, with a roadmap that includes upsert support, full schema evolution, and a Flink source connector for reading Unity Catalog managed tables.

Open Streaming Meets Governed Lakehouse Running Apache Flink With Unity Catalog–Managed Delta Tables

As organizations adopt open streaming engines like Apache Flink alongside centralized governance platforms, interoperability becomes critical. This talk explores how the Flink Delta Connector bridges Flink’s real-time processing model with Delta Lake tables managed by Unity Catalog. We’ll explain how the connector preserves Delta’s ACID guarantees, integrates with UC-managed metadata, and respects centralized governance when writes originate from external Flink clusters. Through architectural deep dives and practical examples, we show how teams can build end-to-end streaming pipelines that combine Flink’s scalability with Databricks’ unified governance and lakehouse architecture.
Talk By: Hao Jiang, Software Engineer, Databricks, Inc. ;

Chapters

FAQs

Why do teams want to use Apache Flink together with Unity Catalog?

Flink is preferred for real-time data ingestion because of its stateful processing, low latency, and rich ecosystem for IoT, clickstream, and ML feature pipelines. Unity Catalog provides table governance, lineage, access control, and auto-optimization — and this video shows how teams can use both together rather than choosing one over the other.

Why wasn't writing Flink data directly to cloud storage paths sufficient?

Writing Parquet files and Delta metadata directly to a cloud storage path creates a table that is not registered in Unity Catalog, making it invisible to governance, lineage tracking, and downstream query optimization. Catalog-managed commits ensure the table is query-ready and fully governed from the moment it is written, unlocking all the benefits Unity Catalog provides.

How does the new Flink Delta Connector ensure exactly-once delivery?

The connector coordinates Flink checkpoints with Unity Catalog catalog commits, so a write is only finalized in the catalog when the corresponding Flink checkpoint successfully completes. This two-phase coordination prevents partial writes from appearing as committed data in the table, giving the same ACID guarantees that batch Delta writes provide.

What is the current status and roadmap for the Flink Delta Connector?

The connector currently supports the Delta Sink API and Flink SQL write patterns, with exactly-once delivery and memory control already available. The roadmap includes upsert (merge) support, full schema evolution, and a Flink source connector that would allow reading Unity Catalog managed tables directly from Flink workloads.

Full transcript

[00:09] Hi. I'm My name is Hao. I'm a software engineer from Databricks working on the open lakehouse team. Uh today I would like to share some of my works on making open stream engines the first class citizen in the governed
[00:21] lakehouse world. By the end of this talk, I would like to say you know how to stream Flink data into the Unity Catalog managed lakehouse with reliability,
[00:34] efficiency, and your confidence. So, let's uh picture a team streaming click stream or change data from Kafka and get into Flink.
[00:48] They want the Flink cuz it's stateful, it has a low latency, and it has a good ecosystem. It's ideal for real-time ingestions. And now this same team moved to Databricks,
[01:01] and they live in a governed lakehouse world. They want governed data, they often want Flink. So, this talk is about how you don't need to make the choice between the two. You can enjoy both.
[01:14] We got the same feedback from our real-world customers. Can't name them, but we have several very important customers with a lot of data. They think Flink is a must, and they
[01:27] came to us asking, "How do I write to uh your Unity Catalog with a managed table?" That's what we want to do for them and for the community.
[01:42] So, here is the thing. If you write only Parquet files and metadata files into a cloud storage, you can call it a a bare working table, but it's far away from a full-fledged
[01:54] the table. If we call it something a table, you at least want a trusted table with a well-maintained catalog with observabilities, lineage, trace, and downstream
[02:07] maintenance. Or better, sometimes you would get automatic compaction and auto organization. If you just write the let's say iceberg or Delta into a cloud
[02:20] storage, you will never get those benefit. What we want is a query ready the moment the table was written.
[02:32] So, here's a gap. From one side, Flink has open source state for low latency, used by IoT, used by ClickHouse, many real-time ML features, a lot of usage. People really
[02:46] want that. On the other side, we have Unity Catalog. We have table governance. We have lineage. We have downstream ready queries. Most important, we have catalog managed
[02:59] commits. So, all these benefits, they come now from two separated world. They both are great, but getting Flink writing to the governed Delta
[03:11] would be a pain right now. And that's a problem we want to solve. To see why it's important, we would like to first review what Unity Catalog can
[03:23] provide you. So, most people think Unity Catalog is just a catalog that access can do access control to our
[03:35] tables. It is, but there's more than that. Catalog managed commits, table lineage, and more can be done in the Unity Catalog. Which means external engines
[03:49] such as Flink or other engines, whatever want to use. They can't just drop files into a directory and call it a table. It has to go through Unity Catalog to do
[04:02] the commit, to tell the downstream, and to tell the entire ecosystem that my commit is done, and it's ready to use. That's exactly what old to the existing the deprecated Flink
[04:16] connectors doesn't do, and that's a problem we try to solve here. So, here's some current state of existing Flink connectors. We do have a
[04:29] Flink connector before in the Delta project. Uh but it has been deprecated for a while, and it's now removed from the open-source repo. It does not support the latest Flink.
[04:43] It's built a very early, so only support Flink 1.0 APIs. It doesn't have advanced Delta features, also because it was developed early and
[04:55] lack of maintenance. So, some advanced features like deletion vectors, column mapping, schema evolution, or advanced checkpoints, all these features, they are not present.
[05:13] They are not present in the old Delta connect Delta Flink connector. Also, it's a path only, which we already described a little bit why path only is not enough for a modern table.
[05:25] You need some real table entries in your catalog. So, the old ones doesn't satisfy this need. As a result, we have so many bad consequences of this.
[05:38] So, some teams, some companies, they are our customer, but when we can't provide a good support to their Flink usage, they start doing their own forks. They add their own developed features. But uh
[05:53] it doesn't get merged back to our repo. So other people's account get a chance to enjoy those. And even worse, we have some customers because of this, they got
[06:05] rid of Delta and they try to switch to Iceberg, which has a better ecosystem on Flink connectors. Both of these consequences are something we don't want to see. And we do have the
[06:18] possibility, we have the chance to do better. And that's uh why I'm talking about this work here. And uh this is the new Delta Flink connector that gap
[06:32] close the gap between the two world. It serves as a bridge. So the first part is it's new. We build uh the new Delta Flink connector on the Delta kernel. Uh if you
[06:45] you are not familiar with kernel, I'll briefly introduce that. But later we have a better introduction. Basically, kernel is some shared library maintained by us, maintained by the Delta Lake communities
[06:59] that incorporate with the latest Delta protocols. And it's in active development, it's in well maintenance mode. So it's well maintained. Whenever you have some improvement in Delta protocol,
[07:12] you can expect that to land on kernel pretty soon. Also, it's built on latest Flink connector APIs. Uh V2 connector also supports Flink 2. some 2.x. So that's a
[07:26] good thing cuz we can enjoy the better, more advanced uh Flink APIs. And when people want to use the kernel-based uh Flink connector, they are no longer bound to the old
[07:38] deprecated Flink API. Also, with a kernel support, we support the modern Delta features. Something I already mentioned, row lineage and CDF.
[07:51] Deletion vectors for better deletion write and merge efficiencies. We have checkpoint way two, which can reuse the existing checkpoint to make the checkpoint creation less painful and
[08:05] more much faster. We can write the better states for data gatings on this table to execute more efficiently. And at the end, we are governed by Unity
[08:18] Catalog. So, now the tables, they are governed by Unity Catalog. They are no longer some random files scattered around your cloud storage. And we have catalog managed commits,
[08:31] which means when you commit to a table, you need to talk to the catalog, which is a which do the conflict resolution and make sure your commits is a fully serializable and visible to all
[08:45] downstream applications. Of course, it also does a credential vending and policy management. Make sure you have the rule-based access control, which makes your data
[08:57] safe and make your table safe. At the end, we also have automatic background optimizations and table maintenance. This would include when your table gets bigger, we execute
[09:11] optimize to compress the data. When your data is older, we do vacuum to remove the data from the storage, which makes sure your storage is always efficiency. Always efficient in the storage.
[09:26] So, right now with the new Flink connector, finally the two world no longer have a gap.
[09:38] So, this is some examples how you can use our latest Flink connector. It's pretty easy and straightforward. Basically, you just need to pass this new configurations to a Delta Sink builder, and then you can stream
[09:52] uh the uh Flink stream into the Delta Sink directly, and the data would all land to the table indicated by the sink. We also support the popular Flink SQL
[10:04] APIs, which allow you to create a catalog. In this case, we already have the data created. We have the table created in that catalog. So, the only thing you need to do here
[10:17] is to create a catalog that connecting that connects to the Unity catalog, the one of the catalog in Unity catalog, and you can enjoy all the tables already exist there.
[10:30] What I didn't show, but we also support is that you can also create a table from a SQL API. If you want, you can create a catalog, then create a tables all from the same API here.
[10:49] So, this is a architecture of our new Flink connector. We have data sources on the top, which can be Kafka, uh other systems like a DBMS, or other streaming systems you want to use. They all feed data into
[11:03] Apache Flink, and Flink would feed the data into our Delta Sink. The Delta Sink provides the following uh standard or advanced features like exactly once delivery, which means if
[11:16] your data appeared twice in Flink retry, don't worry. Delta Sink would uh make sure your data only appear once in your final table, and auto remove the duplicates.
[11:29] And it has a support to stream and a batch mode. If you do a streaming table, it's okay. If you do a fling batch right, it also supports that. And we have full memory control, which
[11:42] means if you run the fling connector, you run the Delta sync on a production system, and you want the streaming data to be run for days, for months, or even longer for years, we have full memory
[11:55] control, make sure you don't suffer from out of memory or other problems. And we have file rolling based on your kids, based on your usage
[12:07] strategies. We can split big files into smaller ones and store them separately. Or if you want the big ones, we also have optimizations that can merge small files into bigger ones. So later, if you
[12:21] read them, you get a better performance. Also, we have the schema evolution. If your data has some evolving schema, let's say you start with a one row, one
[12:33] columns in your schema, then you add more later in your data, the connector could automatically detect this change and evolve your table schema based on your latest uh data. You don't
[12:46] need to worry about this. Of course, if uh this is some feature you don't want to use, you can you have the freedom to turn it off. So, all under your control. And at the bottom, we have Delta kernel
[12:59] provides the fundamental part of the turn of the connector. We also have catalog commit APIs that allow us to connect the data to Unity Catalog.
[13:20] So, this entire connector is built on top of Delta kernel. Delta kernel is a Databricks maintain well maintained and uh to the community. It's a shared implementation of Delta semantics. It's well maintained, contains all the
[13:32] features, and under active development. So, it contains all the modern Delta features. In the world of features we have in the protocol, we will as soon as possible we'll it improve that, we'll
[13:46] implement those features into the Delta kernel. So, the open source world can enjoy all the features just as what we have in inside of Databricks. It's also a strong kernel. It lives in
[13:59] higher system. What you can see here is not only used by Flink connector, it's also used right now by the Spark connector, by Kafka, and by many other connectors within and in the open source
[14:14] world and in Databricks. Delta kernel is a future how we maintain the entire ecosystem and maintain the community. It has a connections from the top side as connection to the
[14:26] open source world. At the bottom, it connects to Delta tables and the Unity Catalog. So, this serve as a connection between the two worlds. Uh here I would like to quickly
[14:39] introduce how the Flink connector can maintain uh exactly once delivery and make sure data would not be corrupted during Flink retries or job failures.
[14:53] So, on the left side we have data pouring in into Flink and Flink would do the writings. It would write a data on the disk as parquet files. These parquet files now only lives on the
[15:06] cloud, lives in your storage, but they are not yet part of the table. And next part is when Flink checkpoint happens. Flink will regularly trigger a
[15:18] checkpoint. And when that happens, we combine the right written parquet files and merge them into commit balls, then submit them to a delta table.
[15:31] Finally, we'll call Unity Catalog to write this commit into Unity Catalog, which is a what we already keep mentioning, the catalog managed commit. Only after the catalog commit is
[15:44] finished, we consider the entire commit to the table is done. So, if something happens in between, for example, if something your job failed at
[15:56] step four or five, you will have your data on this cuz it's already written on step two. But, this data they will never be visible in the table cuz they has not
[16:08] been They have not been committed to the catalog. This guarantees also when Flink retries, it will retry with the same data file and will only will make sure
[16:20] the table is written only once. So, your those during multiple retries and failure recoveries will never suffer duplicated rows or lost rows.
[16:38] So, our system delivers uh for different parties, we deliver different uh functionalities. For streaming team, we have native Flink support, failure recovery, quick start and stop. For platform teams, we have a user support
[16:51] for govern table identity and the central access. For downstream users, we have query ready delta tables and SQL BI ML downstream, all these are ready.
[17:08] So, we also consider performance and operational efficiency. We have cached the delta snapshot allow you to start a Flink job very fast. And when you want to do a restart, we also make sure the job can be canceled
[17:21] at us immediately without waiting for operations. We have the async checkpoint, make sure works happens in background, doesn't block your job or commit, and we have sub-second commit latencies.
[17:35] We have PO integration, which optimize your table in background. We have better maintenance for long-running stream tasks. And here's some numbers we got and we want to share with you.
[17:49] The table can be run the Flink job can run pretty for long term, and we make sure there's no memory leak, there's no problem for long-running jobs. We can support high cardinality partitions. We
[18:01] got a customer who want 1 million plus partitions in their table, and we can support that without problem. When you want to cancel a job, the job can be canceled almost immediately. That
[18:15] doesn't need to No matter you are waiting for the table to reconstruct, you are waiting for some job to execute, we can support canceling the job almost instantly when you want to do that.
[18:32] So, there Here's the today's situation of the product. We have the following features ready. Append only sink. Support to Delta stream and SQL APIs. Latest Flink API support. Catalog commit
[18:46] support. And upcoming is a way what we are actually working on. Upsert support, schema evolution, and dynamic routing to multiple tables. And later we'll also enable end-to-end
[19:00] streaming with Flink. But also set up the Flink source part. Now we only have the sink part ready. You can write. And we also working on the source part, which allows you to read the table.
[19:13] And all the source code and the latest situations, you can find that in Delta Lake repo, which you can scan the code or you have the URLs on top. Uh
[19:25] we have a Yeah. Well, now I'm open to questions. Sorry, I didn't stop early, but we have some time for questions now.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.