Skip to main content

Confluent Tableflow with Databricks: Stream Kafka Topics into Delta Tables in Unity Catalog

Summary

  • Managed ingestion: Tableflow automates converting Kafka topics into Delta Lake or Apache Iceberg tables registered with Unity Catalog.
  • Automatic optimization: handles format conversion, schema management, compaction and garbage collection automatically.
  • Downstream processing: Flink can add further processing while Tableflow maintains the read-only table.

Confluent Tableflow with Databricks: Stream Kafka Topics into Delta Tables in Unity Catalog

Watch: Confluent Tableflow with Databricks: Stream Kafka Topics into Delta Tables in Unity Catalog
Talk by Kasun Indrasiri Gamage, Senior Product Manager, Confluent
Confluent Tableflow turns Kafka topics into Delta Lake or Apache Iceberg tables and registers them with Databricks Unity Catalog, so real-time operational data becomes governed, AI-ready tables without a custom streaming pipeline. This video explains why Kafka-to-lake pipelines are hard, covering format conversion, schema handling, compaction, garbage collection and catalog publishing, and shows how Tableflow uses the schema from Confluent Cloud to create the target table and handle table maintenance for you.
It explains why moving operational Kafka data into a data lake is hard, including format conversion, schema handling, compaction, garbage collection and catalog publishing, and how Tableflow uses the schema from Confluent Cloud to create the target table schema and handle table maintenance. A demo streams stock ticker data from a Kafka topic into Delta tables in your own S3 storage, sets up catalog sync with a service principal client ID and secret, and queries the result in Unity Catalog. The Tableflow table is treated as read-only because Tableflow keeps managing and optimizing it, and Confluent Flink can add further streaming processing.
Sponsored by Confluent.

Chapters

FAQs

How do I get Kafka data into Databricks?

Confluent Tableflow is a managed ingestion service that automatically converts Kafka topics into Delta Lake or Apache Iceberg tables and registers them with Databricks Unity Catalog, eliminating the need for custom streaming pipelines.

What makes Kafka-to-lake pipelines complex?

Format conversion, schema handling, compaction, garbage collection and catalog publishing all require custom code. This video demonstrates how Tableflow handles these automatically using the schema from Confluent Cloud.

Can I analyze real-time Kafka data with Databricks?

Yes—Tableflow streams stock ticker data into Delta tables as a demo, then you can query the result in Unity Catalog or set up dashboards. The Tableflow table is read-only because Tableflow maintains it, but Confluent Flink can add further processing.

What is zero-copy data with Tableflow?

Tableflow syncs Kafka data into your own S3 storage with catalog sync, so you get a managed data pipeline without duplicating data across systems or building complex ETL code.

Full transcript

00:00 Introduction: AI-ready tables from Kafka

OK. So, in this session, I'll walk you through how you can bring your streaming data from Kafka as AI-ready tables to your data lake using Confluent Table Flow, Databricks Unity Catalog, and Delta Lake tables. My name is Kasun Indrasiri. I am a Senior Product Manager at Confluent, working on the Confluent Table Flow product . So let's move on to today's session. If you look at the current data and AI landscape, we make most of our decisions based on them.
This could be a personal decision by a consumer or even a business decision by an organization based on insights gained from AI. Isn't that right? What about making these decisions based on data available last month, last week, or even yesterday? Right? Therefore, getting real-time data for your AI models is extremely important . In fact, real-time data is a vital foundation for modern AI platforms. Now, when thinking about feeding all your real-time data to your AI models, you need to first connect your streaming or operational data to your data lakes and data warehouses.

01:22 The operational and analytical data divide

If you look at the current data landscape of almost any enterprise, you can see that data exists in two basic states. So , on the left you have the operating side. This is where you collect all your operational data. This could be clickstream data, data from your daily business transactions, or even operational data collected in the form of CDC change logs from your operational databases. If you look at the operational state, Apache Kafka has become the de facto standard for collecting, organizing, and storing all your operational data.
And if you look at the right side, here you have analytical data. This is where you perform business analytics to generate a variety of insights from your data. And this is exactly where you feed data into your AI and ML models. Looking at the analytics landscape, all data warehouse and data lake vendors are converging on using open table formats such as Apache Iceberg and Delta Lake. These open table formats provide some key features that were previously missing from conventional data lakes, such as ACID transaction execution, complex SQL queries on large data sets, hidden partitioning, and the list goes on.
The main challenge most organizations face is how to connect operational data with analytics and migrate data from Kafka to Iceberg or Delta Lake formats in your data lake. Typically, you have to build complex, expensive, and error-prone data pipelines that take data from Kafka, perform transformations, and ultimately get these open tabular formats. Now let's take a closer look at some of these data pipelines. In this diagram, as you can see, your operational data is in Kafka.

03:20 Why Kafka-to-lake pipelines are complex

You usually use something like a record connector, right? So, you use a connector to dump raw data into your object store . And after uploading, you need to start preprocessing or improving the quality and structure of the data. This includes, for example, converting from the required data format from raw Kafka records to Parquet and Iceberg or Delta Lake metadata formats. And also creating schemas for data. And since it's a continuous stream, you also need to take care of the problem of a large number of small files, right?
You are constantly writing data, which means you are creating lots of small files. Therefore, you need to perform compaction, garbage collection, and other table maintenance work. Once you have created all these tables, you need to publish them to catalogs so that analytical systems can find and use them. Having done all this, you can get "bronze" quality tables, and on top of them - perform CDC - materialization, updates, deduplication, etc. , to get " silver" or " gold" quality data as a result.
As you can see, it is an extremely tedious process to move all the operational data to the storage, and you have to take on all the hard work of pre- processing. Imagine a scenario where you completely get rid of this routine and your Kafka topic becomes available in a few clicks as a Delta Lake or Iceberg table. That's why we developed Table Flow 4. Right? So, with Table Flow, you can convert your Kafka topic to Iceberg or Delta Lake format in just a few clicks.

05:39 What is Confluent Tableflow?

Table Flow is offered as a managed service in Confluent Cloud. As a Confluent Cloud user, you can simply click "enable Table Flow" for the selected topic and it will be materialized in your object store. So, along with this, Table Flow will also take care of all the data preparation that you saw on the previous slide, and we will also make these tables seamlessly available to downstream computing systems and catalog services.
Now, if you look at the end-to-end data flow within the Confluent data streaming platform, on the left you will see all the different sources related to operational data. These can be databases or various applications . So, using Confluent's output connectors, you can connect all this streaming data and present it as Kafka topics in your Kafka clusters, right? And then, if needed, you can perform additional streaming processing using Confluent Flink to instantly gain insights into the Confluent ecosystem, create new data products, and provide them to your data lake using Table Flow as Iceberg or Delta Lake tables.
Now, when you do this, the bridge between operational and analytical data will be created and it will be completely manageable for you. You no longer have to worry about managing hundreds or thousands of complex ETL jobs. Now let me do a quick demonstration of using Table Flow with Delta Lake and Unity Catalog tables. OK. So, before that, I'll go into some details of Table Flow. Here are the features offered within the Table Flow product.
When you enable Table Flow for a specific Kafka topic, we start materializing the Kafka topic as an Iceberg or Delta Lake table. For this we use the schema register. So, every Kafka topic or every event has a schema associated with it when you use Confluent Cloud, right? And we use this schema to create the target table schema. As part of data preparation, we perform raw record conversion to Parquet and Iceberg format, data schematization, type mapping between your source type and Iceberg or Delta Lake types, and table maintenance.

08:16 Schema-driven table creation and table maintenance

Now these tables are available, for example, in the case of Iceberg, we provide them through Iceberg's built-in REST catalog. So if you have an engine that supports the REST catalog, you can connect directly to the Table Flow catalog and use them. Or you can configure external catalogs such as Unity Catalog, AWS Glue, Snowflake, Horizon Catalog, etc. So, once these tables are made available through these directory services, they will also be stored in your own storage bucket, right?
So, you can use your own bucket and store these tables in it as well. Now you can connect almost any compute engine compatible with Iceberg or Delta Lake to this operational data. So, Table Flow has several unique features. They include, uh, you can present a specific topic in both table formats. You can present the same topic using Iceberg or Delta Lake. And you can also integrate it with many catalog services. This includes Unity Catalog, Glue Catalog, Snowflake Catalog.
This way, you keep only one copy of the data, but make it accessible to many computing engines and cataloging services. OK. Now it's time to do a quick demonstration for you. So, I'm in Confluent Cloud, and here's my Kafka topic. As you can see, I have a schema defined for my Kafka topic, and this is my Kafka application that is constantly getting real-time data about stocks. And as you can see, the data now appears in my Kafka topic.

10:07 Demo: Kafka topic to Delta table

This is all the operational data I currently have that has been successfully ingested into my Confluent Cloud Kafka topic . Let's take a look at some of these events. As you can see, we have data related to stock tickers. My goal is to feed this operational data into my data analytics layer. As I mentioned, you can simply click "enable table flow" and then choose the target table format. Here I choose both formats: Delta Lake and Iceberg .
This means that we will use the same copy of the data, but two different metadata layers will be created for it. Therefore, the data files are common to both formats. This is where you configure your storage. You can customize, in this case I am using S3 as my storage. This is my own data storage space. After that, you can move on to the next step and enable Table Flow. So , to convert a theme to an Iceberg or Delta Lake table, that's all you need to do.
Now if I go to my data contract Yes. If I go to the monitoring section, you will see a new storage location. This is the actual location of the base table in my repository. I can navigate to it in my S3 bucket. And I will be able to see how the data materializes. As I mentioned earlier, you have data files, or Parquet files, created in the repository. And then I have the delta log. This is Delta metadata. And we also have Iceberg metadata.
Truth? Now the next step is to make this data available or query it , right? So, to do this, you can set up directory synchronization. This means that these tables will be automatically connected to subsequent directory services. In this particular scenario, I have already configured Glue and an open Snowflake directory. Now let me add Databricks Unity as a new catalog so that I can use these tables through Databricks Unity as a Delta Lake table.

12:40 Catalog sync for downstream catalogs

So, I created an external location pointing to my repository , and now I can create a directory sync, right? This is the exact storage that will be shared between Confluent and Databricks. I have already created the necessary service principals and everything else. Also set up the necessary permissions for access, as Confluent needs to write data to this external location, as well as publish it to the Unity directory.
You can specify— we currently support Delta as the target format—and then you need to provide the Databricks Unity Catalog workspace details. You can copy the workspace URL and then specify the client ID and secret associated with your service principal. After that, you need to specify the name of the directory. I'm going to use the "table for quick start" as a directory. Now you can start directory synchronization. So, within a few seconds you will see that Unity Catalog is also connected.

13:56 Connecting Unity Catalog with a service principal

If you look at this screen, you will see three directories connected to the same cluster. This means that the same tables are shared across multiple directory services. Depending on the supported table formats, they will be published in the corresponding directory services. The directory is now successfully connected. This means I can find these tables through the Unity Catalog. We use the cluster ID as the schema name.
You can adjust this as needed. Now I can go to the Unity Catalog in my Databricks workspace. And I can find a table with a real- time stream of stock quotes. True? This is the same spreadsheet I had in Confluent Cloud, which is now automatically converted to a Delta Lake spreadsheet. You can configure the necessary permissions to continue executing requests. And now you can go to SQL queries and execute the select statement.

15:05 Querying the Tableflow table in Unity Catalog

Okay, so now you can execute the query. Now you can get all the data that was in your Kafka topic as a Delta Lake table. And now you can consider this a bronze or silver quality table and continue with the rest of the data pipeline. So, you can use almost all Databricks features for this table. This table is mostly treated as a read-only table because Confluent Table Flow is constantly managing and optimizing it. So you need to make it read-only and continue with the rest of the data flow .
In this particular scenario, I created several data pipelines using Databricks to generate business intelligence and create a dashboard based on data from my Kafka topic. Since I'm sure you're all familiar with Databricks, I won't go into detail, but you can continue with the various use cases. Here I selected some daily market data and eventually created some dashboards using the same data that was in my Kafka topic.
An important aspect: if you imagine the data you have in Kafka clusters or topics, how quickly can you get these insights? With Table Flow, we were able to significantly reduce data preparation efforts and make it easier to move operational data into the world of analytics, preparing it for AI. OK. If you are interested in learning more about how Table Flow is used in real-world cases, we will have another afternoon session hosted by Granger.

17:34 Summary: simplifying real-time data for AI

They will detail how they simplified the integration of Kafka operational data with Databricks using Confluent Table Flow. And... so we continue... Yes, we continue to work closely with Databricks on integrating Confluent Table Flow and the Confluent ecosystem with Databricks. We have now completed the phase of moving operational data to analytical data, and we are also working on moving analytical data to operational data using a reverse table flow, which will be available next year.
So, if you are new to Confluent, you can scan the QR code and try Confluent Cloud. And... yes. If you have any further questions, please do not hesitate to contact us at our stands 510 and 537.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.