How to Query Streaming and Iceberg Data Together with Redpanda SQL and Databricks

Summary

Streaming queries: Redpanda SQL queries streaming data before it lands in Iceberg.
Bridge queries: Join live events with Iceberg history in one SQL endpoint.
Iceberg topics: Schematized, partitioned data write directly into Databricks managed tables.

How to Query Streaming and Iceberg Data Together with Redpanda SQL and Databricks

Talk by Matthew Schumpert, Head of Product, Platform, Redpanda
You can query streaming data before it lands in the lakehouse: Redpanda SQL exposes topics as tables behind a Postgres-compatible endpoint, so one SQL query joins live events with Apache Iceberg history. That closes the lakehouse blind spot where teams can analyze history but not the last few seconds, and Redpanda Iceberg topics write schematized, partitioned data straight into Databricks as Iceberg managed tables in Unity Catalog.
The talk explains how Redpanda Iceberg topics write schematized, partitioned data into Databricks as Iceberg managed tables in Unity Catalog, why a one to two minute delay can still remain, and how bridge queries transparently reach from a stream back into Iceberg history. A demo uses IoT telemetry from cell towers, with 72 million records, to find faulty towers with packet loss, comparing a human analyst with Claude querying a single SQL endpoint. It also covers Redpanda's lower total cost of ownership than Kafka, Redpanda Connect CDC connectors and ZeroBus for point-to-point ingestion into the lakehouse.
Sponsored by Redpanda.

Chapters

FAQs

What is the lakehouse blind spot and why does it matter?

Teams can analyze historical data in Iceberg but can't see the last few seconds of streaming data. This video explains that Redpanda SQL closes this blind spot by making live topics queryable as tables alongside Iceberg history.

What are bridge queries and how do they work?

Bridge queries transparently reach from a stream back into Iceberg history in a single SQL statement. This video demonstrates querying 72 million IoT records across both streaming and historical data to find faulty cell towers with packet loss.

How do Iceberg topics write data into Databricks?

Redpanda Iceberg topics write schematized, partitioned data straight into Databricks as Iceberg managed tables in Unity Catalog. This video notes that a one to two minute delay may remain, but schema and partitioning are preserved.

How does Redpanda lower total cost of ownership compared to Kafka?

This video emphasizes that Redpanda's pure C++ streaming platform delivers significantly lower total cost of ownership than Kafka with better performance. This cost advantage makes real-time data infrastructure more affordable for enterprises.

Full transcript

`00:00` Introduction

Okay, thank you all. We are starting. We have a lot of material, so let's start very quickly. Thank you for joining us today. Thank you for giving us some of your time. If you're not familiar with Redpanda, I'll start with the basics. We will talk about eliminating "blind spots " in lakehouse, but I will highlight four main points for you. First, Redpanda is one of the leading streaming data platforms in the world, in fact the leading independent streaming solution provider that modernizes your Kafka infrastructure, saving you a ton of money.
This is the same product that made us our name. We also make streaming data into Databricks fast, easy , secure, and incredibly cheap. We will talk a lot about this . We will discuss the main topic of the talk—Redpanda SQL, i. e. the ability to combine streams and Iceberg into a single Postgres table for single-pass queries, which opens up huge opportunities for many use cases. These are what we call bridge queries, and we'll also talk about how Redpanda SQL delivers real-world data and agent workloads with millisecond freshness.
So, streaming data networks in general are already the connective tissue between applications, microservices, and increasingly, agents that work very efficiently using fresh data and well- written SQL. So this creates a great foundation for an agentic context. As I mentioned, Redpanda powers many of the world's most mission-critical streaming applications and real-time systems for leading Fortune 500 companies. You see some names here, like the New York Stock Exchange.
Rivian, of course, you know them from this conference. Much of their data platform is powered by a combination of Redpanda and Databricks; they migrated their entire streaming platform from Amazon MSK to Redpanda a few years ago. This is because our pure C++ streaming data platform delivers a significantly lower total cost of ownership ( TCO) than Kafka, with much better performance. It is significantly lower. I won't dwell on this, but get to know us.

`01:46` Redpanda vs Kafka: total cost of ownership

Over the past few years, we have expanded to a broader real-time data platform by adding streaming connectivity. The colors are a little pale, sorry. And now SQL, which we'll talk about, as well as tools for managing agents and providing a level of context using this connectivity. We have 300 different connectors. The streaming data platform at the heart of all this, as well as a newcomer to the field— Redpanda SQL. A quick note on Redpanda Connect: it's becoming very, very mature with a broad portfolio of CDC connectors that help you get data from your transactional databases into your streaming tier , your real-time tier, and then into your lakehouse, or directly into your lakehouse, bypassing the streaming tier altogether.
Be sure to pay attention to this. It can also send data downstream to many different applications, including SaaS applications. This is a huge briefcase. All the products we have come in a wide range of form factors and deployment models: from serverless to single-tenant cloud, bring your own cloud (like Databricks running in your own VPC, your cloud account, the three major clouds you see here), and an Enterprise version that can run locally, at the edge, or on your laptop because Redpanda is very compact, compute-efficient, and tightly packaged.
So just keep that in mind. We work wherever you need to handle streaming data, SQL, asynchronous streaming pipelines, etc. Databricks and Redpanda have a long history of successful collaboration. We have been collaborating from an engineering and product perspective for two years, and a year ago here at the Data and AI Summit, we launched our integration with Iceberg managed tables in Unity Catalog. So, as I mentioned, it's a very cost- effective, productive, and powerful streaming data engine that can ingest data into Databricks extremely quickly and easily, as you'll see, and gives you the flexibility to run any event and streaming use case—whether it's latency-tolerant, or very low latency, or just moving data to lakehouse, or any combination of those in a single environment, a single cluster.

`03:34` Iceberg managed tables in Unity Catalog

You don't need different products from different vendors to handle all of these streaming workloads. Last year, we launched full support for Apache Iceberg and Iceberg Managed Tables with Databricks, which actually only reached GA status last month. And here we are on the big screen last year at the keynote as one of the partners in the launch of Iceberg managed spreadsheets. So the first step to world domination here is streaming data directly from all its origins, delivering it in real time to your applications and agents, and then putting it in your lakehouse for analytics in a very flexible way.
Again, the slide looks a little pale, but you have a log form of data and a tabular form of data. The log form is implemented through the Kafka API, so you get the Kafka ecosystem toolkit, and all your Kafka clients connect directly there because we support the full range of Kafka APIs. We are a complete replacement, you don't need to recompile your programs, replace clients, or reconfigure anything. We look like Kafka for your streaming applications, but you get a tabular view of the data where you can use, for example, Spark SQL.
You can take Databricks SQL or Ray and point them directly to an Iceberg-formatted tabular data form registered in your workspace in the Unity Catalog. This is actually the first step, just getting the data into the bronze layer , but it's a very powerful first step and a very flexible way to do it, right? We have n't even gotten to the SQL part yet. You actually get schematized, partitioned, cleaned data with error message queues at your desired latency, where you can tune the engine to deliver data as fast or as slow as you need, depending on your application requirements.

`05:28` Iceberg topics: schematized streaming data in Databricks

This is another example of data transfer in Snowflake. We can also route data to Glue, Big Lake, and other Iceberg directories if you need that data for lakehouse. And all this is just thanks to a few properties of the topic, right? So, no more Flink jobs, Kafka Connect clusters, or writing code for ETL applications manually to create point-to-point connections. Declarative: These are the streams I want to have in my lakehouse.
Boom, I define it with Terraform and just run it. So , you find yourself in Databricks in your workspace. All your tables are available. They also receive Kafka metadata. That is, the topic, the partition, the key, the headers, the timestamp —all of this, the history of the data's provenance as it passes through the Kafka streaming layer, which is Redpanda, is added along with your schematized data in Avro, Proto, or JSON format, partitioned as I mentioned, but there are still "blind spots," and that's the main topic of today's conversation.
So, lakehouse and the streaming layer (or Kafka layer) usually create some kind of isolated data ( silos) within the enterprise. There are several problems. Some of them are simply a barrier to access to Kafka itself. You need skills with Kafka or its ecosystem tools, and you need to build these pipelines to move data between the streaming layer and the lakehouse itself. Another aspect is that you define your representations in a certain way.

`06:41` The lakehouse blind spot: streaming data silos

You might be creating a table via select, like a Delta Live Table, but you've created a specific view of the stream , not the raw stream itself, and it's predefined , and if you want to change it, you have to go ahead and create another view or another table. This is a data engineering task, right? This is not just a declarative approach . It's not just a matter of verifying that the "raw" data is in place, and you still have a delay .
So, even with Panda Iceberg topics making the data transfer process in Databricks and Unity so seamless, there is still a potential 1–2 minute delay. With Delta, you can achieve slightly higher speeds. I spoke with the co-founder of Iceberg yesterday, and he says that the 1–2 minute data arrival time for Iceberg, even V3, is quite optimistic based on his experience with customers. And this is true for us too. So what if your use case , your analytics scenario, is really latency sensitive, meaning you need to have access to fresh data with accuracy down to the second or millisecond to be efficient, responsive, and meet application needs?
Now you need to run Flink or Spark jobs, which requires a lot of skills and resources. Not many people know how to do this. You can't do it ad hoc, and it's not democratized, right? So the idea behind Redpanda SQL is to obviously break down silos. I'm sure you've heard this before, but the ability to seamlessly access fresh millisecond data, extending it to your Lakehouse and Iceberg as a single entity, as a single table—just like in Postgres—with a simple and convenient backend query that you can plug into your applications as if it were a regular Postgres database—is truly powerful.
This is actually a very, very powerful concept. So this can provide ad hoc analytics. This can enable applications that simply interact with what they see as a Postgres database. This can also enable agents that, you know, consume SQL for breakfast, essentially— produce SQL or whatever data for breakfast. So, step two to taking over the world is what we call bridge queries. This is a powerful concept of combining data in Iceberg with your streaming layer data, which is also delivered to applications that might have a 5-millisecond SLA or, you know, gigabytes per second streams with pretty tight SLA requirements.
We power the largest companies in AdTech, gaming, finance, and manufacturing in the world, and move that data to Iceberg. If you saw the ELT presentation at the talk earlier this week, we do for your streams what ELT does for your OLTP data, right? So you have a warehouse-based OLTP database , they convert it to a columnar format so it's analytics-ready. And this happens automatically and naturally right in the data lake, simply by being in the real- time layer, i. e. in the OLTP layer .
The same goes for streams, right? In the background, the streams that you want to make available in the repository, or rather, in the data lake, you just enable this topic property, and boom—they're right there. But the beauty of bridge queries is that they seamlessly bring them together. So if I have a day's worth of data in a streaming platform and months or years in a data lake, and I query for the last 24 hours, maybe only the last minute hasn't arrived and been captured in Unity Catalog yet , right?

`09:42` Bridge queries unify streams and Iceberg tables

So the engine will determine exactly how much data it can pull from Iceberg, and then it will take that same tiny piece of the most recent data via the Kafka API, combine it into a single result, right? No gaps, no duplicates, one query, one pass over the data, no two tasks, no checkpoints or intermediate data storage. Very , very powerful. So, Redpanda SQL is a real- time SQL query engine based on Postgres. Familiar syntax, Postgres is a kind of lingua franca today, many products use it.

`10:19` Redpanda SQL: a Postgres-compatible real-time query engine

You can run common analytical queries directly to streams as a regular user, without learning the Kafka API, using software frameworks, or complex pipelines, and still access data in Iceberg, if you want, directly through Redpanda SQL. It just is, it's part of our platform. It's right there in our cloud interface. Why do we need another OLAP engine? Well, this engine was pretty special. Many of our customers have been asking us for years, " Please give us SQL."
We are tired of using Kafka clients to explore and debug our streaming applications, Flink jobs, etc. And sometimes we just want to do a quick situational analysis of the stream layer, not the data lake. And we have use cases for that. We chose Oxla, which we acquired last year, because their technology was very memory efficient. And this has actually been the " Achilles heel" of some OLAP engines for a long time, as the number of cores has outpaced the memory bandwidth per core.
So, you can get more throughput through the SQL engine if you compress the data into a smaller amount of memory. And that's why Oxla is very strong in this aspect. You can see us on VLDB. We just received approval for a paper for the VLDB conference. You can also check out this lecture at Carnegie Mellon University that we gave to their database development group . We were one of their lecturers earlier this year. This is quite interesting stuff .
So, Redpanda SQL, powered by the C++ MPP engine Oxla, opens up new possibilities. This is democratized Postgres-based SQL access to any Kafka or Redpanda topic. This is the first. Secondly, these are end-to-end queries. It's a single seamless timeline, one query—from milliseconds to months or years of data, depending on what's in your Lakehouse. If you don't know in advance whether your analytical case will require such data freshness and don't want to deal with two systems, or you have a real-time task that needs to be combined with other data—this greatly simplifies the process.
You can combine historical data with real-time data very easily. Again, within a single request. You simply enable this in your Redpanda cluster running on one of the top three cloud services, where our cloud services are available next to your streaming platform in your VPC, right next to Databricks where the data is available. We are connected to Unity Catalog through the Iceberg REST Catalog interface, and you just get a Postgres endpoint.
Eventually, this will be available as an editor directly in our interface. But you can use P-SQL or any other Postgres client to log in and get started. "Under the hood" we do a lot of this work for you. We create a directory: there is an Iceberg directory and a Redpanda directory that points to the Iceberg directory. In our cloud product, all of these things are pre- configured for you. In our self-managed product, you can point to different Iceberg lakes, for example, or even different Kafka directories if you wish.
All you have to do is create a table and get started, because we've already set everything up for you, including everything in your Databricks data lake , right? So you get the SQL Studio environment that we're launching next month, right next to your threads, topics, schemas, and schema registry—if you're familiar with that , you know what I'm talking about. And all of this is available in an intuitive cloud interface, the web UI.
You will also see that you will be able to access other Iceberg data lakes as well. So you can “bring your own Iceberg,” so to speak. Add your own Iceberg tables, connect other catalogs if your data is in, for example, Snowflake, Glue, Big Lake, or MinIO. Yesterday I spoke with a customer who uses this with MinIO. Very cool. Okay, let's do a quick demo of IoT analytics with Agentic Assist. This demonstration is about cell towers , faulty towers experiencing packet loss and other issues.

`13:56` Demo: IoT cell tower analytics with an AI agent

And you can easily explore this data as a human or with the help of an agent. I'm starting this video. This will only take us a few minutes. I'll try to do it this way... that's all. Done . Okay, what's going to happen here: first , we're going to connect to the system. We connect via OIDC. So the system supports OIDC authentication. We log into our cloud service. Here we will simply use the P SQL client. As I said , in the future you will be able to use the SQL editor directly in the interface.
We already have the directories created, as I mentioned: the Iceberg directory and the Red Panda directory that overlays it. So, the Red Panda catalog depends on the Iceberg catalog. Why? Because the streams naturally turn into Iceberg. Stream is the most recent part of the data layer, and Iceberg is the older layer. Now I will grant myself access to these directories and tables, and start working with the data. But I'll also let Claude work with this data.
So, here the human and the agent are competing. So, we're going to run Claude and see what it can detect about this telemetry data and what exactly are the faulty towers in my cellular infrastructure . So I'll just give Claude a simple tip. I created a skill that tells me where to connect, which is my Claude MD cluster. And I just give it a natural language query: "Here's a table. I want to look at the data for the last 7 days because I noticed packet loss.
I got a packet loss notification, but I don't know how many towers it was and which towers it was. This data is usually stored in my streaming platform because often the raw data is only there, but it has a relatively short shelf life, right? So let's see what that looks like in practice. So Claude goes to work, he explores this data set and tries to find anomalies. And I, as a human, am going to do the same thing, right?
I've given myself access to a catalog that was created for me . You see here the signal strength, the packet loss, the error codes—all of that is in the table. Again, it looks like a Postgres table to me, but it's actually a Red Panda topic that extends into Iceberg, as I explained. So we're going to do some bridging queries to that. You see, here I have 7... or is that 72 million records? 72 million records in this table from a Postgres perspective.

`16:25` Querying 72 million records across stream and lakehouse

But again , this is a standard Postgres protocol. I can plug standard Postgres clients into this. Here, in my Red Panda cloud cluster , Red Panda BYOC, I only have 9 million records. Why is that? Because in my streaming layer, the data is only stored for one day. So, one day's data is available through the Kafka API. It's a pretty common situation where your storage has a longer retention period. Here, I have, as you can see, 77 million records.
So there's a lot more information available in the data store, but to me it all looks like one entity . As a single entity, one data set. And so I'm going to start running some queries on it. And I'm going to look, for example, here at the towers that have packet loss, right? And it's going to come back and tell me, okay, I found a tower with packet loss, but I only looked at the last day, right? So remember, that was available in my streaming tier for one day.
So if I just ran a Spark or Flink job, or used, say, KSQL DB or something like that to directly access the Kafka data, it would only find me that one tower. When I extend the query length to 7 days, it transparently goes deeper into Iceberg history and essentially comes back with more results, right? So I'm not going to find one tower that has packet loss, but three. And by the way, I'm not seeing any error messages.
Why am I not seeing any errors? It's going to be hard to debug packet loss without errors. Let's find a few more towers that aren't throwing errors. Maybe there's a firmware issue that's preventing them from throwing errors that would help me solve the real packet loss problem, right? So I'll notice that. Typical data analytics exploration, right? Very seamless, very easy. It doesn't matter if the data is in a stream or in a lakehouse.
Yes , there are more towers, and some of them have packet loss too . These are all the ones that don't give any errors. They're all running the same firmware. Oh , there's a firmware issue. I need to send a technician out to these towers and update the firmware. It's really time-consuming and expensive for the technician. I better do it right and not just limit myself to a subset of the towers. They're going to go next week and find that there were actually more problem towers.
So it's this full end-to-end look at the entire time frame that allows me to get the right analytics, right? So I can find other problems in my data . You know, there's a high signal-to-noise ratio here, right? Or a low signal-to-noise ratio. More noise than signal. Let's see what the agent found. The agent actually found even more than that. He found a signal in the noise. He found a firmware issue. So we're not issuing any errors.
This is the bottom of the table. He also found two other issues in the data just by analyzing and investigating it, finding, "Okay, this clock is stuck, it's not moving." And this one has a duplicate device ID. So there's a bunch of issues that need to be resolved. And now we have an even broader list of towers in different regions to create a list of tasks to send to the field technicians to fix this equipment. So I, as a human, did a pretty good job.
The agent did a better job because agents are really good at writing SQL queries and finding patterns that we might not notice, but they don't know that you might have just one day of data in your streaming platform and a longer history in the data lake. They don't have the context of the entire landscape of your data, right ? So they would essentially just be looking at one side instead of the other. And that's what you have to give them.
You know, sometimes these agents are like interns, as we know. You just have to give them a single A SQL endpoint , a single table, and just say, " Go." That's basically where we are now. Stage three: Profit . Streams become just a set of tables in the data lake , or "streamhouse" if you will, in the streaming corner of your data lake. Applications and agents get fast analytics on " hot" data with millisecond freshness.

`20:03` Streams as tables: ZeroBus and the streamhouse

Also, deep history in Iceberg with no integration hassles and no special skills required. The compute overhead is much lower than traditional streaming methods because it's a very memory- and compute-efficient Postgres engine. It runs on minimal resources, and you can still use ZeroBus for point-to-point integration if you don't need streaming for other reasons. You could just plug your application directly into ZeroBus , but then you're doing point-to-point and just going from the application to the data lake, right?
If you already have a need for streaming data delivery, stream buffering, and service hooking , which is what most people do in their applications, you can use Redpanda for that. Tick the box, get the data into the data lake, and then query it if you want, or use Red and DB SQL—whichever is the best SQL engine for the job. Want to learn more? Our booth 402 is right here. It's very, very close. Come see us. We'll also be at the VLDB conference.
Check out the general availability (GA) announcement we made a few weeks ago, and thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.