A Comprehensive Guide to Streaming on the Data Intelligence Platform

Summary

• Streaming Fundamentals: This video provides a comprehensive guide to streaming on the Databricks Data and AI platform, covering core concepts and best practices for real-time data pipelines.
• Data Ingestion Patterns: Learn various streaming patterns and architectures for ingesting and processing continuous data streams efficiently within the lakehouse.
• Production Deployment: Discover how to deploy and manage streaming workloads in production on the Databricks Data and AI platform for reliability and scalability.

A Comprehensive Guide to Streaming on the Data Intelligence Platform

Talk by Indrajit Roy, Director of Engineering, Databricks ; Ray Zhu, Director, Product Management, Databricks
Is stream processing the future? We think so — and we’re building it with you using the latest capabilities in Apache Spark™ Structured Streaming. If you're a power user, this video is for you: we’ll demo new advanced features, from state transformations to real-time mode. If you prefer simplicity, this video is also for you: we’ll show how DLT simplifies managing streaming pipelines. And if you’re somewhere in between, we’ve got you covered — we’ll explain when to use your own streaming jobs versus DLT.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to the agenda
• 1:15 Streaming vs. batch concepts
• 8:11 Understanding the Enzyme engine
• 11:20 Databricks platform integration
• 14:02 Choosing between engines
• 16:43 Expressiveness vs. ease of use
• 19:52 Innovations in streaming
• 25:40 Innovations in materialized views
• 27:59 Streaming and DLT demo

FAQs

What streaming patterns does Databricks support?

This video covers various streaming patterns and architectures for implementing real-time data pipelines on the Databricks Data and AI platform.

How do you deploy streaming workloads in production?

According to this video, Databricks provides tools and best practices for deploying and managing streaming workloads reliably and at scale in production environments.

Micro-batch vs. continuous streaming: which is better for low-latency operational workloads?

Micro-batch streaming processes data in small batches at regular intervals and is better for scenarios where you can tolerate slight delays in exchange for higher throughput and fault tolerance. This video explains how micro-batch streaming, implemented by Apache Spark Structured Streaming, excels at processing high-volume data while maintaining exactly-once semantics and supporting complex transformations. Micro-batch is ideal for most enterprise workloads including real-time analytics and data pipelines.

Structured Streaming versus Kafka Streams?

Structured Streaming on Databricks and Kafka Streams are complementary streaming technologies with different strengths—Structured Streaming excels at large-scale data processing on distributed clusters while Kafka Streams specializes in lightweight client-side processing. This video compares their architectures, showing how Structured Streaming provides better fault tolerance and cluster scaling for enterprise workloads. The choice depends on whether you need serverless scalability or embedded stream processing.

Best streaming engine 2026

The best streaming engine for 2026 is Apache Spark Structured Streaming on Databricks, which combines high-throughput data processing with low-latency results and unified batch-streaming semantics. This video demonstrates how the Databricks platform provides the infrastructure, governance, and tools needed to build reliable streaming applications at enterprise scale. The platform's integration with data governance, ML capabilities, and analytics makes it the ideal choice for modern streaming workloads.

Full transcript

0:00 Introduction to the agenda

All right, good morning guys. I'm super excited to be here to talk to you all about streaming. I'm from the product team and uh here with me I have introd. He's from the engineering team and uh in case you can't tell who's who. Intro also put some very beautiful profile pictures on the slide so you won't mistaken with that. Um it's awesome to see a crowd to to come to the streaming talk. A quick uh brief on the agenda for today.
So I started with a bit of overview uh in terms of like uh what's streaming, what's bash because I think for data bricks especially like you know on the spark and structure streaming engine we have a very unique and expansive view of what streaming means uh for the platform. Then we'll talk a bit about the two different incremental processing engines that enables the different um data engine products at data bricks. and we'll talk a bit about these different product interfaces.
Then uh introduce uh you know some of the exciting innovations and new features we've released into these two different engines. And then finally we'll have a exciting demo um for you guys. So some basic stuff to begin with streaming versus batch. Maybe before I get started can I have a quick show of hands? How many of you guys today already use streaming in your data pipelines in production? So awesome. Quite a lot of you guys.

1:15 Streaming vs. batch concepts

How many of you guys actually use streaming for operational low latency operational use cases like fraud detection, content recommendations? Also quite handful. We have some pretty amazing announcement for you guys uh today and also at the keynote. So it's a pretty savvy audience. So when I say what is streaming then for sure uh it won't be these things. So what if I say like these things? So we're probably getting close to uh you know what streaming is for you guys.
But then really if you think about these different technologies and products what do they have in common? What does streaming really mean in the data space? So often times what I hear people saying they associate or you know streaming with it's continuous it's low latency. While all of these things are true but actually fundamentally I want to talk about the fundamental semantics of stream processing and how we define uh streaming at databicks.
So at its core, the semantics of streaming is that it processes only new data from the source and it's using a checkpointing mechanism to track what data has been processed from the source and it only process new data uh from the previous uh processing run. Now follow that line of definition. So it's amazing to see actually the source of the streaming doesn't have to be message bus system like Kinesis or Kafka or PubSub.
You can also stream from cloud storage like S3 or um you know Google cloud storage. Conceptually you can keep loading new files as they land in the bucket. That's no different from continuously read messages uh from a say Kafka topic. Now the second interesting thing uh that's pretty unique to the structure streaming space is that you can actually run streaming processing semantics with triggered mode. It doesn't have to run 24x7.
You can certainly run continuous but it doesn't have to. If you sort of put time in slow motion for continuous stream processing because it's stateful it keeps tracks of what has been processed. You can actually stop the process at any given point in time and it resume back from the previous state and continue to process new data. So with that you can actually run streaming processing semantics every hour or every day.
So when you combine these two very interesting things that I'm sure many of you guys have heard of or use a connector called autoloader super popular on data bricks it basically use a streaming engine under the hood to continuously or using a triggered mode to ingest new files new pocket files as they land in cloud storage uh systems. Now why do people want to do streaming? There are a few advantages of doing stream processing.
Number one is it's super efficient because it only processes new data. it never reprocess the data that has already been processed. Number two, it can achieve low latency. Well, it doesn't have to be low latency, but if you want to achieve for absolutely low latency for things like operational use cases, real-time workloads, streaming allows you to do that. And number three, it's fall tolerant because it keeps tracks of the checkpoint for what has been processed and it has a state store meaning it's super fall tolerant.
You can, you know, shut down the stream processing for intentional reasons or during some error or crashes. you can bring it back up. It'll resume the process without having you to worry about where I left over last time. Well, then the question becomes like all of these things sounds great like why would I not do streaming? I should do streaming for everything. Like why would I not do that? Well, because there are certain complexity which I will explain a bit.
One is development. So when it comes to streaming uh events can come out of order and because of the semantics of streaming you process the data then you write it out you don't look back. So for handling out of order events particularly if you do like stream to stream joins where you have two concurrent streams they're moving at different pace things can get quite complicated to reason about the logic. Then the second aspects is with regards to like operational.
So all the stream processing happens in state. Um state can get fairly large. The data stream size might be variant. You had to figure out what's the cluster size and scale up and back down as the data volumeing changes many of the streaming workloads and spiky in nature. Then you know when it comes to like non-happy case something went wrong you have to go back in time fix some data things gets even more complex I'll give you a simple example on sort of out of order I think many of you guys probably already know uh it's a very savvy audience um you know pipet okay I'm doing a simple calculation of the average temperatures that I collect from different sensors and I want to aggregate by an hourly window so in the case of stream let's say at the first hour I receive these two rows these two records they're both temperature generated from the first hour.
So it's super simple. I can just do the average you know 24 + 22 divided by two that's 23°. Then let's say at the second hour I receive this. So two of the records of generate from the second hour but one of the record actually late arrival that I receive the data at second hour but actually the data was generating the first hour. So it's easy to calculate the average for the second hour but the question becomes what do I do with this late arrival data because I've already calculated the first hour average and I've emitted results out.
Now in this case you actually you know you can't quite really say you know I can add up 28 and 23 and do an average because you also need the count information from the first hour because you sort of don't know uh you know the count of the data from the first hour that landed on 23. So in this case for you to correct the result you actually have to remember in your coding logic to store the count or sum information from the first average in order to correct the results.
So things gets fairly complicated. Then what about batch like how is batch different? So batch different in a way that the semantics of batch processing is every time you run the process you look at all the data that's currently available in the source. So the results of a batch query always reflect what's currently in the source and your current batch query. Now in reality people don't reprocess everything in the source.
Typically your batch binary is some form of like a partition like uh you know one day's worth of data, one hour worth of data. But as new data arriving that partition, if you run the batch query, it reprocess everything that's currently in the in the in the partition. So same example how would that look like if we do sort of um batch semantics instead same thing for the first hour I receive this it's super straightforward I calculate the average and then in the second hour three records getting added to it so for batch semantics when you run the query at the second hour you actually have information about all the five records so the average calculation becomes very simple in the second hour you'll just correct all the results so as you program the processing logic you don't need to think about like out of order how do I store count and average information you just simply write a average query and group by the hour it'll just correct itself right so just to sort of sum up a little bit some of the you know the differences between stream and batch these semantics I think the pros of stream processing is that it's always efficient it only processes new data and for you to you know for cases where you absolutely need low latency streaming allows you to achieve that absolutely low latency from minutes to even subsecond for real-time operation applications and use cases.

8:11 Understanding the Enzyme engine

But on the downside, you do have to mind be mindful of the complexities that's involved with stream processing. As you write your processing logic, you have to think about and consider for out of order how you correct the results. And that's where you have we have a concept called a watermark which allows you to basically buffer the data for certain period of time before you emit the results. But even with watermark, there could be like super later out of order arrival data.
Then sometimes the data will just be dropped on the floor. So the results won't be 100% accurate. So that's sort of the trade-off with uh with streaming. Then on the other side for batch as sort of we illustrate with this simple example, the processing logic is super simple. Like it's very easy to reason about, super simple for people to write a query about and the results are always accurate. So whatever is currently in the data source, it it's guaranteed that the processing output will reflect everything the current source.
But on the sort of downside, it's not as efficient as streaming because it involves some reprocessing. For the previous example, the first two rows were being reprocessed again and it'll be slower than streaming. So you might be able to do minutes lengthy depending on the data size, but you're not going to do like seconds or sub-second. Um, so these are sort of the the semantic difference and some of the trade-offs. Then enzyme, actually quick show of hand, how many of you guys have heard of Enzyme?
And uh okay so this is a streaming audience that's why you haven't heard of enzyme because enzyme is the engine incremental engine for batch semantics. So what does that mean is as a developer as a user you still write a batch query as if you're doing a group by of the average temperature uh group by an hour enzy incremental engine for the batch processing semantics. So behind the scenes it'll figure out how to incrementally compute that query versus you define the logic yourself.
So in the example that I just gave u before what enzyme will do is that when it emits the first hours average results it'll actually also store the count and sum information in parallel. It's hidden from the query results so that you're not seeing that so that when it receive the second batch of the records and realizes a late arrival it'll use the previously stored count and sum information to update the previous results without recomputing every single records from the first um sort of batch.
So in enzyme is basically a incremental engine that powers batch processing semantics. So what that means is it evates one of the cons in batch processing which is all the data will be reprocessed within the partition boundary that we've defined. It'll figure out how to incrementally processing uh the data whenever possible. So this is sort of the new comparison again like Enzen helps you sort of take advantage of the simplicity of the bash query but at the same time it'll try to make it efficient uh so that you don't reprocess the data um whenever it's possible now this is how these things sort of manifest on the data data bricks data intelligence platform if you're doing data processing data engine workloads under the hood it's all running on top of the Apache Spark and structure streaming uh engines so structure streaming is a streaming processing semantics engine.

11:20 Databricks platform integration

You can use that in continuous mode. You can use that in trigger mode. It powers a lot of our connectors from autoloader to Kinesis, Kafka, Pops, Possar. Then we have Apache Spark which is a batch processing semantic tick. When you do spark read, it reads everything from the source that you've defined. Now on top of the Spock and structure streaming engine, uh we ship the protocol delta live tables at DOT. That's an abstraction uh that simplifies bash and stream processing um on top of on data bricks.
So when you use dot flow is sort of the structure streaming concept essentially wrap up of the structure streaming engine. When you use flow you can write into streaming tables various syncs like a kafka topic and OLTP database that's when you're using the stream processing semantics that's where you have to think about auto order watermark these type of concepts. Then on the other hand uh we have materialized views uh which is the concept for batch processing semantics and under the hood if you use materialized views uh in serverless enzyme engine will kick in.
So you just write a simple batch query the engine will take care of uh the incremental logic behind the scenes. So you don't need to think about it now. So these are sort of the two um sides of uh what we call like fundamentally it's efficient incremental processing. On one hand, we have the streaming engine. You can achieve low latency, but you have to worry and think about the stream processing semantics. On the other hand, we have the enzyme engine.
It does incrementalization for you. It won't be as efficient as a streaming engine, but you know, you get the simplicity of a batch query, but at the same time, it'll be more efficient than as if you to re you know, to do the recomput of the partition uh yourself. Okay. So, now we have these two things. We have streaming uh which is stream processing semantics. We have Enzan engine does incremental computation for batch processing semantics.
When should I use what you have these two things? So obviously for people for you know a handful of you guys who are doing these super low latency operational applications where you need super real time then obviously you have to do streaming. So that's straightforward. But then for the middle one for a lot of you guys who are doing data engine pipeline let's say I'm doing ingestion transformation. um my pipeline has an SLA of maybe minutes or dozens of or maybe hourly SLA.
Now I have these two things. Uh when should I use what? Should I do streaming or should I do enzyme with materialized views? So real simple through sort of the medallion architecture. There are three layers. So these are some general recommendations based on all the patterns I've seen with our customers doing. So let's start with bronze. So the characteristic character characteristics of the bronze layer is that it's typically injection workloads, right?

14:02 Choosing between engines

Typically it's stateless processing. You just grab data from the source with cloud storage with its message bus and ingest the raw data as is into the bronze layer. And typically these are fairly large volume. So you cannot afford like reprocessing a bunch of these data. So at a bronze layer streaming is almost always uh a better choice because you don't get exposed to the complexity of streaming because it's stateless. there's no you don't really care about out of order and you just process a lot of um data raw data from the source you cannot afford for recmp compute and this is why sort of on the databix platform uh you know connectors like autoloader kinesis kafka these are super super popular used with a streaming engine to bring data from outside into the bronze layer then we have the silver layer that's a little bit more complicated it's transformation so many times it involve like lookups uh aggregations and joins then you know the answer sort of depends like if for your workloads if you're using serverless for your workloads the enzyme engine knows how to incrementalizely compute the results you should go with materialized views because it just makes your life so much simpler in terms of how to write the query how to operationalize a query how to handle sort of non-happy path and scenarios but if your pipeline has low latency requirements you need to deliver the final results to the downstream analytics or dashboards within let's say a few minutes then you can opt in for using streaming to do that type of processing but be sure to be mindful of these um streaming complexities up front.
Then finally we have the go layer. So that's the last mile uh aggregation workloads. So typically these are stateful operations but typically the goal layer has a much smaller data size and many times you know the go layer is owned by downstream analytics teams not like you guys or the data teams the platform teams. So typically these folks jobs are doing analytics not handling like the complexity of data engine data pipelines.
So I would say that most of the time materialized views with enzyme engine is a much better choice for the goal. It just avoids a lot of complexities with that efficiency. So that's what I have to say. I want to you know get interested on the on the stage to talk to you guys about the innovations we've been doing in these two incremental engines. Thanks. So if there's one slide summary for the rest of the section of this talk, this is that summary.

16:43 Expressiveness vs. ease of use

Okay. Um I see two attributes that we care about when we build a streaming engine or any engine that can do stream processing. One how expressive it is, how powerful it is and the second one is how easy it is to use it. Now when I think of structured streaming I think of something which is really powerful. You can use it to write any complex business logic. Achieve whatever you can imagine. Okay. It's not for the faint of the heart.
It's easy to use compared to many other systems but not the easiest to use of course. And then materialized views is on the right hand side. Something which is very simple to use. If you know SQL, if you can express your logic in a subset of SQL, we will incrementalize it for you. And that's the distinction between these two things. So as you do your own workloads, you try to figure out which one to use where. This is the slide probably which will help guide you.
Okay, a show of hands. It's summer, so it's probably time to do some vacations. How many of you actually enjoy the planning process of vacation? Uh those who really care about Show your hands. Yes. Those who care about the planning process, they enjoy it. Not that many. Okay. And how many of you just want to get to the destination and then enjoy like have a beer? Lots of you. There are some who are in the middle. I don't know what you guys do.
Maybe you guys plan for it and then your spouses say, "No, that's not what we're doing." Okay, that happens to me for sure. Okay. So I think of streaming as this which is a sporty car. You can have manual transmission. If you are the person who cares about control, if you care about the journey, this is your vehicle. But if you want to go to the office and you don't want to get into traffic, you don't care about driving, you don't want to be cursing others or get cursed, this is basically merchandise views for you.
You just express your destination and you get there. and we do the heavy lifting for you. Let's put a little bit more concrete example here. So when I think of streaming, I think of a stream of data coming in. I may think of like breaking it up into hundreds of streams, writing to many downstream tables, thinking about maybe late arriving data, maybe even doing a site complex algorithm or summarization um using a state store that's streaming and we write all the constructs that are needed.
You have watermarks, you have for each batch, you have state store and so on. On the other hand, let's say you are writing this final nice jazzy little dashboard and you want to just summarize data. You can simply write a create materialized view, select all data, do an aggregate as long as the aggregation is supported by us, it's already incrementalized. That's it. Don't have to do anything than that. So what's new in structured streaming?
So before I can tell you what's new in structured streaming, I need to recap what are the fundamental challenges in streaming because that's what we are trying to fix for you guys. I think there are top five issues in streaming that people have to think about and Ray touched upon them. One is machines can fail. So you need to make sure that you get exactly one semantics. You don't want duplicates, right? Similarly, the second one is data might be coming out of order and you need to handle it.

19:52 Innovations in streaming

The third one is what do you do about your updates? There is some data which has been processed. Should you emit it or should you not emit it? That's a difficult decision. Somebody needs to give you the API to help decide that. Then some of you want really low latency and others want to write really complex business logic. So these are the five challenges we have been trying to address and spark streaming has been the forefront in solving these issues.
If you look at the last 3 years that graph shows our usage has increased three times in three years. That's 3x and these are millions of jobs running each week. The first three challenges that I talked about exactly one semantics u the out of order events and the update mode these are existing features that we have made sure they're available to you guys from the last couple of years. So no changes there. You can use watermarks, you can use append mode, you can use your update mode.
That's all great. What I've been focusing on is these last two things. One, low latency. And so now there's a new trigger called real-time mode which you can use and actually achieve subsecond latency. And I'm going to talk more about it. The second one is state transformations. We recently launched a new API called transform with state. I'm not going to go deep into it because today there is a session which goes into transform with state.
I would strongly encourage all of you to sign up for it. At a very high level, what's happening is we have automated a lot of things that were to be done manually. For example, handling triggers, uh timers, handling how state is clean up. All of that is pretty much automated with this new API. So if you are using state APIs, this is the talk that you should go to. It's there. Look at your calendar today. Okay, let's talk about real-time mode.
This is the one I'm really excited about. We have been hearing from clients on and on. Hey, they want subsecond latency. Their banks who come to us say they want to do fraud detection. It's millions of dollars per month which is at stake and urgency is the critical thing. And we're not talking about minutes. We're not talking about seconds. We're really talking about tens to hundreds of milliseconds. This is not everybody's use case.
This but this is use case for some folks which is but critical for them. So if you look at the place where workloads reside today, if you had the latency spectrum, most of you are familiar with the right hand side which is spark streaming is really good at ingestion and ETL and it does it like minutes, tens of minutes and even like hours depending on the workload. So it's really good at that setting. What we have been focusing on is now on the left hand side which is if you have an operational use case now we can do it.
You should try out realtime mode it will give you sub-second latency and not just sub-second latency consistent subsecond latency. That's the critical part that's harder to achieve. So what are the key ideas? Um our team ended up doing lots of innovation here. I just want to give you intuition around two areas. One, you can think of it whenever we do batch processing. Whenever you took a batch of data, we want to process it.
There's a fixed overhead associated with it. So it's obvious that when you try to do record byreord processing, you want to really reduce that overhead. So that's one of the innovations we did. The second one is when a data comes in any record that comes in we want to make sure it doesn't wait at any stage neither in an operator nor in a pipeline stage and for that we had to do many changes. We have an enhanced scheduleuler.
We have created a new streaming shuffle and we had to actually change most of our operators. They are more streaming in nature. They are long lived. They do not wait anymore. As a result, you can get to hundreds of milliseconds very easily. One of the big questions is what's the programming model? Do you now need to learn something new? The answer is no. It's the same old API. The only thing you need to change is change the trigger.
You just have to say realtime trigger. That's it. And then everything should just work. We are in private preview. So you can we actually in public preview. So we are in gated public preview and you can try it out for production as well today. So if you are interested just talk to us or send us an email. Come talk to us after this session is over and we'll be happy to help you out. But what what's more is open source is in our DNA and you'll hear more about it in the coming sessions.
We are going to open source realtime mode itself. We have interest from lots of contributors in the open source community who are interested in this feature but also want to contribute to it. So therefore we are open sourcing it. Okay. I told a lot about streaming. I do want to tell you a little bit about materialized views as well. To appreciate what we do in materialized view, you need to think about what the problem really is.

25:40 Innovations in materialized views

You end up writing an arbitrary SQL statement. You're just giving your intent and somebody somehow is magically incrementalizing it. That's a really hard problem. That's a really hard database problem. Most of the products out there external to data bricks maybe solve a little piece of there. There are hardly any products by the way and the those who do they solve this much area in that space. We try to do larger set of patterns that we can synchronize.
And our goal, our team's goal has been can we make it bigger and bigger in terms of the patterns that we can support so that you can write any workload in this materialized view concept and we incrementize it for you. Here's an example very simple example. You might be joining a customer data with a orders table. And what you want is maybe an aggregation or sum of total amount of customer sales. Your customer table might change, your orders state table might change, but you don't do any logic around it.
You're not handling anything. You just write the specification of the materialized view and it is getting incrementally refreshed. That's the power of using metalized view. We already support a lot of constructs here. You can use group by joins unions and many others. What we are now focusing on in enhancing that set of pattern which can be autominalized. For example, you can now write nested aggregates. You can add expectations.
We have a new cost model which can figure out what is the right technique which is the most efficient in incrementalizing this particular MV. We're also making changes to the UI. If you use the new lakeflow UI, you'll start seeing suggestions which say, hey, maybe switch on your deletion vectors or row ids and this will automatically incrementalize the materialized view. Moreover, we also now have a standard cluster which can basically make sure that your TCO is lower.
Okay, this is all theory. Theory everything works right. So the good thing is can we do a demo around it and this might be the thing which helps you see how to use streaming as well as multilize view. I'm going to use an example which you can download directly from DB demos. So just after this meeting if you want you can try it out. Just install DB demos you'll be able to see it. Okay let's see. Okay. Yeah. So you can import this uh DB demos and try it out.

27:59 Streaming and DLT demo

The use case is about just analyzing a set of loans. So there are three different layers here. The first layer is the bronze layer which basically includes the autoloader. I'll talk about it. The second layer is basically where we'll try to clean up our data, look at maybe expectations. And the final layer is the layer from which you can create a dashboard. We are using DT for this use cases. You can use DT. You can also use structure streaming directly for streaming tables.
Either one is fine. Main goal is all of this is going to be real time. So whatever you're seeing is basically reading the data and then projecting. So there are three kinds of tables here. The first one is a loans table which gets updated every few minutes. And then let's say the third one is a historical data which gets updated every few week. The bronze layer as I was saying you can just use autoloader to create a streaming table.
Uh but before we do that given that this is a set of queries we'll just go to the DT UI. Um it should show up right now. Okay, we'll go to the DT UI and then we're going to just run this because it's going to take some time to load and then populate all the tables. So once you have the DT UI, you can just call start. That should just start the process. It will try to acquire resources. Then it will do an analysis of the graph.
While it does that, we can go and look at the actual tables and I'll talk you through those tables. Okay, so let's look at the bronze layer. So this layer we'll use a streaming table. We need to use streaming otherwise you will load all the data in one go and our whole hope is that we are going to incrementally process the data which is going to be cheaper. So the first table is a streaming table where you have transactions.
So you just have to say select from a stream. In this case you just give the uh destination the input location. The second one is a metadata table. I'll later show you the metadata columns and the values in there. We'll use it to enrich our tables. The third one is the historical transaction which remember I told you it gets updated only once a week. Therefore, you can put a property saying that it should just trigger maybe once every six hours because there's really no data before that.
Okay. Before we do the silver layer, we can just go and see where the pipeline is. The pipeline is still running. So we can go back and look at the silver layer. In the silver layer, this is the cool thing. You can write expectations. Basically, you can put assertions on the data quality. The first one is a streaming table where we are going to do a join to enrich it. Here we're going to add one column which is the metadata column.
I'm going to show you details of it later. And this one is called the accounting treatment column. So just watch out for that column when I show you the tables later on. The second one the constraint that this expectation is really cool because you can actually say that I want this table to have only data which has positive balance. So you can put data constraints but more than that you can actually quarantine your data.
You can look at data which is bad data and put it in a table of its own. And here this is the quarantine bad transactions table. Finally, you can create a materialized view which can have this enriched but good data. Okay, let's go and look at the UI. Everything has completed. So we can now dig deeper, drill down and see how the data looks like. So here is one of those examples which has bad data. So this is the quarantine data.
So you see balances one of the balance columns should have less than equal to zero value. On the other hand the data that was good the table which where we had removed all the bad data here all the balances are positive. And in fact if you look at the last column you'll see that there is an extra column which is called the accounting treatment. This has the data that we used to enrich. And here is the table which has the accountant treatment information.
Okay. So let's now look at the gold layer. Oh actually we can look at this materialized view also. This is pretty much same as the other tables. It is more enriched. It has lots of columns. I'm not going to go through all of them here. So you're just looking at a subset of these columns. Finally let's look at the gold layer. This is the last set of tables which we are going to use to create our dashboards. So here's a metalized view where you can do aggregations, you can do unions, um you can do joins if needed.
We are not doing a join here though. We can create aggregates which are based on the cost center. We can create aggregates which are based on the country. And we can now look at the materialized views which have these values. So if you go back to the UI, there are two materialized views over here uh which have these aggregates around the country. This one is the one with the country code and you can see the two columns.
What would be even more interesting is creating a dashboard, right? So we can just click on the dashboard from the same UI, click on create dashboard, go to the data source and you can just type the name of the materialized view. So just copy the name new loan balances by country, paste it in the UI. The UI will figure out the data set for you and you can just load it. Finally, you can create a visualization. Very easy.
Drag and drop. Click on it and we can use assistant as well. So if you don't know what you're going to draw, you can just mention it. So here I'm going to say plot by country code and you'll see I have a typo. Actually I wrote county code and assistant still figures out this is actually country code and creates a visualization. Okay, that's it from the demo side. So hopefully this gives you a flavor of where you can use streaming and where you can use merchandise views.
All of these demos are available to you guys. So feel free to ask us questions. Ray and I are here.

Learn more about the Databricks Data and AI platform.