Spark Real-Time Mode: Sub-Second Train Telemetry at Scale

Summary

Dutch Railways processes 2.6 billion sensor data points per hour from 10,000 sensors per train using Spark Real-Time Mode on the Databricks Data and AI platform, achieving 215 milliseconds of end-to-end processing latency on complex workloads.
A single-line trigger change converts existing analytics pipelines to real-time, and adopting the Protobuff binary protocol reduced network overhead by 90% compared to prior approaches.
Native AMQP and MQTT connectors link operational systems directly to Databricks, enabling a unified medallion architecture with PySpark and SQL for both batch and real-time workloads across one of Europe's densest rail networks.

Spark Real-Time Mode: Sub-Second Train Telemetry at Scale

Dutch Railways processes 2.6 billion sensor data points per hour from 10,000 sensors per train using a unified Databricks platform. This talk showcases how Spark Real-Time Mode enables sub-second end-to-end latency for operational monitoring, predictive maintenance, and real-time alerting on one of Europe's densest rail networks.
Learn how a medallion architecture with unified PySpark and SQL frameworks replaced siloed pipelines, how a single-line trigger change converted analytics pipelines to real-time, how Protobuff reduced network overhead by 90%, and how native AMQP and MQTT connectors now connect operational systems directly to Databricks. See live performance results demonstrating 215 milliseconds of processing latency on complex workloads, and explore the emerging paradigm where operational systems and data platforms converge.
🤝

Chapters

FAQs

What is Spark Real-Time Mode and how does it differ from standard streaming?

Spark Real-Time Mode enables continuous processing with sub-second end-to-end latency, unlike traditional micro-batch streaming. In this video, Dutch Railways achieved 215 milliseconds of processing latency on complex workloads by switching from a standard analytics pipeline trigger to the real-time trigger with a single line of code change.

How does Dutch Railways use Databricks for train monitoring?

Dutch Railways processes 2.6 billion sensor data points per hour from 10,000 sensors per train using a unified Databricks Data and AI platform. The platform supports operational monitoring, predictive maintenance, and real-time alerting across one of Europe's densest rail networks, which operates around 400 trains simultaneously and serves over 1.1 million passengers daily.

How did Dutch Railways reduce network overhead with Protobuff?

By adopting the Protobuff binary serialization protocol as the data wire format, Dutch Railways reduced network overhead by 90% compared to prior approaches. This optimization was a key step in meeting the sub-2-second latency requirement for real-time train telemetry.

What native connectors does Databricks provide for IoT and operational systems?

This video describes native AMQP and MQTT connectors that allow operational systems to connect directly to Databricks without additional middleware. These connectors were part of Dutch Railways' solution to build a unified medallion architecture combining PySpark and SQL for both real-time and batch workloads.

Full transcript

[00:09] So I'm about writer. I am uh from Keemini. I'm a data architectctor over there. I've been working uh over the past three years with the Dutch railways. One of my other accounts is POS in which I do a lot of things which is a Dutch uh mail service. Um I'm working on data platforms, data
[00:24] ingestion and I have a special focus on uh real time analytics of which we are telling you today and I'm presenting today with Anand Pingle. Hey hello everyone. I'm Anant. I work for data bricks as a specialist solutions architect and I help customers mainly on data engineering and warehousing topics and a lot recently
[00:42] with real-time streaming. So excited to be here. Back to you all. So we have to tell you about the Netherlands and about the Dutch railways because we are presenting here in the states and trains in the states are vastly different than how we uh go about that in the
[00:59] Netherlands. So let's first put things in perspective. The Netherlands is not really a large country in at all. Like if you have the stretch between San Francisco and Los Angeles, we can fit the Netherlands in between there. And as far as I know, there are not a whole lot
[01:14] of trains running in that particular part in the States. Within the Netherlands, we operate about 400 trains simultaneously at every point during the day. 400 trains. It's quite a lot. On that stretch that I pointed out over there on the left hand, that's the the
[01:29] the road I take from Amsterdam to Antoven a couple of times. That stretch I can travel that every 10 minutes. So I don't look at when my train is leaving. I just go to the station and take a train. So this also means that it is one of the densest and
[01:45] busiest uh networks that we have in Europe, maybe even in the world. We've uh we have over one 1.1 million people taking a trip every day and we serve over 350 stations nationwide, almost
[02:00] 400. But yeah, it was not this looked nice on the on the slides, you know. Um you also see like a map on the the the lines that we operate and you see how dense it is, how how much space is being occupied by just the lines that are there and this is all trains going
[02:17] anywhere and everywhere. So we have some challenges to also keep this punctual and make sure that every train gets gets in the station on time and gets back on time. But we are not only operating trains at the Dutch railways. It's much more than
[02:32] that. It's quite a complex company because besides running trains, we also own uh the stations themselves. So we is a real estate company. We have the shops which are in the stations are operated by the Dutch railway. So it's also like a commercial company. The scheduling to
[02:50] keep the hill the entire network running and make sure that the trains are not like being delayed because they run into each other is also done by the same company. the management of the fleet like repairs and stuff like that. We're huge centers all over the Netherlands to make sure that we can do the repairs
[03:06] ourselves and of course we are we are still Dutch of course. So we also rent out bikes. We do need to rent out bikes otherwise yeah still Dutch you know. So to put that also a bit in in in into a perspective like I made these pictures
[03:22] last week. This is at west central. It's a bit below Amsterdam in the greater Amsterdam area as an American would say, but it's one of the larger stations that we have. And these are the bike sheds. More precisely, these are the rental bikes that we have over there. And it's, as you can see, it's quite a lot that we
[03:38] that we went. All these bikes are gone on every weekday at 10:00 a.m. At 10:00 a.m., these are gone. And they are back by 12. And it's not for tourists. It's not for like people visiting. It it's all for for people that are commuting from one place to the other. in a last
[03:54] mile distance as we call it is being done by these these bikes. So back to the problem at hand back to data because we are talking today about trains and the train telemetry that's coming off those systems. A train can be in our
[04:09] let's in this room we can call it a large IoT device. If I tell this in the Netherlands at the at the company itself, then I probably have an issue. But for now, for this room, we will call it an IoT device and quite a large IoT device because it has approximately 10,000 sensors. We have different trains
[04:25] operating within the Netherlands with approximately every trains has about 10,000 sensors on board. Every sensor, every value of that sensor is being sent to the Dutch railways at every second. So quick quick maths here, but our most
[04:41] modern train has 16,000 sensors. We operate 46 of them at any point during the day. This means that per hour we have 2.6 billion data points coming in. 2.6 billion. That's quite a lot. For the more common train, which is a bit older,
[04:56] less sensors, 8,000 of them, 195 trains approximately operating every day means 5.6 billion data points that we need to handle. So you can see that this is quite a large problem that we have or at least challenge I should say. So
[05:15] when we started with this we had we have kind of kind of a problem because every train that you see over here is running through the Netherlands and it's a bit different. It's built by a different manufacturer. It has different sensors. It have different uh common common things that we need to adapt for. It's
[05:31] not a lot, but it's enough that that it had each train had its own code base, which also meant it had its own compute to run on and also meant it had its own output. By not syncing all of these, the the team actually grinded to a halt because at some point you're only
[05:47] running operations on the code that you already have and you're not innovating anymore. So, we had to find a way to actually handle this. So with data bricks and the philosophy of a medallion architecture and a way that we need to think about data, we went from war
[06:04] sensors to actually a data product and we went from building single pipelines into an architecture in which we have common things that we can share across pipelines such that everything that we need to design new only takes us a bit of work and not every time all the work.
[06:23] So how do we do that? And again this is about real time mode but before you can get fast you need to have a proper plan in place. So this is essentially our plan on the bottom we have data bricks like like the framework logic that data bricks provides spark for in that sense
[06:38] and spark does a lot of stuff for us like uh scaling making sure uh hard things like uh planning are being done for us uh isolation of tasks all that kind of nice things is all being handled by spark already. So let's use that.
[06:55] It's very it's also very open. We can still it's still open source. We can run different formats in it. We can do different things with it. On top of that, not everything is handled by Spark. So the Dutch railways have a specific way of handling for example
[07:10] logging for example for quality metrics that we want to to get out of there. For this you can build a platform a framework that's working for every and all of your pipelines. So nothing has nothing to do with trains. you can take that that framework logic uh platform
[07:25] logic and put it in another company with the same with the same agreements. Then lastly there is business logic and this is the layer in which we actually talk about trains and this is the layer that we actually do something. But suddenly we have a different format
[07:42] because in this sense we only need to do the top layer for every time we develop a new pipeline instead of doing both getting to know the framework doing some platform logic on it and then still doing the business logic. What's important when designing such a
[07:59] framework is that you keep your developers happy and developers are happy when they have freedom. So from the framework to the platform logic, we use the dataf frame API. The dataf frame API is a core on which spark runs and the platform logic uses the dataf frame API to actually extend on that.
[08:16] Secondly, when going to the business logic, we actually give our developers the freedom to just use SparkSQL or even use Pispark in that manner because they have access to PISPAR and and SQL they are still free to do whatever they want. They there's still a lot of freedom in
[08:32] there. This is important because as soon as we get a new developer, we don't want them to teach us our framework, our our way of doing stuff because that would mean that we don't want a good data engineer. We want a good person when of of knowing NS of knowing the Dutch
[08:49] railways. We don't want that. We want good data engineers and we want to put them in their strength. So when we apply this to our issue that we currently have then we have the trains and they are streaming into a Webbit MQ server on AMQP server to be
[09:05] more more precise. And then we have a connector which is in the in the framework logic that is applicable to the entirety of data bricks. Then we have some general decoding and transformations. Our logging how we we handle certain things. On top of that some business logic is
[09:20] being applied and from that point on is being put into a file and this is actually our pipelines. This can be seen as something like spark the collaborative pipelines. We don't use it yet but we might be needing to look into that. And when we combine that file with
[09:37] some environment parameters. So for example for dev acceptance production we can actually start running that and essentially at the end there is a delta connector to put it all in unity catalog. So the combination of a file and the environment variables together is runnable meaning that we don't actually
[09:56] every time create our own run logic that's being borrowed from the platform itself. So from the blue layer and then we use microbatch streaming to actually get analytical data product. So now we have a nice medallion architecture. We have bronze, silver,
[10:11] gold. We have it analytically stored. We have like the tables and we can run analytics on this and even that even that already gave a huge performance difference because we were way more focused on the data products at hand. Instead of developing each product sing as a single instance, we were able to
[10:28] get huge uh u uh pro uh performance improvements. Cost went down by by 75%. Storage was only a fifth of what it once was and especially the access time was drastically reduced because we had a much cleaner way of saving the data.
[10:47] This means that we do not only save cost within our own compute but we also save cost in storage in storage things that we uh that we need to store. So the delta tables itself is smaller and downstream our developers are also happy because the the data product is much
[11:02] more accessible. It means also that it's le less effort to use meaning they save on compute as well. So this was already people were very happy when we presented this. This was about half a year ago. But if you do good things then the business always wants more. So this was
[11:20] the moment they came to us and said like hey we really like what you do analytically. Can you do it also real time? Can you do it for our operation? Small thing. If you want us to be happy about that you need to get it under 2,000 milliseconds. So 2 seconds of end
[11:38] to end latency. That means that when the train produces the the the data points, we need to have it in a dashboard within two seconds. And for a year ago, we would be very nervous about that because Spark would not be able to handle that or would be very difficult for Spark to handle that.
[11:55] Currently with real-time modes, this has suddenly become feasible. What's important that we first ask ourselves from why do we need this? And this comes back to the congested network that the that the Netherlands the railway roads of the Netherlands are. So if a train
[12:12] for example wants to leave or depart from the station and the door is not closing then most of the time the train conductor needs to apply some fixes. It calls train traffic control and says like my door is not closing. Yes, you have to turn a button and before the button is found or the good button is
[12:28] the correct button is found. we are 5 minutes on the way but in 10 minutes the next train needs to depart already maybe. So this means that we actually have a problem when this happens with the current way if we have it real time the call to the train traffic control also means the train traffic control can
[12:45] directly look into the train. It can directly see what's going on in that train. So the train conductor now pushes the button and the expert in the in the control room says no no no not that button the one next to it. This saves of course a huge amount of time and a huge
[13:01] amount of debugging. Another thing that we want to do, we know a lot about predictive maintenance. We know when trains are going to break down. So that means that we put them in maintenance before they break down on the tracks because if they break down on the tracks, we have an issue. Uh
[13:16] personnel can't move from A to B. Passengers are not move going from A to B and will be angry. We need emergency repairs. We need to tow the train away. So we really don't want this to happen. This is like like tens of thousands of euros every time this happens. So
[13:33] we already prevented a lot but some things you cannot prevent like for example when uh when it has frozen over overnight. Then we have on the on the overhead wire and the connector to the overhead wire there's some ice ice particles forming. When that that
[13:49] happens we see in the sensor data that the connection between those things are not correct. And if that happens for too long, the connector will cut the overhead wire. Now you can imagine if the overhead wire is cut, then we have a repair to do for like 3 4 hours. Very
[14:05] costly, very inefficient. But since we now have live access to the sensors, we want live access to the sensors. We can scan for that pattern, alert train traffic control that it is happening. They call the train driver and they say like, "Hey, can you put up your other
[14:21] connector?" And suddenly we go from a $50,000 repair and angry customers to no impact at all. So this is huge if we can can achieve this. Now I wouldn't be here if we didn't achieve it. So let's be honest, probably we will be fine, you
[14:36] know. Um and we will be fine quite soon because we have this entire framework in place and maybe you you ask yourself like why is he going on and going on about this framework? But this is the moment that that framework is going to
[14:52] work for us because the only thing we actually had to change was one line of code. We changed the trigger from microbatching mode to realtime mode and that was it.
[15:08] Yeah. So quite a lot of ramp up for this of course. So so take it in while you can but one line of code change complete completely owned by data engineers by the way. So this is end to end run on SparkSQL meaning that where you normally would have like a software engineer to do the
[15:24] first things or build something and the software engineer would leave because the most important thing are are done data still changes and you get that that that that une
[15:39] that uh operations is running behind because you are not properly maintaining your product all gone here. So that's really awesome to see. So this was a positive story. Of course we had still some issues. So so let's go to those. Um because although it was
[15:55] working, we had like a 30 megaby per second issue. Uh it was uh namely the case that uh we exported everything in JSON and you can see like a sensor data like a true false value or single integer is only a couple bits of
[16:11] information. However, the name of that sensor is like 20 30 uh characters long and we all encode that in a very inefficient format. So essentially if we if we had one consumer on our output uh our output
[16:29] network was going by increasing by 30 MB per second and the the client that's actually using it also needed to have a pretty good internet connection. So this is not scalable. Normally you wouldn't have that because delta and park which is on the line that
[16:45] compresses all the data hugely so that there is no problem in that but suddenly when we export everything in a singular manner this becomes an issue so we fix this by applying protobuff and protobuff is a binary protocol in which you share
[17:00] much like unity catalog shows you like how how your table is being set up you have a schema file which you share pre-share with all your customers you just publish it somewhere and say like this is a schema we're going to use and certainly we can export it not as an
[17:16] inefficient JSON but as very efficient binary data not readable for the human eye but is very efficient for computers also to actually use this data and suddenly the same data fits in 3.15 megabytes instead of 30 megabytes a reduction of 90% and suddenly we can put
[17:34] 10 customers in the same network space that we normally would have like with one customer vastly difference. So if we talk about results then we need to explain how we actually look at these results. How we look at realtime streaming results. So the first m
[17:50] latency is actually the latency of the the the event hitting our system hitting the MQP server. Meaning that there's a 4G connection going on. The train might be out of out of bounds. uh and our most uh uh modern train so the ICNG has about
[18:06] 760 milliseconds of latency for 95% of the messages. So the P95 means 95% of the messages will be at the AMQP server within 760 milliseconds. Given that we had that target of 2 seconds and this is
[18:23] already quite a lot of budget you know the processing latency the P95 this is what data bricks add. So we need to sort that file we need to apply bronze silver and then then export it again to Kafka. This all happens uh for 95% of the
[18:39] messages for our most complex system in 215 milliseconds. 50% of the messages even appears within 160 milliseconds. So imagine like that 2.6 billion records that we were data points that we were talking about all being processed within
[18:55] 250 milliseconds. That's huge because it also means that overall we we we made our target of 2,000 milliseconds by a long shot because we we we only spend 10% of the budget within data bricks. Suddenly
[19:11] operations is really feasible with data bricks. The only thing that was now happening is that we had an analytical pipeline which is doing some complex work and a real-time pipeline which is doing some complex work. By the way, we all run this on single node clusters which are not really big. So costwise this is
[19:28] already quite quite low but we can even get this lower and less complex because what we did is we we we are going to offboard our entire analytical pipeline. We just run the fast track. So it's running it through realtime mode and from the Kafka server we don't run it
[19:46] like continuously but every 10 10 minutes we like grab everything which is in Kafka and put it in delta which is less commits on your delta table less optimizing on your delta table higher performance less compute that you need to run you can run it all on serless if
[20:02] you want. So this is also like a huge thing to to acknowledge and this is all nice and I could put it on slides and in graphs and in everything that that I just showed you and you have to believe me on my words. So some of the people in this this room I know but most of you I
[20:19] don't. So maybe I should just show you what that is actually working. So um this is hard with a mouse. Now let me get out of the slideshow and into some some actual things. So this is the
[20:34] font that we set up just for this class. We we vibe coded this as if things don't look really even like my vibe coding skills not not like actual things. So when we look at this this is the Netherlands and you see these dots.
[20:49] These dots are the trains that are currently and one is in the in the ocean. So probably there was a sensor broken there. Uh but if you look correctly and we will zoom in in a bit. All these trains are currently live reporting data. We are now having a live
[21:06] look into the train network of the Netherlands. This is actually happening as we look at it. So if we zoom in on Amsterdam for example uh where a lot of trains are standing still and and and on in parking mode, you actually see them moving.
[21:22] you can stand on a Dutch station, open this app and actually see happening what's happening over here. So let's let's dive into that a bit. So let's look at what this train is actually doing. So this is one of our most modern trains. It is now reporting uh um so
[21:40] this is the train. We have a nice picture of it and here you see the processing latency. you have 1 140 milliseconds of uh first more latency uh processing latency of 300 to 500 milliseconds and note that this since we are now in San Francisco which is quite
[21:56] a long way from Amsterdam there are some hicks in that because we have a lot more latency on the network itself and we have some front end latency um so when we go down we don't only have like that information because this
[22:11] information is like yeah this is nice for the viewer and nice for you all. But we also have this information which is much more important. And these are all the sensors that are being reported and processed every speak. And I can tell you about this, but I won't reach the bottom in this page. So let's go a bit quicker.
[22:32] So yeah, this this will take a while by the way. So and note that every value that you see is live updated like like it's not that that we that we that we skim some things, but this is all live updated. Am I always at the bottom? I don't know. No, let's go a bit faster. And we are at the bottom. So these are 16,000 sensors
[22:49] all being updated in a live manner. It I think this is pretty awesome that we can do this now with with end to end data bricks. So to make this actually I need to get back as well. You know that's takes some time.
[23:06] Um, normally I do this with a keyboard, but my laptop is closed, so that takes a bit longer. And then um, if we go back then, then we can also see
[23:22] like a more live overview of the trains itself. So these are all the trains and when you see a red dot blinking that means that that train is reporting data at that point in time to this dashboard. So you see it like a prism tree and every time a red dot appears something is happening on that train. So I think
[23:39] this is quite awesome and it enables all the use cases that we were just talking about. And to show you that it is actually live here on the live thing. Get back down again. Normally this doesn't take so long. Sorry guys.
[24:01] And what I want to show you is that that like with some of these values that they are actually changed. So we now scrolling to the temperature values which are all the way down unfortunately. Uh and we will see that some of these values are actually changing if we are lucky. So if we look at the the V vap
[24:16] VAPC things and sometimes you see a value like slightly changing over time and you also see like it's now about 20° in the Netherlands and that's also comes uh 20° of Celsius and you see over there 20 22 and imagine the point 77°
[24:34] which it actually is. So this is all happening live and I hope that this also shows the vast amount of data that we can process with virtually no compute at all. Um and with that uh I hand over to Anant which uh is going to tell who's
[24:50] going to tell you a bit about how this is happening in the background. So um you need to use the try the clicker once. Oh, it works now. Why does it work for you and not for me? That's that's so that's so unfair.
[25:08] So yeah, thanks W. So I had the pleasure of collaborating with the Dutch railways on this uh project from the data brick side. So uh I'm going to talk to you a little bit about real time mode and the tech that actually uh powers what you just saw. Um so so what exactly is real
[25:25] time mode right? So it's essentially uh low latency processing delivered inside spark structure streaming. So the way Spark structure streaming traditionally works is in a microbatch mode. So you have microbatches which
[25:40] have fixed offsets of data that they're processing and these micro batches can be as short as 500 milliseconds. But as you can imagine this is great for throughput but when it comes to latency that's not great because when you're processing one microbatch let's say new
[25:55] data arrives at this point you can only process it in the next micro batch. So that was like an issue with spark. So um or not an issue but rather where spark was lacking right. So if I show you this um slide with the latency
[26:11] spectrum. So when it came to use cases like streaming ETL where we need seconds to uh minutes kind of latencies spark was dominating. It's really great for high throughput analytical workloads. But uh when it comes to these ultra low
[26:26] latency use cases, so all the ones that require an end toend subsecond latency. So let's say yeah cases like these where it's operational monitoring, alerting or fraud detection where you need an action taken on the data as soon as it arrives
[26:41] within or less than second, Spark was not able to do it. So um what our engineering team did was um yeah they they created real-time mode um is an architecture change of spark but for the end user like vault mentioned
[26:58] it's just one line of change you switch the trigger type from u uh processing time trigger to a real-time trigger and you have uh real-time streaming enabled um within spark um I'm not going to talk more a lot
[27:13] about the architecture But just more on like what kind of use cases we see and what you can achieve with this. Uh traditionally before we had realtime mode um we needed two different engines typically. So one for your um let's say
[27:30] seconds to minutes latency kind of use cases and the other one for your ultra low latency use cases and customers typically used a second platform like flink and um this meant this this meant you had two different stacks to operate two different engineering teams two
[27:46] different expertise in the team and um yeah so there was a real need for this and luckily now with real-time mode uh which is also by the way uh open sourced in Apache Spark you can have it all in one platform and with uh with spark end
[28:01] to end so you can cover all kinds of uh low latency use cases uh with with spark. I'm going to quickly talk about what kind of applications and use cases we usually see in the field that customers are now taking over with realtime mode. So they are more let's
[28:18] say a lot of use cases are are fraud detection for example. So you say you have new transactions coming in. You need to on the fly calculate new features score using your model and then take a decision based on that. So should I block this transaction or should I not
[28:33] block this transaction? Uh we have some personalized offers use cases. So for example on an e-commerce website you have a user whose cart was full. Now they are suddenly uh removing stuff from the item. They don't want it and we want them to um yeah buy it. So we have to
[28:51] send them personalized offers. So these kinds of use cases are also typically seen uh here. Another use cases are the set of IoT use cases. So you have uh monitoring, operational monitoring, operational alerting. We also see these kind of use cases. And yeah, the it
[29:08] unlocks a lot of different use cases now that you couldn't previously do with Spark. Um like I mentioned um we with real-time mode it is essentially the same spark API. So it's the same data frames that you're used to the same transformations
[29:25] the same stateless stateful aggregations uh stateless transformations stateful aggregations that you do it's just a switch in the real time uh so it's just switch in the trigger mode. So yeah what RTM is essentially giving is yeah
[29:41] continuous processing. So this is the architecture change we get low uh ultra low latency processing. So we have seen so when we did the P already we got like latencies almost around like 60 milliseconds P99 which was was great.
[29:56] I've done a lot of other use cases with customers where we have sub 30 millisecond end to end latencies. Um so it is fast. Uh it also scales. we've done uh this is already 10,000 sensors per train that comes so 10 billion data
[30:13] points per hour but yeah we we can support the scale that spark usually does um it's a easy change uh oneline change and it's also open source and this is what uh NS uses for the use case that W discussed as well
[30:30] uh a bit about realtime mode it is since March it is generally available on uh on data bricks. So runtime 16.4 and above support python and scala it works with uh classic compute and um yeah as of
[30:47] actually this week so early this week you can also now use it with spark declarative pipelines. So we have a lot of customers uh that are uh standardizing on declarative pipelines and you can also do it uh with serverless declarative pipelines uh as
[31:02] of uh early this week. It also supports some things that we don't support in classic compute for example um autoscaling. So this is also something that's supported with the declarative pipeline. So please uh give it a try whenever you like. Uh a bit on
[31:20] what's coming. So while we are now already ready and able to do a lot of the use cases that you would typically need. Uh we have a nice road map. So we see uh lake base being like a low latency store that a lot of customers need. So we have a native sync that has
[31:37] gone public preview. So you can use that. We have some rolling window aggregate. So these are like hey in the last 15 minutes I want to count something or sum something up. So these are typically what you need for real-time feature engineering. So this is also in preview right now. Um
[31:53] declarative pipelines also in preview and we're working on streamtoream join support. So this is something that's going to come soon. It's it's not yet there. But um yeah, a lot of times customers can also go around with stream and static joins which are also supported.
[32:10] Uh I think you will all get the the slide uh later on but we have some blogs, we have some reference docs and uh some solution accelerators available. So you can just spin them up on your workspace and u and go forward with it.
[32:26] and please uh give it a try. Uh I will now hand it over back to Vout cuz he has something also more exciting to share. Yes. So first one one more more thing because almost glimpses over it but the fact
[32:42] that this is open source is huge. The fact that they now support a native sync into lakebase, I've already had the pleasure of trying it out means that you can have aggregate data with less than 200 milliseconds of latency. This changes radically changes the paradigm
[32:59] that we are currently having. So normally you will have your operational system then you have your some software engineering and then you will have an app or a front end. So with this possibility that paradigm changed because where normally data
[33:15] would be sort of a side product we now have like the operation immediately database or your real-time engine and then your app. So is and and with that also comes all the nice things of
[33:31] the governance that unity catalog for example or lakebase provides all the governance that you have over there. So if you think about that where now the data world and operational world are often very separated but because people want it to be this is
[33:47] going to change in the next year in the next two years what's important for us is that we then also support the language of the operational systems because if I go to my train builders and I say hey I really want this to be to implement this
[34:03] operationally but then you have to to start speaking Spark to me uh speak uh Kafka to me then they say like yeah this this thing is not speaking spark and we are not going to change it because it's highly regulated. So we need to tailor to the operation.
[34:20] We need to tailor to ensure that we can manage this and then essentially we can run it into lakebase which is our old b database and have it in a database and from there on we can build apps on it but we can also put it on a websocket
[34:35] which I just showed you is running into a a website. So these connectors are not there yet but to enable this use case we did create them. This means that there is native support now within real time
[34:51] mode for both AMQP the MQTT protocol and some protoraph extensions that we had to write to make it sure that your your schema evolution and stuff like that and your schema generation all work seamlessly within Spark.
[35:07] This is opening the world of real time mode not just for the data department but we are actually being able to run against every operational system because if we can't run against operational system we just put in another connector we already built SQS for another
[35:24] customer we have at Gemini and if we find out there is another protocol then we will put cloud to work and we will have another one this is in preview the only the AMQB connector is already there in its first version MQTT and extended protobuff will
[35:40] be jo will be uploaded this week and after that in a few months we will announce this to the public. So this is actually a preview for only the people that are now still in the room. So unfortunately for the people just leaving um but try it out see if it works for
[35:56] you guys. Um and with that we uh we would sincerely want to thank you for for listening to us and uh letting us keep you take you through the story of of NS Dutch railways. So thank you very much very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.