Sponsored by: Solace | From Lakehouse to Event House: Making Databricks Act on Data in Real Time

Summary

The core problem Solace addresses is that most organizations have only the 'decide' component of the sense-decide-act loop working well on Databricks; getting real-time events from operational systems into the platform and routing model decisions back out before a transaction clears are the harder challenges.
Solace's Event Mesh pairs with the Databricks Data and AI platform to create an Event House—a bi-directional, always-on data flow that routes fraud flags, model scores, and operational decisions back to payment platforms, logistics systems, and edge devices in real time.
This video covers three production use cases—real-time fraud detection with immediate payment decisions, airline flight delay prediction before cascading cancellations, and predictive maintenance before equipment failure—with deployments at United Airlines and CIBC.

Chapters

FAQs

What problem does Solace solve in the context of Databricks deployments?

Databricks excels at the decide part of the sense-decide-act loop—running models, analyzing data, and generating insights. The harder problems are getting real-time events from payment platforms, logistics systems, and IoT sensors into Databricks continuously, and getting model decisions back out to those operational systems fast enough to act before a transaction clears or a flight departs.

What is an Event House in the Solace and Databricks context?

An Event House is the combination of Solace's Event Mesh with the Databricks Data and AI platform, creating a continuously flowing, bi-directional data architecture. Instead of batch ETL delivering stale data, the Event Mesh routes events from any producer to any consumer in real time—including feedback loops that deliver Databricks model scores back to the operational systems that generated the original events.

How does Solace integrate with AI agents on Databricks?

Solace provides MCP server integration that connects AI agent components within the event mesh, enabling agents to subscribe to live event streams and publish decisions back to downstream systems. This allows Databricks-hosted models and agents to participate in real-time workflows rather than only responding to periodic batch queries.

What production deployments are highlighted in this video?

The video highlights production deployments at United Airlines—where the architecture predicts flight delays before they cascade into cancellations—and CIBC, a financial institution using real-time fraud detection. These deployments demonstrate that the sense-decide-act loop with Solace and the Databricks Data and AI platform operates at enterprise scale in latency-sensitive environments.

Full transcript

[00:07] All right. Can you guys still hear me? Okay. So, I'll do a little bit of intro and then we'll skip over to the slides so we can keep moving. So, my name is uh Mike Hillman. I am the uh director of technology for the partnerships for a
[00:19] company called Solace and we're partners with uh data bricks. We do um eventbased distri event based basically data movement. So we move data from any consumer to any uh any uh any producer
[00:33] to any consumer. We do this in real time. We do this reliably. We do it globally. So the talk I'm going to be giving today is going to be about how you can take advantage of that granular real time data movement in the context
[00:46] of your data bricks deployments. So that's what we'll be talking about. And here we go. Thanks guys. Thank you. We're good. I appreciate that. You're good. You're good. Thank you.
[00:58] Okay. So, I've got through this. So, that's that's what we're going to be doing today. Um,
[01:11] all right. I'm going to use my keyboard. Um, okay. So when we think about um when we think about uh data moving around in the context of of data bricks analytics and events there's a pattern right that every high
[01:24] performing architecture that I've worked with um employs it deploys and it's it's a loop right so you sense what's happening in your environment you run it through models right and you make a decision and then you act on that and
[01:37] the reason why I want to call this out the reason for you know for this presentation is that most organizations s have one piece working really well and the other two kind of struggle right so datab data bricks is an exceptional
[01:49] decide engine uh the models are strong the compute is there unity catalog keeps things governed really well as you guys know um and that part is well solved what's harder and what I see teams
[02:01] spending more and more time on is the sense and the act side right so getting events from the real world into data bricks cleanly continuously and I at a low latency and then getting the output of those decisions back out into the
[02:14] operation systems that need to act on them. So, it's not into a dashboard that someone checks tomorrow morning. It's into a system that responds right now. Um, and that's what we'll be talking about uh today with this session.
[02:27] Um, so what does it look like to not have uh the loop closing as you as I like to describe it? Um, first of all, a stale data. So, this is the most common
[02:39] issue I've seen. uh your operational systems, your payment platforms, logistics systems, sensor networks, they're all generating events continuously, right? But those events are being um they're being collected,
[02:52] they're being staged, and they're being batch loaded into data bricks, usually on a schedule u hourly, sometimes daily. Uh by the time your model sees the data, the moment that that it should have or
[03:04] could have acted on that has passed. So you're making decisions about yesterday or last hour. um siloed pipelines is next. So every source uh speaks a different language, right? Uses a
[03:16] different transport protocol. Your IoT devices are on MQTT, your ERP systems are on JMS. Uh your SAS platforms probably have REST web hooks. Um each of these require usually a custom
[03:29] integration to land data in data bricks. And each of those integrations becomes a point of failure and a point of maintenance. One-way flow. Uh this is the one that I think is most underappreciated. Uh
[03:41] you've built a great model. It produces prediction or or um classifications or anomaly score something like that. And then what right the output sits maybe it sits in a delta table. Someone queries it. Maybe it's uh makes it into a report
[03:54] somewhere but it doesn't automatically trigger the downstream system that needs to act on it. Right? That would benefit it from the most. Uh the model's making the decisions that nobody can really act on in real time.
[04:07] And finally, hybrid complexity. So most enterprises aren't running everything in one cloud. You've got an on premises infrastructure, multiple cloud providers, uh maybe edge compute at some
[04:19] remote location. And getting data flowing cleanly across all of that, right, with consistent governance is genuinely a a hard thing to do. And the stitching costs can add up fast if you if you if you put it together. So these
[04:32] four things together are why organizations I've seen end up with great data data bricks environments that don't fully deliver on the potential. Uh
[04:45] decision is only as valuable as the speed at which it reacts right to and with the systems that need to react to it. All right. So what does this solution space look like? Uh the shift that I'm
[04:57] going to be talking about here is is is a shift in thinking. It's it's shifting it's it's from going from thinking of your lakehouse as a destination, right? A place where data arrives and gets analyzed to thinking of it as a
[05:11] participant in a continuous real-time flow. So data comes in as events happen, models run continuously or near continuously. Decisions go back out the moment they're made. The whole system behaves less like a data warehouse and
[05:24] more like a nervous system. Uh three things have to happen for that to work. First, you need to be able to ingest data from any source. We've talked about this. It's low latency and with quality guarantees. Um, it's not batch jobs.
[05:37] It's not polling. Actual event driven ingestion where the data arrives in data bricks seconds after it happens in the real world. Next, your AI and analytics need to be running on that fresh data. So, a model trained last week on last
[05:50] month's data is doing its best, but it's it's not doing the best that it could do, right? um operating on live operational events is going to do something that's really qualitatively different. And third, as we've
[06:02] mentioned, the output needs a path back out. So, it's a real-time operational systems, customerf facing applications, uh downstream services, and all of them need to receive uh the decision your models make in real time, reliably, and
[06:15] at scale. And that's where it gets difficult. um not through a manual export, not through a daily batch, but through the same event driven infrastructure that's feeding data in. Uh when all three of these are working,
[06:27] you have what I'm calling an event house for the talk, but it's a lakehouse that's a um that's a live participant in your operations. It's not just an analysis tool that sits on the side.
[06:39] Um so if we dig a little deeper into this, we'll start on the left on the sense side. Uh your environments generated advance continuously, right? could be a temperature sensor on a piece of manufacturing equipment, a transaction on the payment network,
[06:51] right? So these events are happening in real time across multiple systems using different protocols and different formats. Um the first challenge is getting those those uh events into data bricks cleanly, right? So the classic
[07:04] approach is to have the source system write directly to a landing zone. Data bricks maybe goes out and picks it up. Um that works at small scale, but it starts to fall apart when you have dozens of source systems or more.
[07:16] different schemas, variable event rates, different qualities of service, right? Different requirements. And what you actually want is a layer in between that handles your routing, handles your filtering, handles schema validation,
[07:29] right? Fan out, these types of things. So that by the time an event reaches the Delta Lake, it's been validated, it's in the right format, and it's been delivered to the right tables, right? And to the right teams through that. And
[07:42] that's the event message you see here um in the middle of the diagram. uh you can think of it as uh intelligent infrastructure for event flow. It's not dumb transport, right? It's actively routing uh with governance built in.
[07:54] Events go from the IoT fleet right to the IoT team's delta tables. Payment events go to the fraud team where they're supposed to go, right? And it's all governed by topic level access controls. So the right data reaches the
[08:07] right consumers and only the right consumers. Um now data bricks has this you know clean validated continuously flowing uh data and a spark micro integration handles the actual data movement. We'll talk about that a little
[08:20] more in just a second. um key to understand or to grab is it's birectional and it's got um exactly once semantics right so data moves in the decisions move out right uh through this
[08:32] the same connected infrastructure uh data bricks does its job models run predictions are made anomalies are detected recommendations are generated and this is where obviously the investment in unity catalog and data
[08:44] bricks AI the whole platform pays off but the difference here is that the models are running on data that's seconds old, right? Not hours old. And then that decision needs to get back
[08:56] out, right? So a fraud score needs to reach the payment system before the actual transaction settles. A predicted failure needs to create a maintenance work order. Um in the traditional lakehouse architecture that I'm familiar
[09:08] with, right? There's no clean path for that. Uh some maybe sometimes uh a query table's queried, someone builds a report, someone exports a CSV, right? the decision is minutes, hours or even days old. Sometimes um in an event house
[09:22] infrastructure as as depicted here, uh the output of data bricks flows back into the same event infrastructure, right? Downstream operational systems are subscribed to the results they care
[09:34] about. The moment a model produces a decision, that decision is distributed right to all the right systems again at the right time with the same quality guarantees as the inbound data had. uh
[09:46] the return path is what closes the loop here. Uh and closing the loop is the difference between AI that generates insights and AI that actually changes outcomes.
[10:01] Um so a little bit about the spark micro integration. So the micro integration is the connector between this event platform I'm talking about and of course data bricks. First of all um it has aentic it supports a gentic AI integration. So Solace has an agent
[10:13] mesh. It connects to the data bricks AI agent components uses rest or or MCP. It's a real-time foundation for agentic AI workflows.
[10:25] connector handles both inbound and outbound event distribution. Exactly once delivery and this is critical um as you guys I'm sure are all familiar with, right? And it but it's harder to do than it sounds in a
[10:37] distributed system. Network failures happen, right? processes restart consumers disconnect exactly once means that every event is processed exactly once it's not dropped it's not duplicated for ML training that matters
[10:50] enormously right a duplicate can skew your model a loss record is a blind spot in the model and then exactly once with replay means you can also go back and reprocess historical events if you
[11:03] change your model or discover a bug in your pipeline it's Delta Lake native so events land directly in delta tables. They're not in a staging area that then has to be loaded loaded somewhere else. The schema
[11:15] contracts are enforced at the source before data enters your lakehouse. So you're not dealing with schema drift, right? Or malformed records downstream. It's unity catalog align. So your governance policies lineage,
[11:27] discoverability, access control extend to the event layer. Uh streaming data is treated the same way as your batch data from a governance perspective. Um auditors can trace an event from origin to model output.
[11:40] And lastly, it's decoupled. This one matters for long-term maintainability. Uh your data producers, the you know the source systems that are generating events and your data bricks consumers are independent. So the schema change in
[11:52] one doesn't require redeployment to the other and you can add new event sources without uh touching your spark jobs. Um and for those of you using uh mosaic AI agents, the same pattern extends to the
[12:04] agent workflows. Agents can subscribe to operational event streams instead of querying databases on demand. They have current context, right? They can be triggered by events rather than waiting to be called and uh vector search
[12:17] integration means agent responses uh can be grounded in the current enterprise data versus right what it was trained on or even databases that could be out of sync or or with stale data. Hot off the
[12:30] press. uh we also support uh the zerobus uh connectivity that data bricks has introduced and again this is another way to get data in with all the same qualities of service that I've mentioned
[12:47] um okay let's talk about some examples now uh we'll start with fraud detection this is probably the cleanest uh or the clearest example of why latency matters a payment transaction fires uh the question the fraud team needs to answer is is this legitimate right if you're
[13:00] running that question through a batch process. They aggregate transactions. Um maybe it's, you know, could be hours old and you have an answer minutes or hours or even minutes is too late, right? Um so the fraud's already happened. In the
[13:13] real-time loop, the transaction event arrives in data bricks within milliseconds of it being initiated. The model runs, the fraud score comes back, and that score is distributed back to the payment network in real time before the traction the transaction clears. So
[13:26] we're talking about a round trip measured in milliseconds, not minutes, right? So the decision that's made and acted on while the event is still in flight so you can actually do something of more value. Um architectural uh key
[13:38] here is that return path I've been talking about. Um it's not enough to detect fraud and data bricks. The detection has to reach the system that can do something about it right in the right time. Um flight operations. So airlines are interesting case because
[13:51] the data bricks environment is processing this huge variety of event types simultaneously. You've got weather data, crew positioning, air traffic feeds, etc. Um each of those is an input stream. Um and the model's trying to
[14:04] predict in real time what flights are at risk of delay and what the downstream cascade uh would look like. Right? When a prediction emerges, say a 40-minute delay probability on a a connection hub,
[14:17] the decision has to fan out instantly to multiple downstream systems. So gate scheduling, uh passenger notification, connection flight holds, right? Each of those is a different downstream consumer on the same data bricks prediction. The
[14:30] event structure um handles this fan out. So one prediction is distributed to many different um operational systems. It's all simultaneous each receiving the specific fields that they care about,
[14:42] right? And that's not possible with a shared database query where everyone pulls, right? You can end up with race conditions, inconsistent reads, consumers working from different versions of the same fact, etc.
[14:54] Uh predictive maintenance is another popular one. Manufacturing and industrial operations gives us a kind of real-time loop where the time scale is longer, but the stakes can be just as high. A sensor on a piece of equipment
[15:06] starts showing vibration readings that are maybe outside the normal range. It's not alarming. It's not a red alert. It's just a pattern that's recognized. In a batch world, this data could accumulate, maybe eventually get analyzed and put into a report that's daily or weekly uh
[15:20] output. Um by then the bearing may have failed um um and the line then becomes down. You're in emergency repair mode which is costly in the real time of an infrastructure. That telemetry stream flows continuously into data bricks. The
[15:33] model which is learned what early failure patterns look like identifies the anomaly within seconds of it emerging. Um a maintenance work order is automatically generated and it's routed to the right crew. Right? So the repair happens during scheduled downtime. it's
[15:46] not an emergency and the loop from the sensor reading uh to the maintenance uh action closes in in seconds of it occurring. And lastly uh supply chain logistics. Um so supply chain is where uh the geograph
[15:59] the geographic complexity of the real time uh data problem becomes most visible. You've got data coming in from vehicles on the road, warehouses, suppliers, retail locations, right? uh across multiple countries, multiple
[16:12] cloud regions, multiple hypervisors potentially with the same uh reliability and some of them without the same reliability, right, of connectivity. And the event mesh has to handle all that heterogeneity. Uh a delivery vehicle
[16:25] with a a low connectivity um kind of zone needs its telemetry buffered and delivered reliably when the connectivity is restored. A warehouse system in one cloud needs to exchange events with the logistics platforms in another cloud. Uh
[16:38] data bricks model models need all of that data continuously to keep their demand forecast and routing optimizations current. Uh when data bricks uh detects a supply chain disruption, right, a supplier
[16:51] delay, maybe it's a shipment uh reroute, an inventory shortfall, that signal needs to go back out to the systems that need to respond. So it's procurement, it's logistics, it's retail replenishment. Has to do it in real time, right? Not in tonight's batch run.
[17:06] Um, all right. So, let's take a look at a few uh real examples. Um, United Airlines is using this architecture I just talked about in their LMX pipeline, right? That's a logistics management uh system. It's real operation events
[17:19] flowing through real-time event layer into analytics with decisions flowing back out to operational systems. They're not piloting it. This is in production today. Uh, CIBC is another one. Um, they're
[17:31] deploying this for real-time financial decisions. Uh banking is one of the most demanding environments for the for data architecture, right? Strict regulatory requirements, high volume requirements, zero tolerance and data loss. Um all
[17:44] these needs are are also have to be met in in in milliseconds. And the fact that the architecture meets those requirements in a regulator in a regulated uh banking environment, it's pretty meaningful for us in terms of its maturity. Uh and then standard charter
[17:58] uh as well. Um the data bricks connector, it's validated. Uh these deployments handle millions of events per second. They span cloud, on-rem and edge environments. Um and uh the time from starting an integration to having
[18:11] it running in production is measured in days, not months of custom development. So it's real quick to deploy. All right, before I close, I got a few minutes. Um I want to address a question
[18:23] that I hear a lot, which is what if I already have something in place that does something like this, right? Which is a fair question. Um and the answer is it depends right on what you're trying to do. Point-to-point streaming solutions where your system produces
[18:35] data and one system consumes it in a controlled environment. Um uh they're they're well served by a lot of tools. The challenges show up when you have architecture needs that grow beyond the initial scope. So what does that mean?
[18:47] Right? First uh gaps tend to appear um is at the edges of the enterprise. So most streaming solutions were designed for cloudnative single environment deployments. They handle data well when everything's on the same network using
[19:00] the same protocol under the same team's control. But most large enterprises aren't that homogeneous. You've got manufacturing equipment running on their own individual protocols. You got legacy systems in an on-prem data center
[19:12] running their different protocols, legacy protocols. And bridging all of this is where you have significant engineering effort that quietly gets consumed, right? And where pipelines can break. Uh the second gap is the return path. So
[19:25] we've talked about this already. Uh most streaming investments are heavily optimized for inbound data movement, right? Pipelines built to get data u into an analytics program, right? The reverse direction, getting stuff back out. It really is kind of an
[19:37] afterthought. Um so, uh it's solved with polling or or scheduled reports, one-off migrations again that are fragile and slow. Uh but a genuine birectional event infrastructure uh is is less common. And
[19:51] the third area is governance at scale. So you've got dozens of event streams flowing to dozens of consuming uh teams across uh different environments. uh scale becomes highly costly and that's where uh this solution uh shines right
[20:05] so the capabilities here right multi-protocol support global event mesh spans the environment it's birectional flow by design um and built-in governance right we're not we're not talking about replacing infrastructure
[20:18] right we're talking about filling the gaps that show up when the scope of the problem expands beyond what uh simpler point solutions were designed for and my last slide here what what I hope you take away from this is that the
[20:30] architecture we've walked through is not about replacing what you've built in data bricks. It's about complementing it, completing it, right? Getting data uh from more sources faster with better quality and building the return path so
[20:43] that decisions on your models actually right make changes to operational systems uh the right operational systems and in a time that you can use to actually take valid valid action. Right? Right. So, this pattern is proven. We've
[20:56] deployed it uh plenty of times. The integration uh is data bricks validated and available today. And if you're curious what this might look like for your specific environment, right, your sources, your event volumes, downstream
[21:08] systems, what have you, I'd be happy to chat if you guys have any questions. Other than that, I thank you for your time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.