Securing the Grid: Modernizing E&U Security Operations with Lakewatch (AI-Native SIEM)

Summary

Energy and utility companies face a convergence of OT/ICS ransomware threats, strict NERC CIP and TSA regulations, and an explosion of IT/OT telemetry that overwhelms traditional SIEMs with ingest-based pricing and proprietary data silos.
Lakewatch, Databricks' open AI-native SIEM, eliminates ingest taxes by storing all security data in open-format object storage with OCSF-aligned schemas and enables SQL-based detection engineering and agentic threat hunting over petabytes of historical data.
This video presents Energy Queensland's journey from traditional SIEM to Lakewatch, demonstrating data ingestion, detection migration, Genie AI-driven threat investigation, and case management for critical infrastructure security operations.

Securing the Grid: Modernizing E&U Security Operations with Lakewatch (AI-Native SIEM)

Energy & Utilities (E&U) face a perfect storm: surging OT/ICS ransomware, strict NERC CIP/TSA regulations, and an explosion of IT/OT telemetry. Traditional SIEMs, hampered by ingest-based pricing and data silos, can no longer keep pace. Enter Lakewatch: Databricks’ open, AI-native SIEM, built natively on the Lakehouse. this video provides insights from Energy Queensland's journey of navigating the challenges of traditional SIEM and the impact a security lakehouse on Databricks can deliver. We'll discuss how Databricks Lakewatch, the lakehouse-native SIEM, eliminates "ingest taxes" and unifies IT, cloud, and OT logs through OCSF-aligned schemas; how SQL-based detection engineering and agentic threat hunting open up petabytes of historical data in seconds; and how an open, decoupled storage and compute architecture delivers the visibility, long-term retention, and AI-readiness critical infrastructure operators need to increase their situational awareness.
Talk By: Dylan Mills, Senior Security Engineer, Rearc ; Rakesh Cherukuri, Solutions Architect, Databricks ; Shane Atherton, Account Executive - Energy, Databricks ;

Chapters

FAQs

What security challenges are specific to energy and utility companies?

Energy and utility companies face a unique combination of threats: surging OT/ICS ransomware targeting operational technology like power grid control systems, strict regulatory frameworks including NERC CIP and TSA directives, and a massive volume of IT and OT telemetry from increasingly decentralized infrastructure. The distinction between IT and OT breach response also matters—IT systems can be taken offline for patching, but OT systems controlling physical infrastructure often cannot.

What is Lakewatch and how does it address energy sector security needs?

Lakewatch is Databricks' open, AI-native SIEM built natively on the lakehouse, designed to overcome the limitations of traditional SIEMs for critical infrastructure operators. It eliminates ingest-based pricing by storing data in cheap open-format object storage, unifies IT and OT logs through OCSF-aligned schemas, and enables SQL-based detection engineering and agentic threat hunting over long historical retention periods.

How did Energy Queensland deploy Lakewatch?

This video presents Energy Queensland's journey from traditional SIEM constraints to deploying Lakewatch as their security operations platform. The deployment involved ingesting security data through Lakewatch connectors, migrating existing detection rules, and implementing Genie-powered AI threat investigation workflows alongside case management with a security hierarchy for triaging informational versus critical alerts.

How does Genie assist with threat investigation in Lakewatch?

In Lakewatch, Genie provides a natural-language interface for security analysts to investigate threats by querying petabytes of historical security data without writing complex SQL. Analysts can ask questions about suspicious activity, retrieve correlated events across data sources, and generate investigation summaries—significantly reducing the time and expertise required for threat triage.

Full transcript

[00:08] All right. Good afternoon, everyone. Welcome to our breakout session. Uh securing the grid, modernizing energy and utility security operations with Lakewatch. Today, we're pleased to co-host with one of our strategic partners, Rearc.
[00:22] Let's give you some introductions. So, my name is Shane Atherton. I'm an account executive at Databricks based out of Brisbane in Australia. We're a partner with some of our energy and utility customers. Rakesh? >> I'm Rakesh Chugani, one of the solutions
[00:35] architect with Databricks. I'm based in Brisbane, Australia as well. I look after energy and utility customers. Yeah, thanks for joining us, Dylan. >> Uh Dylan Mills, I'm a lead cybersecurity engineer at Rearc. Uh based in Eastern
[00:47] US. Uh I've been working on Databricks with cyber for a couple of years now. >> Thank you, guys. All right. Sorry, before we kick off today, I should have said we got our forward-looking statement. Um as you
[00:59] know, roadmaps change. Please uh consider this as we run through the presentation and into the future. Um as always, to try and make Data and AI Summit better every single year, we appreciate feedback, honest direct
[01:11] feedback. So, please go for it and make sure you uh you let us know how you find the presentation. Okay. So, what to expect from today? I want to start with a little bit of a
[01:24] survey in the room. So, who here is in the energy and utility sector? Okay. We've got a nice blend. Um look, for those of you who aren't in the sector, don't worry.
[01:36] I'm going to give you a really simple 101 guide on what's been happening in the energy industry. Some of the transformation, some of the challenges, some of the opportunities. Next, we're going to talk about the
[01:48] intersection between industry and security. Because those of you who are in the energy and utility industry know that security is not an option. Lastly, where the rubber hits the road,
[02:01] how technology can help improve your security posture using Databricks. We'll do this through the lens of a demo and a customer story from one of our customers from back home, Energy Queensland.
[02:14] So, that's what we got for you today. So, reliable energy has become an intrinsic part of the way that we live our lives. It's everywhere in everything that we do.
[02:27] But, many of us use energy without much thought of what happens between the light switch and the power station. What was once a simple, reliable flow of energy has now turned into
[02:40] something that is complex and decentralized. And because of this, for our utility organizations, your critical infrastructure is now a key target for cyber attacks and bad actors.
[02:53] Now, today we're going to talk about securing the grid. But, before we talk about securing it, first we need to talk about how the grid has changed. A simple overview. So, let's start with a traditional grid.
[03:06] My grandparents' grid. Coal-fired power station, transmission lines, distribution network, and consumer. A simple, reliable flow of energy from generation
[03:18] through to consumption. This served us well for decades and decades and decades. And it's really the adoption of renewables where you start to see the first change.
[03:31] Now, our energy generation profile is tied to what's happening outside, if the sun is shining or the wind is blowing. And the way that that energy is generated is very high highs, very low lows.
[03:44] And for the first time, our utility organizations need to deal with energy coming back the other way on a grid that was only ever built for one-way delivery. But so over the past 10 years we've seen
[03:56] an explosion of innovation both on the consumer side and on the utility side. EVs have brought with them the need for more network capacity, but EVs and smart controls also have the ability to pull
[04:10] excess energy out of the grid when it's not needed. But the biggest shift we've seen is the rise of the prosumer. What used to be a simple consumer at the end of that value chain
[04:23] now has the ability to generate their own energy through solar panels on the roof, store that energy in a battery, maybe use it later, or sell it back to the grid when the grid needs it the most
[04:35] for a tidy little profit. So as you can see, the building blocks of our grid today look very different to what they were not so long ago.
[04:48] What about the energy flowing through the grid? Well, those changes that we made to the grid have caused our consumer habits and energy profiles to change. I'm going to share a little bit about the Australian energy industry as a proof point.
[05:01] This graph here from our home state in Queensland is what it's commonly called the duck curve, and many of you in the industry would have heard of this. It shows the difference in power demand between 2011 2025.
[05:15] As you can see, as renewables flood the network around lunchtime, a big drop in demand. Everyone gets home, the sun is no longer shining, big peak. And this gap between minimum system load peak demand is only getting
[05:28] further and further apart. A few more points I'd like to share with you that back this up. Australia has the highest adoption of residential rooftop solar than any other
[05:40] country in the world, over 30% of residences. It generates so much energy that our energy market regularly turns negative, meaning we got too much energy in the grid, not enough demand.
[05:53] The counterpoint to that, coal. We are decommissioning coal-fired power plants faster than any comparable country. The goal is in the next 10 years to have 2/3 of our coal-fired power fleet decommissioned.
[06:06] So, it's not just the building blocks of the grid that's changed, it's the way that we operate the grid that's changed as well. Now, what does all this mean for utility organizations?
[06:18] Well, the modern grid is controlled by software and digital communication. We're seeing more and more automation and digitalization on the OT side. But, the real value lies in combining
[06:31] your corporate system data, your work orders, your asset information, your weather data with what's happening out on the network. And for the digital utility, this is now table stakes.
[06:44] The worlds of IT and OT are no longer simply coming together, they are converging at a rapid rate. And because of this, the utility of the future needs to first become a data company.
[06:58] Now, with all this transformation and complexity, as you'd expect, there's a bunch of implications, especially around security. Rakesh, I know you've been working with some of our customers around security. Why don't you come and share some of
[07:10] your perspective? >> Yeah, sure, Shane. The energy transition we are talking isn't just changing how Can you guys hear me now? Okay. Sorry.
[07:23] So, the energy transition that Shane was talking isn't just changing how the grid operates. It's also throwing a challenge to the secure teams and putting them in a position where old rules of defense doesn't apply anymore.
[07:37] Let me Let me take you to January 2024. Lviv, Ukraine. It's freezing outside with sub-zero temperatures.
[07:50] A piece of malware lands on a control network of distinct heating utility. It doesn't steal data or drop ransomware. It just sends valid commands that is
[08:02] designed to obey them to the control industrial controllers. The result, 600 apartment buildings lose their heat for 2 days.
[08:14] Imagine the families, the kids, and the elderly people going through that situation. This was Frosty group, a piece of malware that can talk to Modbus, the
[08:26] protocol that has been running the industrial controllers for past 4 decades or so. It just It didn't break anything, right? It just used the system the way it was
[08:38] supposed to be used or designed to use. When a breach happens in IT, you lose data or you lose trust, and you probably end up writing a press
[08:50] release about it. But, when a OT attack hits the critical infrastructure, our society grinds to halt. We're talking hospitals losing power,
[09:02] the water plants going dark, financial systems freezing. These attackers aren't just trying to steal the data. They are nation-state actors trying to
[09:14] the foundations of our daily life. So, how did we get here? Why an attack like Frosty group or other instance we are seeing these days are
[09:26] possible now? Well, for decades for decades, the operational technology that runs the grid, whether the SCADA systems, the PLCs, and the
[09:38] RTUs, they are designed for isolation. Air-gapped, separate systems, separate teams.
[09:50] That old is over. The numbers tells us the story. As Shane mentioned earlier, Australia has the highest rooftop solar eruption than anywhere else. What that means is we now have 4.2 million rooftop solar sites across the
[10:04] country. Every one of it is a inverter sitting on the customer side of the meter.
[10:16] Globally, IEA estimates 30 to 40 billion industrial connected devices. Every one of it is an attack surface that didn't exist a decade ago.
[10:28] Finally, the SANS Institute report that nearly 40% of organizations now share the IT and OT security budgets. Because the perimeters are dissolving, the teams and budgets are merging,
[10:41] the line between IT and OT is not architectural anymore. It's organizational. And the your security infrastructure, the architecture needs to follow the suit with a one single fight stack
[10:55] rather than having multiple stacks. So, that's the attack surface getting bigger and bigger every year. Now, let me show you what actually coming along.
[11:12] Across the growing attack surface that we saw earlier, what we are seeing is threats are accelerating. Again, the numbers can talk to themselves, but without going in detail, I just want to call out just last year
[11:24] alone, the ransomware groups targeting the industrial sites jumped to 49%. What we are seeing is it's not just more attacks happening.
[11:36] There are more attackers in the industry trying to the critical infrastructure. But the number I want to you to focus is on this one. 42 days.
[11:49] According to Dragos, that's the average ransomware dwell time in OT environment. Six weeks or sometimes even more, an adversary sits inside your control
[12:01] network before they act on it. Now, let's tie that back to our traditional SIM platform. Which limits the ingestion and data retention
[12:13] because of storage and ingestion-based costing, you have a massive structural blind spot. And it's not just attackers are moving faster. The global regulators, security
[12:27] regulations are evolving and then trying to keep up with the changes happening across the globe. We don't have to go to individual regional legislation across because if we strip away
[12:40] the different frameworks, acronyms, they're all asking for the same operational ask. The regulators want to know, do you have do you have visibility across any attack that might happen across all
[12:53] of your telemetry sources, whether it's IT or OT? Do you have enough retention logs? Do you have the speed to be able to query and investigate any threats?
[13:07] You cannot answer those questions with three different tools and a one Excel sheet, right? We need one unified single platform that can where you can instantly query a copy of your telemetry data across all
[13:19] the sources. But getting that unified view where our world play world rules of play book failed.
[13:32] As I mentioned earlier, when the stakes are so high, the old rules of defense doesn't apply anymore. Because the traditional same platforms are fundamentally broken for this new era.
[13:45] Let me talk you through that. At first, due due to ingestion and storage-based pricing, organizations are often forced to make a trade-off.
[13:57] Often discarding up to 75% of telemetry sources or logs just to stay under the budget, actively creating their own blind spots.
[14:09] Second, we have seen earlier this year at RSA conference, attackers are now leveraging swarms of AI agents to that run 24/7 to uh weaponize the vulnerabilities, right?
[14:22] The with that uh the zero day uh graph that we have seen earlier, it shows that the uh time to exploit is collapsed to hours or even seconds nowadays.
[14:35] The human led SOC team don't stand a chance against these kind of machine level AI storm agents trying to attack continuously.
[14:47] Third, the data we do manage to keep is locked into proprietary formats. You can't easily correlate the logs across the telemetry sources. You can't suddenly bring modern AI capabilities to your data because
[15:02] the vendor holds the keys. We need a new or different architecture approach.
[15:14] What we call it as a the open security lakehouse. The open security lakehouse is an architecture pattern that is built right directly onto the Lakehouse concept.
[15:30] Just like any enterprise data Lakehouse, you ingest all your telemetry logs irrespective of type and volume into one single unified platform. You store them in the open formats in a cost-effective layer.
[15:42] And then with you don't have to unlike any traditional seam, you don't have to discard any critical logs. You can store all of your logs which gives you a complete visibility across your estate.
[15:59] On top of that open data formats, you layer the composable open agents which can automate the SOC operations. Whether it's right from data ingestion, authoring authoring detection rules, threat hunting,
[16:12] or even triaging alerts. With this architecture, which we shift the SOC operations from human bottleneck to human supervised. So, empower SOC teams with the AI-powered agents.
[16:31] My colleague Dylan will dive deep into the architecture shortly and then show you in in action how how you how do you implement with a customer story. But as Shane mentioned, we've been working with one of our customers uh Energy Queensland in Australia who were
[16:44] in the right in the middle of energy transition, also trying to navigate all of these challenges we saw. Shane, why don't you share a bit more about the uh Energy Queensland uh to share some of their insights from the
[16:57] journey? Thanks, Ritesh. All right, let me give you a quick introduction to Energy Queensland if
[17:09] you're not too sure who they are. So, they're largely uh DNSP or a distribution network service provider. They run two distribution networks, uh one regional, one metro, and also a retail arm.
[17:22] They service the great state of Queensland. It's around 6 million people that live in an area roughly two and a half times the size of Texas. So, relatively small population spread over a large uh area.
[17:35] As you'd expect, this means they've got quite a large distribution network, around 210,000 km of poles and wires, making them one of the largest DNSPs in the southern hemisphere.
[17:47] So, their customers include, as you'd expect, retail Oh, sorry, um residential, uh commercial, but also critical services. Think ambulance, telecommunication network, uh emergency response. Living and breathing many of
[18:01] the challenges we spoke about today. And for Energy Queensland, security is a non-negotiable. Job zero. Built into everything that they do, given how critical they are.
[18:16] And we got this feel when we first started partnering with their security team. And they shared with this this overarching goal they've been trying to achieve for some time. We need to increase our situational awareness without increasing cost.
[18:29] Now, naturally, we took this and we used it as a overarching mantra for what would soon become an MVP, minimum viable product. Now, with any project, there's a myriad of metrics and goals that we need to
[18:41] achieve. But there's just two I'll share with you because they give you an idea of what we were able to deliver with Energy Queensland. Increase log ingestion by over 300%
[18:53] and increase log retention up to 7 years. More sources, more data for a longer period of time at no additional cost.
[19:05] Dillon will share with us very shortly what we built as part of that MVP, but there's a moment I want to share with you all, and it's actually super funny. This This quote has come up now three times since being here at Data and AI
[19:18] Summit. So, I feel it might not have the same impact, but let's do it anyway. We're lucky enough to witness a lightbulb moment with our customer in real time. He turned to us in a meeting and he said, "Do you know what, guys?
[19:31] It's not a security problem. It's a big data problem." And this is what happens when we remove the constraints of legacy tooling, and we look at this as simply a challenge for technology to solve.
[19:45] So, if there's nothing else that you take from our session today, I ask you please keep that in the back of your mind when you go back to to work on Monday. Um Look, let's get into this MVP. We
[19:57] welcome Dylan to the stage. Can you all hear me, right? All right. Awesome. So, uh I'm going to start with
[20:11] um a generalized Lake Watch engagement architecture that we kind of follow uh in a lot of our different um engagements with clients. So, uh Rearc is a one of a few uh Databricks partners that um
[20:23] implement Lake Watch uh for a well little while now on um a selection of different customers in energy and utility, but also many other different um industries. Um the big question here with Lake Watch
[20:35] is if it's a big data problem, do you need a big data team? Ideally, Lake Watch kind of takes that away and it obfuscates and makes it easier for a cyber team to understand and be able to
[20:47] deploy on Databricks without needing that like, you know, defined detection engineering or data engineering background, right? So, uh Lake Watch, if you haven't heard of it before, I'll say it at a high level, uh open agentic SIM designed to
[21:00] simplify cybersecurity data collection, detection, investigation, and involves a lot of uh additional skills that you might need in Genie code as well. Um if you're interested, uh they have a fantastic demo of it uh down in the Expo
[21:12] hall, um but we'll get started into um how we kind of break things out, right? So, when we help with MVPs like we did with EQL, our goal is to find the main like subsection of use cases that you
[21:24] need to prove to gain trust on the system. Um for EQL, that story revolved mostly around uh augmentation, parity, and long-term historical analysis. Those are kind of the problems that they were
[21:36] having ex- with the existing um platform outside of just, you know, direct cost and license usage. Um what does a new Lake Watch MVP look like? Uh well, so if you go from left to
[21:49] right, we kind of break it out into the data ingestion piece, transfer transformation, and then detection and analysis, and integration at the end. Um so, I'm going to go straight into each of these sections starting with
[22:01] data ingestion. So, uh big concern compared to Splunk um that they were using previously is, you know, that system is on schema on read. Um they want to hit these long-term analysis goals, but
[22:15] it takes a lot more work up front to build those well-defined tables and like have that ready to go. So, that's the main thing that we focused on. Uh in order to do that, uh you can see in the kind of gold sections
[22:27] are the IT sources we decided to um focus on. So, EDR, network, firewall, uh identity and access, and and cloud API activity. Those are the things in the IT world that they were existing doing
[22:39] today that they wanted to see how it works over here and then scale out, you know, within just these data sources, but also the other ones like threat intelligence, OT, endpoint logs, business apps, all those different
[22:51] things. So, um target of our MVP though, uh we wanted to make sure those those worked, um but on the right side at the time when we did this engagement, um the ingestion
[23:05] method was solely autoloader. So if you're not familiar with autoloader, it's you know, Databricks way to bring in incrementally from cloud storage. So a lot of it was dedicated to you know, or dependent on a data routing tool.
[23:19] So there were some custom things that we had to do in this engagement where you know, if they're using a cribble or a data bond, you know, there's some small tweaks, some JSON formatting that's being added, some metadata that's in applied that you know, the event time
[23:32] plus the time it was collected, right? So there's additional things that we you know, showed them how to do, gave them those examples, showed them the different like standardizations like OCSF normalization.
[23:44] The really cool things though on the data engine ingest piece, I only have two of them here, but just since coming to you know, the conference, my you know, information's already outdated. There's already you know, how many different you
[23:56] know, ways to get data into the platform, but I'm going to focus on these two. Zero bus is the most exciting. So it's you know, a capability to have a direct write API into the lakehouse. If you have something like a data
[24:08] routing tool and you're you know, adding latency to your entire detection pipeline by you know, first putting it on to S3, this kind of gives you the availability to send it direct for your
[24:20] most important logs or you know, it also opens up the point of getting endpoint logs, getting logs from systems that a little bit harder to collect from. Also Lakeflow Connect. So this is a really interesting one. Re-Arc is a like
[24:34] beta contributor to this project right now. Lakeflow community connectors are a kind of a way, you might have heard of managed connectors from Databricks. This is kind of a way for the community to provide capabilities to that too.
[24:47] So you know, we they obviously have the main ones like Salesforce and things like that, but you know, cybersecurity teams want to integrate with Wiz or Sneak, those types of data sources. So we we build those out and provide those as like connectors that,
[25:00] you know, we can expand the ease of use of the platform prior to these MVPs rather than, you know, during. Jump over to the next section. So,
[25:13] this is detection engineering. So, uh you got your data into the platform, how do you monitor it, right? There's two buckets we've mostly seen with customers that they kind of fall into. There's customers that want
[25:26] um a provided out of the box managed detection library that's open source, right? You know, they understand that these are kind of commodity, you know, detections. There's going to be some tweaks that are needed made, some safe
[25:38] lists that are needed integrate. Um but for the most part, they want to just, you know, hit the ground running and, you know, get those get those operational immediately. The second bucket is teams that have existing detection logic from
[25:51] their current team, right? So, they have all of these detections that are written in KQL, SPL, you know, CrowdStrike's query language, and whatever, you know, under the sun. Um and they want to convert that over to either PySpark SQL
[26:05] to get it working with the Lake Watch platform. Uh there's a couple of things that we've done on this front. So, accelerating that kind of flow, we built some transpilation
[26:17] trans per some trans pilers to kind of convert from SPL to PySpark. Um we have some kind of more soft skill like cheat sheets for what the common, you know, pitfalls are when you're thinking of a system where you're just
[26:29] searching through a reverse index versus having a structured set like SQL. And I would be remiss not to mention the third bucket, which is I want all of my detection engineering to be agentic,
[26:41] right? I don't want to ever write another the query. I just want to describe what those detections look like. Uh we've had a lot of luck with very descriptive frameworks. If you never heard of Palantir's ADS framework, it's a fantastic one if you want to
[26:54] start going down that route. It basically you get as far as describing all of the technical context, all of the, you know, false positives, everything that you want the detection to be. Give that into a markdown file to Genie and just let it
[27:06] see what it does, right? So, um pipelines are another key concept that I wanted to bring up. Um Lake Watch um allows you to kind of or Databricks
[27:18] itself kind of allows you to have like two different theories of how you want your data to be monitored. The first is streaming. It's super low latency based on that Spark Structured Streaming. That's mostly for resurfacing EDR
[27:31] alerts. You like things that are coming into the system, I want to know about it. I don't necessarily need to do anything on it. You know, I just want to know about it really quickly. Uh then you get into things like batch aggregation. This is the traditional kind of queries that you've seen, you
[27:44] know, I want to run this on a 15-minute schedule and I want it to look back 24 um 24 hours. Things like OCSF authentication, geographically impossible access, you know, I would love to travel to Australia from here in under an hour,
[27:57] but that's not feasible and I'm sure my SOC team wouldn't like it. So, um we're going to jump over to the next section, which is investigations. So, EQ L's main concern, like we mentioned, was long-running searches that couldn't
[28:10] really do easily in Splunk. Uh I certainly remember working in with data models and trying to do some Splunk magic to make, you know, bigger data sets smaller and more indexed so that you can kind of do these sort of things in a incident response scenario or
[28:23] something like that. Um the separation and scalability of the Delta Lake and the serverless compute makes that super easy. Uh you don't really have to worry now about having this big search run. Whether it
[28:36] works or not, it's also going to affect the indexer at the end of the day. So, by separating those two things out, it opens it up uh a lot more for the opportunities to just, you know, baseline everything, get as much data in as possible, dedicate that serverless
[28:49] compute cluster to that, and I have this, and it's not going to affect the cloud storage cuz it's all API, right? Uh I'm not going to spend too much time on notebooks, but uh I as a personal uh thing just wanted to bring them up
[29:01] because it's made investigation flows so much nicer. Um I came from a world of having, you know, 20 tabs of queries open, 10 of them were duplicates because they're all slight versions of each other that I lost track of. Five of them
[29:14] were charts of different things that different executives wanted, right? Um but the real star of uh this is really Genie. So, um at the time with EQL, we only really had access to
[29:27] Genie spaces. Uh Databricks has this kind of uh habit of introducing a new feature exactly when I need it. Uh so, the second I needed to start looking at Genie investigations, they came up with research agent. What
[29:40] research agent lets you do is kind of build that plan of attack and build the queries out iteratively, so you can do things like investigations when you don't know what you're looking for yet. Um made it a whole lot easier, and
[29:53] thankfully, came out exactly when I needed it. Um but yeah, uh before uh the conference, we had a lot more Genie space at our disposal, uh but during this conference, I'm no joke overwhelmed
[30:05] about the amount of things that I'm going to be needing to fly home and test at the uh by the end of next week, right? Um so, let's just jump into a live demo. Um I'm going to show you some of the capabilities that you're seeing here around uh Lake Watch and Genie
[30:17] code. This is all in the re-arc lab, so it's all live, so bear with me if something goes terribly wrong. I hope you all understand. Let's see here.
[30:33] Switch it to two. All right. So, what you're seeing now is the uh the newest of this week uh edition of of Lake Watch. Um I had to modify my my uh records a little bit uh just because, you know, the the newest
[30:46] fresh version at uh DIS is coming out now. So, I'm just going to get it started by telling it uh run a check against my Leak Watch Prod OPC UA Bronze table. Uh so, this is a a simulated data set that, courtesy of
[30:59] Databricks, I was provided for um some OT simulations. Uh I also have a um a few records of CVEs uh critical CVEs related to OT devices.
[31:12] What this is going to do in Genie code with a, you know, two or four sentence prompt is be able to look at both of those tables independently, correlate them, see if there's any matches between them,
[31:26] and then if there are, I'm asking it to then do a triage investigation. So, it's going to understand what it needs to see after it knows what it's, you know, what the devices are. And if it finds anything, I want it to
[31:38] then create a security case. So, now I, with all of you, you know, fine people here, we can just go and leave and get coffee. We can come back in 10 minutes and, you know, we have a fully completed investigation. Um I do have to
[31:50] stay up here the whole time though, so I'm going to show a few other kind of capabilities while this is running, but I just wanted to mention that like this, as you're seeing here, is this kind of like step process, right? So, it's it's exploring the CVE data, it's
[32:02] pulling it out, it's thinking about what it saw. Uh once it's done thinking about what it saw, it's going to think about the next step. Um There's some cool things that you can do uh in Genie code. So,
[32:15] the first thing I wanted to mention was user instructions. Um user instructions are our way of telling, kind of like a review. Uh you have your your analyst that didn't investigate, and they did something great, they did something
[32:27] poorly. Uh this is kind of our way of fixing those those poor things, right? Um or just things that always going to be the case. I always wanted to know that the time zone is
[32:39] Brisbane, Australia. I never remember what the time zone sequel functions are for, you know, what I'm working in. Just going to let Genie handle that always, and I don't have to worry about it. Um some other things are I'm just going to tell it where the tables are. Uh this is
[32:52] more of a demo uh situation where, you know, I don't want it to think about all of my catalogs and all of my schemas within it. So, you know, in this use case, just look at these two. Um and then finally um just added some
[33:04] some effort uh mechanism so that it doesn't spin for too long, right? That insistent instruction uh file just lives within your workspace. Uh it can either be your user session or it can be
[33:16] a workspace session. So, does your team follow all the same rules or does an analyst have have their own set of rules? You know, those kind of correlate together, so you can kind of build one one good uh overview. So, this is still running. Um it's it's
[33:30] starting to find some information though. So, based on the CVE table, it was able to see that's um there's two identified devices from my OPC UA data set. Uh most notably the Rockwell Control Logix 5580.
[33:43] Um so, there is apparently a critical vulnerability uh CVE 2021-22681. Uh it found that and it correlated it that there is a device in our data that
[33:55] relates to it. It's going to do an investigation on it now. Analyzing anomalies, you know, value stats, looking for errors within the data set itself. Um then once it's done with that, it's going to kind of package it. I also told
[34:08] it specifically I wanted to format it in MITRE framework so that it kind of follows a a threat vector that I can uh look at later. While it thinks about that though, I'm going to bring it back to the IT world
[34:20] real quick. Uh so, uh everything that you do in Genie has this threat history. Um it's great for analysts when, you know, they did an investigation a week ago or 2 weeks ago, and they want to look back on it. It's also great when
[34:32] you do demos and you have something running a little bit longer than anticipated. You can show everybody something something else. So, I'm going to look at um Let's see. Detecting malicious activity in Databricks AI Gateway. So, this is
[34:45] top of mind of everybody especially you know within the the the cybersecurity forum yesterday about how do you actually monitor AI usage? Um So, if your you know AI is doing
[34:57] investigations, how do you then monitor that? You know, who is watching the watchers? Um This is a completely finished one. So, I basically just asked it, "Hey, do an analysis. If you find any possible
[35:09] malicious activity, relate that to the MITRE ATT&CK framework. That's their AI specific threat framework." Um it found some things. It was able to you know kind of identify some low-level
[35:21] signals. Hey, this person has extreme output. Uh this medium priority, you know, there's some model enumeration happening. There's some specific users in our environment that you know have some weird hours. In the case of Re-Arc,
[35:34] we're working all hours. So, this is perfectly normal. So, I can add that to the instructions to say don't care if somebody's working at you know midnight. Um And then some other things. So, then it gives me some recommended actions, you know,
[35:46] confirm with the users, investigate this person specifically, consider some lake watch detections to kind of have a persistent model for this. So, we got 4 minutes left. Let's see where we're at with this one. Still running. Um
[36:03] There's also some iterative things that it's going to do is when it makes some changes, when it's kind of thinking about things, it'll ask you whether you want to do specific things like you know create if you want to create the security security case, can I do this? You can set set it to
[36:16] always allow. You can also set it to always you know keep you in the loop when it's running. Um While this is going, I'm just going to show off some of the security other security cases that were built with
[36:28] Genie code. Just let that load in there. But I'm going to show the same one
[36:40] that you know, I did as a test run. Ideally when you get it in here, all of this is markdown. All of it is kind of like the report is managed for
[36:52] you. It's added to the description. The analyst comes in in the morning, you know, they might be assigned it. Maybe they it's open in the security cases UI. They but they can come in here and they already know not
[37:04] only what the detection was in in at the beginning, but also what was that initial triage. So, it auto assigned it to me. I can tell it in progress. I can, you know, take a look at these things. I can
[37:18] either pick it up as a human analyst and say I'm going to, you know, finish out what I need to do with this or it can read back in the security case into context and say like, hey, you know, let's let's have a working session
[37:30] and figure out what's going on here. I maybe wanted to correlate some other things before I determine, hey, you know, it's actually bad. Let's you start raising some alarms. But yeah, looks like it just finished up
[37:42] on the end. So, it's creating the Lake Worth security case. Fully builds out everything from description and markdown to triage findings to everything that you need when you're doing an
[37:54] investigation. So, it looks like we have 2 minutes left. Um We probably have some time for some some questions if anybody has any. Um
[38:14] Awesome. Uh Yeah, it looks like 1 minute is not enough for questions, but we're going to Rakesh Jain and I go outside and answer any of these questions but just wanted to round it out by saying
[38:27] that you know Lake Watch kind of let's you bring that like data engineering big data problem you know office gate the hard things away and you know let you bring your data in so you can actually start working on it right
[38:39] that's what we did at EQL had some great great feedback from this that team and it's only getting better as of you know this this conference this week is things are doing great so yeah appreciate everybody coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.