Lakewatch: Agentic Security for Threat Detection at Enterprise Scale

Summary

LLMs have compressed vulnerability discovery and exploitation timelines from 23 days to under 2 days, making legacy SIEMs unable to keep pace with modern attack sophistication — motivating Databricks to build Lakewatch, an open agentic SIEM on the lakehouse architecture.
Lakewatch stores all security telemetry in open formats within Unity Catalog with unlimited scale and no per-byte licensing costs, enabling security agents to automate threat detection and response while security teams retain years of historical data for analysis.
National Australia Bank, with 8 million customers and 16,000 data pipelines, is an early Lakewatch adopter that chose it to widen security telemetry beyond traditional log pattern matching to include customer, incident, and employee login data on a platform it already trusted.

Lakewatch: Agentic Security for Threat Detection at Enterprise Scale

Security teams face an unprecedented threat landscape. LLMs have accelerated vulnerability discovery and exploitation timelines from 23 days to under 2 days. Legacy SIEMs cannot keep pace with the volume, velocity, and sophistication of modern attacks. Databricks Lakewatch, the open agentic SIEM, fundamentally reimagines how organizations detect threats by moving security analytics onto a lakehouse architecture with unlimited scale and no per-byte licensing costs.
Lakewatch combines all your security telemetry in open formats, logs, events, incident data, and user activity, enabling agents to automate threat detection and response at scale. Unlike traditional SIEMs that require expensive data filtering, Lakewatch stores years of security history in Unity Catalog with full governance. Security teams gain access to the same data platform used by data scientists, enabling collaboration while defensive agents fight back against attacker-deployed agents in real time.
🤝

Chapters

FAQs

What is Databricks Lakewatch and how does it differ from a legacy SIEM?

Lakewatch is Databricks' open agentic SIEM built on the lakehouse architecture, storing security telemetry in open formats within Unity Catalog with unlimited scalability and no per-byte licensing costs. Legacy SIEMs require expensive data filtering because of cost constraints and cannot keep pace with modern attack volume and velocity, whereas Lakewatch stores years of security history and lets agents automate detection and response at scale.

Why have AI-driven attack timelines become so much shorter?

As described in this video, LLMs have accelerated both vulnerability discovery and exploitation, compressing timelines from 23 days to under 2 days. This means security teams relying on legacy tools and manual processes can no longer respond fast enough, requiring automated, agent-driven detection and response to match attacker speed.

What is the scale of National Australia Bank's security and data operations?

NAB serves 8 million customers, manages over one trillion dollars in assets, employs 10,000 people in technology, and runs 16,000 data pipelines across 2,500 applications. Patrick from NAB describes choosing Lakewatch to widen security telemetry beyond traditional indicators of compromise to include customer, incident, and employee login data, using a platform the bank already trusted.

How does Lakewatch enable security and data science teams to work together?

Because Lakewatch stores security data on the same Databricks platform used by data scientists, security analysts gain access to the same tools, compute, and governance infrastructure as the broader data organization. This shared environment enables collaboration on threat models and allows defensive security agents to be built using the same agent development patterns used across the organization.

Full transcript

[00:19] All right, welcome back to Summit Live. I'm Ari Kaplan, global head of evangelism at Databricks, and super happy to have Patrick here from National Australia Bank. Welcome. Thank you. And and I hear you are one of like the early premier implementers of Lake Watch. Yeah, and we're we're proud to be.
[00:35] Yeah, I actually had the opportunity to visit your headquarters there. Incredible operation. You were doing a lot of stuff. I understand you're one of the early adopters of that whole, you know, lake base and everything. Yes. Yes. We We're We're We're big fans
[00:50] of Databricks and what they've done. They've They've helped our business a lot. And and it's it's one of the reasons why we we decided decided to step into the Lake Watch environment cuz we have a high trust relationship with Databricks, and it was a good opportunity to to push into an area that I think needs
[01:06] quite a bit of work. Awesome. So, tell us a bit about the scale of what you're doing at NAB and really how that relates to cybersecurity. So, with 8 million customers, trillion dollars in assets, uh principally a business bank, so we're
[01:21] the largest business bank in Australia, and so that's our bread and butter. So, we do have a consumer business and institutional business, but the small and medium-sized businesses is is really the heartland of our company. Um we've got 10,000 people in technology. And
[01:37] about 16,000 pipelines, 2,500 applications. There's a lot of legacy and a lot of complexity uh that we have to deal with. That is huge, huge scale, massive scale. So, what was like the aha moment that made you and your team say, "You know, we need security analytics directly on
[01:53] the lakehouse versus like the, you know, legacy or traditional SIM?" Well, I just think that the world is changing so quickly that that the legacy security data that that companies have ingested into those legacy SIMs um is just in today's world insufficient.
[02:10] You've got to widen the telemetry substantially. You have to look at customer data. You've got to look at incident data. You've got to look at login data from customers and and even our employees. All of that needs to get fed in to try and find anomalies against a much wider sphere than just the legacy
[02:26] indicators of compromise doing pattern matching in server logs. Which is still useful and you need to do, but I think you've got to do a much wider thing. So, it was a natural choice for for us to use uh you all for this. Yeah, it's incredible. I'm just my brain is just still going to the scale, um but
[02:42] you are in banking and banking's a regulated industry. So, you know, there's a lot of concern, uh you know, lineage, access control, um and unity catalog and you know, when I was out there, that was a big part of how that
[02:57] all fits in and want to hear more about that. Yeah, on uh we we do obviously we're uh we're a institution of public trust. Our customers trust us with their money and how they and they trust us with the information about how they how they spend their money. And so, it's it's really important that
[03:13] we get that right. So, we do spend a lot of time on on security. We spend a lot of time on data lineage. It's a very difficult thing to solve and when you have as many applications we do that some cases are many decades old. Um uh but yeah, you're some of the capabilities especially in the last uh
[03:30] call it 2 years that have come to light out of out of the Databricks platform has had made a huge difference in how we manage our data. Incredible. So, your SOC analysts, do they get governed access to the same like data that the data science teams leverage?
[03:45] Well, I I think everyone's got a different uh different role. Even our data science teams, some of them are very focused on credit risk models. Some of them are very focused on operational measures. Some of them are very focused on enabling AI. Some of them
[04:02] are going to be looking at security data. So, again, now the if you think about each of those role profiles, there's a Venn diagram of data and they're not all separate circles. There's a lot of overlap between the different data sources for the different use cases. And again, it's why we
[04:17] believe in the in the lakehouse is the right design. Awesome. So, last question is if another CISO or head of security engineering is watching this thinking about moving security analytics to the lakehouse, like what's one piece of advice that you give to them?
[04:33] Yeah, I I I think uh I I don't know if I could I just think just one. There's There's a couple I'd say. First of all, the nature of the people in security will need to change. Your traditional security person is a traditionally a SecOps person or a
[04:49] packet inspector or incident responder or um they tend to be more forensic. Uh in in tomorrow's world, increasingly they're going to be software developers and data experts. And so, that that that retooling of the the skills inside of security operation, I think is probably a huge change.
[05:06] Second is moving to a genetic is is much a uh technical problem as it is a human problem. So, we have to retrain ourselves and our people in how they think about what their job is. So, it's a you know,
[05:22] change is hard and and I think be prepared for change. Awesome. Well, I super appreciate you coming out. Um th- this experience that you've had undertaken, it's really incredible and I know the listeners are super excited to encouraging them to
[05:37] incorporate Lake Watch. Yeah, great. Thank you. Appreciate it, Ari. Take care. See you. All right, we're back. That was a great video by Patrick and Nav. Uh it's great to see how they're using Lake Watch. I'm I'm joined now by with the lovely Maria. And we've also got another special
[05:52] visitor with us today. Do you want to introduce yourself, Andrew? Sure. Sure. Yeah, nice to meet everyone. I'm Andrew. I'm the general manager of Lake Watch, which is our new security lakehouse product. I'm excited to tell you about it today. Excellent. Go ahead. quickly on what is Lake Watch. Yeah. Yeah, so Lake Watch is is built on
[06:10] the Databricks platform. It's a tool for security teams to help them find threats within all their infrastructure. It's in the SIM category, security information and event management. And that's kind of the core tool that the security teams are going to use to pull in logs from
[06:26] all the different devices throughout your company and then look for potential threats and find those before the attackers can do any damage. Okay. But SIMs I mean SIMs not a new thing. It's been around for a long time, right? So around for a long time, yeah. special about Lake Watch? Why should people, you know, consider Lake Watch
[06:41] when there's all these other SIMs that are already out? Yeah. And why now, maybe? I think so. Yeah. Yeah, absolutely. Absolutely. So a few things have changed. I mean, probably this isn't news to anybody out there, but the latest LLMs have made it really easy to find threats, previously unknown vulnerabilities, and potential
[06:56] potentially exploited as threats in just about any piece of software out there. And this new landscape means that the attackers are moving faster and at much larger scale than ever before. And historically, the legacy SIM players are just not able to keep up with that. You
[07:13] need to analyze a lot more data and you have to be able to remove the manual processes or automate a lot of the manual processes. You know, today, for example, if I was to log in here in San Francisco and then, you know, a couple minutes later somewhere around the
[07:29] world, that might be, you know, a signal that I might have been compromised. Yep. But that goes into a queue. A human analyst has to look at that and try to find out if that was really a threat or not. What we're doing is we're having agents automate a lot of that work. Very nice. Uh. And then in terms of, you know, we have
[07:46] a lot of practitioners watching and listening to us, right? So, what is the message we need to take to the security teams on why to use Lake Watch? Yeah. Yeah, yeah. Yeah, so the so the core of it is that security is fundamentally actually a big data big data science problem. You know, if you think about it, what these teams are
[08:01] doing is ingesting lots of log data, and there's an ETL job to kind of clean that data up, normalize it, then you have queries, you know, real-time streaming queries that are trying to find the potentially threatening activities within those logs. Uh, and then you have
[08:18] the investigation, the analysts that have to look at each one of those alerts and do a little data science problem. So, at the core, security is a data science problem, and you need a new set of tools in order to be able to to meet that need. So, the the ask for everyone who's a data AI expert is we need your help. We need you to talk to your
[08:34] security colleagues, and uh, they need your help to meet the coming threats. And it's it's great that we're using agents inside of Lake Watch to be able to basically find all these potential bad actors because uh, on the outside, you know, people are using agents as well to try to attack in, right? Exactly. Exactly. Exactly. That's that's
[08:50] the big that's the big new thing that's happening now is the agents are being used on the attack side, so we need agents fighting agents. Otherwise, you know, the defenders are moving only at human speed today, so. So, how much work do we have to do with Lake Lake Watch? Does it come with
[09:05] specific specialized agents already that people can start using on their logs? Yeah. Yeah, so this actually kind of gets to the the the origin story of Lake Watch as well, where we've actually had customers on Databricks analyzing security data for quite a long time. Mhm. And the vision for Lake Watch was really
[09:22] how do we make this a lot easier? Um, take the common patterns, the common workflows, the common data that security teams need to ingest, make it easier to ingest that. The common ETL jobs they want to run, help have agents that automate building a lot of those things. Um, and the same thing on the
[09:39] building detection rules and the response investigation side. So, yes, Lake Watch is an app that's built on top of Databricks, leverages the entire platform, but makes it really easy to use for the security practitioners. And there's really because it's, you know, built on on a platform that scales
[09:56] with all these with the the data being stored in open open formats, there's really no limit to how much data you can put in there, right? Exactly. Exactly. Yes, that's one of the one of the big value propositions here is historically existing legacy solutions, they are both technically limited to how much
[10:12] data they can pull in. Also, there's cost associated with that. It's often very high. It's the number one thing I hear from customers is it costs way too much to bring in all the data I need and so I'm filtering that data down, picking and choosing what I actually load into
[10:28] my SIM, and with Lake Watch, you don't have any of those restrictions cuz it's built on Databricks. Well, thank you so much for joining us, Andrew. It's been really great.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.