The Open Security Lakehouse: Building Agentic Cyber Defense with Lakewatch

Summary

Legacy SIEMs flood security teams with alerts while hiding critical context, and Lakewatch—Databricks' open, agentic SIEM—addresses this by ingesting all security data into a unified lakehouse normalized to the OCSF standard.
Lakewatch enables Detection-as-Code workflows, behavioral ML, and large-scale analytics over security data alongside business and application context, reducing false positives and bringing software engineering rigor to threat detection.
AI agents in Lakewatch autonomously gather context, analyze security signals, triage alerts, and integrate with tools like Jira, helping modern SOCs move from reactive alert handling to proactive agentic threat hunting.

The Open Security Lakehouse: Building Agentic Cyber Defense with Lakewatch

Legacy SIEMs are drowning security teams in alerts but starving them of insight. As threats grow more complex and attackers operate at machine speed, cybersecurity must be treated as a data and AI problem at scale. In this video, we'll walk through a Cybersecurity Intelligence Platform powered by Lakewatch, Databricks' open, agentic SIEM. We'll show how to streamline security data ingestion and automated cloud log collection, then rapidly normalize it into the OCSF standard to cut time-to-value. From there, we'll explore large-scale analytics, behavioural ML, and Detection-as-Code workflows that bring software engineering rigor to detection and response. The highlight is Agentic Threat Hunting with Lakewatch: AI agents that reason over security signals alongside business and application data, leveraging internal knowledge to add critical context, reduce false positives, and automate complex triage. Join us to see how modern SOCs can move beyond the traditional SIEM toward true security intelligence with Lakewatch and the Databricks Data and AI platform.
Talk By: Alysson Souza, Product Specialist - Security, Databricks ; Zac Davies, ANZ Field CTO, Databricks ;

Chapters

FAQs

What is Lakewatch and how does it differ from a traditional SIEM?

Lakewatch is Databricks' open, agentic SIEM built natively on the lakehouse, designed to overcome the limitations of legacy SIEMs such as ingest-based pricing, proprietary data silos, and limited AI capabilities. It ingests security data from cloud logs and other sources, normalizes it into the OCSF open standard, and applies AI agents for automated triage and threat hunting.

What is OCSF and why does Lakewatch use it?

OCSF (Open Cybersecurity Schema Framework) is an open standard for normalizing security event data across different sources and tools. Lakewatch normalizes ingested data through OCSF-aligned bronze, silver, and gold layers, enabling consistent analytics and detection logic regardless of the original log source.

How does agentic threat hunting work in Lakewatch?

Lakewatch's AI agents reason over security signals alongside business and application data, leveraging internal knowledge bases to add contextual understanding to alerts. These agents autonomously gather evidence, reduce false positives, and automate complex triage workflows—helping security teams respond at machine speed against modern threats.

What is Detection-as-Code and how does Lakewatch support it?

Detection-as-Code applies software engineering practices—version control, CI/CD pipelines, and peer review—to writing and managing threat detection rules. Lakewatch supports this by allowing teams to author detection logic using Genie and MITRE ATT&CK frameworks, import community rules from a marketplace, and deploy them through YAML-based configuration.

Full transcript

[00:08] Hi. Uh, good morning everyone. Thanks for for joining us today. I hope you enjoyed the conference so far. Uh, my name is Alison Sza. I'm a product specialist for security. I'm based in Australia. >> And my name is Zach. I'm also based out of Australia and I'm the field CTO for
[00:21] Australia at Data Bricks. Uh today we'll be talking a little bit about how you can start building an open security lakehouse and how that can help modernize the security operations center. But although the majority of the
[00:33] talk will be focused on the how to do it, I wanted to just start with a brief recap on why. So you probably heard on the keynotes that and have seen everywhere these days that how AI is
[00:45] accelerating how exploits are being used. So the time between a vulnerability being found and actually being used is reducing drastically and this is not something specifically that was introduced by mythos but it
[00:57] definitely accelerated a trend that was already underway. Uh on the defender side though uh we see a lot of different challenges. So first uh defenders have an incomplete view of their data state.
[01:10] So a lot of the times they miss important context and if you were in in the conference as well yesterday you have heard that AI doesn't have an intelligence problem it has a context problem. So missing data uh is a big
[01:22] one. The second is humans are getting flooded with alerts. Even the most capable socks don't have capacity to look at everything that is happening especially if they don't have capacity to drill down into all the this these
[01:34] different signals they're getting everywhere. So humans are fighting agents more and more because attackers are using them uh to be more effective. And the first is that even when you have the data, it's actually locked in in
[01:46] proprietary silos which makes hard to leverage it comprehensively. So what is the solution we are proposing? So the idea is you get all of
[01:58] your data and you have heard this before on the enterprise data space. uh it's not very sim not not very different from what you're hearing on migrations from uh legacy data warehouses but the idea is you get all your data you bring it
[02:10] into a cheap object storage uh you put that using open formats and now on top of it you can actually lay in uh your agents and of course these
[02:23] agents are the ones that are going to help extend your team and help you identify uh these threats uh more effectively So that's what that's why we introduced Lakewatch
[02:35] >> and hopefully you've all seen the keynote, but we thought it'd be better than us introducing it, we would let the product introduce itself. Um, so we're going to have Genie introduce Lakewatch and explain what's going on uh and what
[02:47] its capabilities are. Hopefully it's all legible for you in the back, but if it isn't, it's it's basically explaining that it's helpful for the data ingestion, the normalization, the detection, the triage
[03:01] and investigation component, all of that rich context in one spot with all of the agent capability as well. So, I'll give it back to Allison to
[03:13] continue on and explain kind of how we build this up. We'll expand it a little bit more on how these agents would work for the three key areas of the system. The first one being on the ingestion side. So how agents can help you bring
[03:26] the data in and normalize it and put it to a standard format that you can use. Then now that you have your data in the lake in an open format normalized, how you can use agents to actually help you start building detections and enriching
[03:40] the data with other data sets that you have in your lake. And finally, how you can use that on the response side both for investigating invest in investigating triaging and responding to these threats.
[03:56] Now, so we've seen all the components. We need to kind of break that down into each of the sections of the talk. So, we're going to go and lay the foundation now and we're going to go and see how we actually pull data into the lake and use
[04:09] Lakewatch to do that. And we'll see as we go through the talk, we'll talk about the detection enrichment and using triage within Lakewatch. And then we'll talk a bit about custom agents at the end. So I'll let Allison kind of walk you through a demo in a little bit. But
[04:23] what we need to talk about and understand is Can I have the cooker, please? Yeah. For this part of bringing the data in, it's not just simple enough to say, hey, let's connect up a data source. we need
[04:36] to have it not just in an open data format but an open structure as well. So if you aren't familiar with OCSF, it is the open cyber security schema framework. And the notion is is that there should be a standardized way to
[04:48] store all of the security formats. And there's a pretty big kind of collateral effect of having everyone do this because it means that if someone finds a useful way to structure a detection on a particular source and everyone's got an
[05:01] open way of schematizing their whole data, then they can share it and just use that anywhere. So it'll be in an open format. You can access it from anywhere in the catalog and you'll have it in an open structure and you can use all of the same detections and tooling
[05:14] and make that much more powerful. So Lakewatch made a decision to use OCSF as a standard way for us to structure data and put that as a common data model. So you've got more openness.
[05:27] So if we go and do a little bit of a demo and show you how to bring in data. Now I'll let Alison kind of guide us through this one. Um so here here we are in the regular
[05:39] kind of data bricks workspace and I have created a catalog. Uh I'm hope hopefully everyone is familiar. So I have a cat catalog with all my security logs in the space and I we generated this using a a
[05:52] synthetic application. So we can simulate logs. I have different tables with different logs and I also have a volume with raw data files. So I have some cloud trail data here and I put a
[06:04] simple JSON file. So you can see it's just the regular kind of format just like AWS would export to your bucket. We haven't made any changes. So it's a complex JSON with a lot of different lines and a a lot of different fields.
[06:17] Uh and I have a subfolder there with many more similar to the one that I have. So I copy this and if I have lick watch enabled, the way you would access it is through the app switcher. So it appears as another application there.
[06:32] So this is the overview page. But in this case we want to start by ingesting a new data source. So we will go into the data source page. I already have a few data sources
[06:45] preconfigured here. As you can see there is a little list there. But for this one we want to now add a new data source. So there's a few different ways of ingesting data. Um you can see a little bit at the bottom right now there's a
[06:58] few different connectors where we would go directly to the source system and pull the data in if it's not already kind of on cloud storage. But in this case we have the data already in S3. So we are actually just going to load it
[07:10] from there. So if I give it a name I'll paste the location that we copied before from the volume. For this one I'm just going to load all of my data. And the first thing we'll do is we use the auto
[07:22] configure to identify what the data set is and what the structure of the files are. So it will take a few seconds. Basically it's loading a sample file some sample files understanding the structure of the
[07:34] folder. Uh it will identify critical things like timestamps. Uh and after a little bit it will actually analyze the data source and that will be displayed to me as a sample so I can validate that
[07:46] it loaded it correctly. So, it's loading now the preview and there we are. So, you can see it added some metadata, but essentially this is my raw JSON file. It's a single
[07:59] column with all my data in it and just some metadata. Like I said, that will help Lakew Watch identify new new files as they arrive. Now, I'm going to click next.
[08:11] So, now I have here my bronze, silver, and gold layers. Right now I only have my data into my bronze which is the one that we just loaded and we saw but I want lake watch to help me normalize this. So it identified that this is a
[08:24] cloud trail data. So it suggested me to use the cloud trail f uh parser. And essentially this is a preset that will now map all events from my bronze into a silver table that contains cloud trail
[08:37] specific data. And finally, at the gold layer, you will see that it's mapping those to the column to the schemas and column names that are applicable to the OCSF
[08:50] framework. We'll look a little bit into the advanced options in a second. But now that it loaded, if I take a look at how my silver data looks, you can see that it has all the transformations there that it has done from my bronze to take it to silver. So this is again how
[09:04] you would do but you can off uh trans change if needed. But at the bottom you see that now I have uh my my data there. I can make the transformations using the advanced uh column.
[09:18] If I look at my gold layer it's same thing. It it's taking out from my silver applying additional transformations that map the events back into my OCSF table.
[09:30] Um like I said you do have some advanced options. You can make additional transformations. You can enable checks. So you can validate the health of the data that is flowing in. Um and you can adjust of course the the frequency of
[09:44] the data. Everything is created as a yl file. This is important for a CI/CD pipeline. So you can incorporate that uh in your workflows as well. So I'm just going to hit uh create data
[09:56] source. And essentially what this is doing is starting a process of loading this data and it's going to make all of these transformations. Uh but this is not locked in the system in any way.
[10:08] After this is done, all of this flows back into your catalog which means that you can continue to access it in any data bricks tooling or any tooling that can read open formats outside of data bricks as well. So you have my bronze uh
[10:20] my silver and gold schemas with all the tables that we showed there for that particular data source.
[10:33] So the thing that we've shown right now is doing the ingestion through lakewatch. Data bricks is obviously a broad platform and over the years we've built a lot of primitives to do a lot of different data ingestion. So we can use things like zerobus spark decive pipelines. We can use lakeflow connect
[10:46] to bring in different SAS applications. All of those capabilities are still there and can still work and plug and play with components of lakewatch. So, we're not going to go through all the different ways to pull in data into Lakewatch today, but we've shown you one of them, and we can show you and tell
[10:58] you that it's all composable within the existing platform, and it makes all sorts of wonderful things quite possible. So, now that we've seen how to get the data in, we need to start actually doing
[11:10] something useful with that data. We need to be actually finding needles in the haystack, finding those things to build cases off and do investigations with. And we're going to focus in here on the
[11:22] detection authoring in particular and we're going to see how we can build up a set of those with Genie. And one thing I do want to highlight is that even though we're releasing Lakewatch this year and we're building a
[11:34] whole bunch of capability to make this easier, customs have been doing this for a while. So at this conference in the last couple of days uh there have been talks from Atassian and Adobe and I personally worked with Atlassian to help them do this a number of months ago last
[11:47] year without Lakewatch. So all the capabilities in the platform exist and again just to reiterate the primitives of not just the ingestion but building up capabilities to do transformations and complex machine learning modeling to
[11:59] do more nuanced alerting is possible. So I'm going to give it back to Allison who's going to do another demo uh of looking at how we can build these detections in Lakewatch. >> Yeah, we'll focus on this one on how
[12:11] Lakewatch uses the same primitives but simplifies that process for you when you're authoring them. So we're back here to our Lakewatch UI and this time we are going to focus a
[12:24] little bit on the detection rules. But before we go in author one, I want to quickly glance into the marketplace. So we do have a few packs here that helps you both on the data source presets to normalize the data like we showed
[12:36] before. We have some sample notebooks to help you in investigations and we also have some sample detection rules that you can easily deploy directly into your environment. So you can have a list you can see a list there but if you click
[12:48] you have more details like the proposed severity and fidelity the minor attack mapping and the rule itself that you can customize. So if I import one of these rules essentially it will appear to me uh if I go into my
[13:02] detection rules. So it will automatically appear here and now I have imported it. Uh you can see all the details as I displayed before. If I click on it you will see the same SQL u
[13:14] logic that we we showed you. Uh you'll see the metadata attached to it on the left hand side. Similar to the uh data source preset. It also generates a YML file to help with CI/CD and I can just
[13:26] turn it on and then it will start being executed. Now what if I wanted to create a new detection rule that is not on the library? We'll do this using Genie. But first let's do some simple questions
[13:39] just to understand what it can do. So I'm going to ask what data sources do you have available? As data sources are ingested, they get automatically added into this genie room. And unlike the gen room and the
[13:51] data brick space, this one is tuned a little bit into the investigation and detection engineering. So it understands the context of what an analyst, a security analyst is doing here. So it found all my t my tables. It found the
[14:03] data sources I have. It identified my gold tables in OCSF. I'm going to ask him about API activity. Can you give me some insights? Again, the agents are really good at going into the data source. It will
[14:16] understand the structure of the table. Um, it will do some distribution of the the types of events that exist many times. Uh, here is asking me for permission to do it. So, I'm just going to authorize for the current thread. So,
[14:28] it can continue to do this iteration. But essentially now it will go understand the format again, run some queries just to see what is the data like, what do I have in my tables? Uh, it's looking for my top API operations
[14:41] for example. So it can give me a summary of what are the most common API activities that I have here and then it summarize everything uh in the bottom for me. You typically like LLMs do you
[14:53] will propose some next actions. So it's very good at proposing something like that. But I want to change a little bit and I want to ask him if he has additional context. I want to ask can you provide me guidance on my attack?
[15:05] This is not on a specific table but it's built into the system as a tool or capability. So as you see it's getting the TTPs and it's understanding and you will have that into the context as well of this
[15:17] conversation. So it's a brief summary. Yes, I do. I can help it. I can help you with it. Here are some of the uh how it relates to your environment and do you need any help to explore anything
[15:30] further? So yes, I'm going to ask help me build a detection. It has the context of the analysis we're doing on the API activity and now it has
[15:43] the context that I was asking about mitro attack. So it's again loading all of that and it's going to make a few proposals, a few different rules. I'm going to pick one of those. Let's say let's build number four denied session
[15:56] token requests. So it's going to start building a SQL query that would run against that
[16:08] particular detection type. The same thing the LLM is running the analysis. It's understanding the events. It understands the techniques and what it should be looking for and it's going to start executing some SQL to understand how we would catch these type of events.
[16:25] Now, this usually takes a few a few seconds, a minute, but then you will come back with a good understanding of what it is. Here's what I found. There's a quick description. This is the query that I ran to find it. Would you like me to create this for you? And you can say
[16:38] yes, please. Uh, so it found it created some metadata, found the query. You can of course ask it to change it. And it asks, do you want to create? If you authorize
[16:50] it in a second, it will prompt me for an authorization. Can I go to the to the rule creation page? Yes, it automatically takes me there. And here is the same one that you would get if you were manually creating a rule. But of course, this time we are
[17:03] asking Genie to do it for us. So now I am here on my rule builder uh UI. Uh it's going to again run some tests. It's going to add the metadata that it needs. It's going to give it a name, suggest
[17:16] some fidelity or severity for the rule that I can ask it to modify. Uh it's going to add minor attack mappings because we know the techniques and sub techchniques we were looking for. So it pulled all the details. It gives me
[17:30] again another summary and if I it will ask me do you want me to apply this? Now you can ask it to modify it.
[17:46] I'm going to allow it to authorize. And you can see that it just populates things for me. Here again, you have another chance to review. Just make sure that everything that it populated is as you expected. All the fields are populated. All the metadata is extracted,
[18:04] my severity, the objective of the rule, the schedule, which of course you can change according to your needs. And finally, when you hit create, it will flow back. And if I search here for this new rule that we just created,
[18:34] So we just went through loading data into the lake, putting it into an open format. We explored how you can use data bricks primitives like notebooks and SQL directly in the lake because the data is ultimately inside your unit catalog. But
[18:46] we showed you how the agents within uh lakewatch itself helps you generate those detection rules mapping them to the frameworks that you want to align to. Now we are going to go a little bit into
[18:59] the last pillar. How do I respond now that I have multiple detections running? How can I go and use agents to expand my team and be able to look at more threats that are coming into my environment?
[19:12] So, one of the issues that we hear constantly when talking to teams today is that even the teams, the largest teams in financial services that have the biggest budgets to have security teams, they struggle. Uh, and the reason
[19:25] is there is so much noise coming through. If you look at like the high severity cases, 95% of them are false positives or benign positives. But what they miss is the capacity of looking at
[19:37] the lower severity that maybe will carry good signals that can be used for a potential threat. Uh and the reason is really they need to suppress that because they need to focus on something, right? And most of the times they need to either ignore or suppress that.
[19:51] Uh so this means that the more alerts that we have, the more logs that we have, we need to kind of focus extract all the noise and focus only on the things that matter. But the teams are constrained today with the capacity to
[20:04] do that. So we are going to show a little bit now how agents can help with that triage and identifying what's noise versus what is a a strong signal.
[20:16] So, we'll go back into Lakewatch here. And we're kind of going to go and look at the security cases section over here on the left, and I'm going to go and filter down to just those that are
[20:29] informational. We're going to get rid of all those important and critical, and get into one of these cases. And we can see here it's denied IP, attempted access uh three times. We're just going
[20:41] to ask Genie to summarize the case. And whilst it summarizes that, we can see the the notable we can see a whole bunch of this metadata about what has been escalated and raised in the case. And
[20:54] Genie is going to go and give us that summary. And you can see we've also got these abilities to go and categorize things ourselves. Actually, this is me. It's saying I failed to log in three times. And I
[21:06] believe that I often forget to use my VPN in Australia. So I think we can classify the one as false positive. But I'm going to give it to Allison and he's going to go through a bit more of a complicated one that we looked through earlier and we prepared.
[21:18] >> So here we have a similar alert, a similar detection, but this time uh it's using that generated data. And now it found something and it asked me, do you want me to investigate this user? And because I have so much different data
[21:31] sets inside this lake, agents are really good at reasoning and going to the next and the next and finding more and more signals. So you can see it's looking at a lot of tables. It's make making queries. It's saying, "Wow, this is now
[21:43] very concerning. This user is found doing a lot of odd things that they weren't doing before across a whole bunch different of of different data sets." So, same alert, but with an investigation that went a little bit
[21:56] deeper, leveraging the agent capabilities. Now, I'm able to actually find signals that are stronger than just user failed to sign in due to forgetting to log into the VPN. this user is actually trying to download sensitive
[22:09] data from buckets. It's sometimes it's failing. So I'm suggesting that you escalate this to critical. So different from Zach's message, I now have a high severity ticket with some comments of the investigation. And in this case, I
[22:22] have exported this to Jira for my case management. So the same content is now as a case in Lakewatch, but also in my case management system. So someone can triage and be assigned to it.
[22:38] So we've kind of walked through I will take that. Yep. We've kind of walked through kind of the whole end to end now as it sits within Lakewatch and we've seen we can go through and investigate quite powerful uh capabilities and we can combine a lot of that data and this
[22:51] is why context matters. So kind of just to put a picture in everyone's mind of what we've seen so far. we've kind of built up and pulled a whole bunch of data into lakewatch and we showed you how to do that for the cloud trail data but we've got a whole bunch of extra
[23:04] data here as well and we've pulled that through and normalized it from bronze to silver to gold using the OCSF schema that lakewatch supports out of the box and then we've gone through and we've seen how we can build detections and we
[23:17] can have those raised into cases at various levels of severity in this case we looked at one that wasformational and we've seen that we've got the ability to have Jira automatically synced with Lakewatch and pushed those
[23:29] cases in and all those updates automatically. We then went through and used Genie to do a deep investigation and it pulled all the data from the existing data sources at the various levels when it
[23:42] needs more granularity. And then we've also got the ability because we're in the lake and we've got all that enterprise data available. It's able to see that we've got relevant financial data to this case and it can pull that in and contextualize more than you would
[23:55] have been with just the security logs alone and then it's been able to make a decision that we probably should raise this to critical but it's asked for our approval to do so and human in the loop we've escalated it and then that's also been synced back to Jira with the case
[24:08] notes from Genie. So we've kind of covered the out of the box wakewatch capability and if we want to push the envelope a little bit further and go into the full set of
[24:21] capabilities that data bricks offers what can we do and I think one of the things that I'd like to talk about is going through and how do you build your own agents and scale them up to do things more autonomously than rather sit with lakewatch and genie in the loop
[24:33] right this doesn't scale having someone go through and do case by case with Genie it's faster than what you do today but we want to do something more powerful So, I guess we should cover some basics and I'll go through this pretty quickly,
[24:46] but what is a security agent? It's it's really just an agent that's got all of that context and the ability to interact with the systems and tools and optionally take action through those. Now, there's kind of a bit of a gradient
[24:59] to how these systems work and operate. And a lot of the time with these AI systems, they started out as just chat bots and they've escalated pretty quickly into getting more abilities and to look at systems as readon, go and get various tools to interact with more
[25:11] narrow permissions. And then it's okay, they've got more existing permissions, but the human has to be in the loop to approve. And then it's okay, they can do things autonomously, but maybe only things that can be reversed and unwound. Uh lake based branching is a good
[25:24] example of that. And then maybe we want them to do high impact decisions automatically. And when I'm talking about the next parts of the talk, I'm not talk about not talking about starting here. Uh we would never want to
[25:36] start here. You're almost always going to start here. And what we've been seeing with Genie is it's a bit of somewhere in between, right? We've been engaging with it. It's been asking us to operate with human in a loop. So breaking down what actually is in an
[25:50] agent will help us understand how to build one effectively. Um, first you're going to kind of propose like a goal like, okay, I want you to investigate this case is a pretty simple goal. U, I'm not going to go down into the model
[26:02] too much, but you probably want the the latest and greatest. And then you're going to give it some really clear instructions. So these are kind of the system prompts like, hey, you're a security triage agent. Go and do this. And these are the ways we like to
[26:14] operate internally. The most important thing is context. And we've been talking a lot about context the last two days. things like hey what are the details of the alert you actually want me to investigate how do you handle that internally with the existing runbooks
[26:27] and what are the existing patterns for this user and how can I go and find that and to get that context it'll probably have to have access to tools so it might be that it's using genie as a tool to go and do more analysis it might be that
[26:40] there's more existing cases in Jira that are similar and have got good learnings that we can take from or there might be a need to go and look at slack or exist uh invoke an existing MCP server. And
[26:52] then there's the control loop. This is kind of all of the magic that turns an LLM into an agent. I'm not going to cover this now because it goes beyond the scope of this talk, but that's kind of just a bunch of conditional branching logic. And then the guardrails, which is
[27:05] also very very important. So this kind of covers things like approval gates, sandboxing, which we've just released capability for in the platform, lease privilege access, and how you scope down tools. So now that we understand we need all of
[27:18] these components, we kind of need to go ahead and do it. But before we do it, there's still one more thing, which is you can't just go into production with an agent on vibes alone. We need to have a bit more of a quantitative way of
[27:31] approaching this. So eval critical. So evaluations are a way that we can measure the quality of a system. This is not anything new for ML solutions or AI solutions, but I just
[27:43] want to reiterate how important this is because if you don't have a way to direct yourself, you won't know if things are getting worse or improving. So for example, if you take an existing flow of like asking an agent a question and hey, you get a response, there are
[27:57] telemetry events that are emitted by agents which we can harvest and collect. And with the responses themselves and the telemetry, we can start making assessments of how the agent has behaved. Did it get the severity right?
[28:12] Did it have the relevant assessment based on the correct information? And were the correct approvals raised at the correct times? Now, these are just examples of what you could ask and how it could be structured. And then the question you might ask is, well, okay,
[28:25] it's fair to ask those questions, but who gets to decide if it's right or wrong? Uh a good way to do that is to have a ground truth data set of maybe existing cases that humans have reviewed that you can set the agent loose on and compare them. Uh or you can use things
[28:37] like LLM as a judge where you have a separate LLM or a series of LM look at the existing data and make an assessment or a mix of both. Now the good thing here that I want to point out I think is very important is that your existing
[28:50] security workloads are already producing the data that is required for these eval. So you're not starting from zero in any of these situations. So we know what we need to build an agent. We know that we need to kind of
[29:05] go and do these evals, but we've also then got this decision to make around do we want to make a generalized security agent or a specialized security agent. So this is kind of fairly obvious I would think, but I will cover it. And
[29:17] the specialized agents are obviously more narrow. They've got way less scope and context. You're giving them a very well- definfined task. And the way that task gets kind of broken down, we can talk about in a moment. But you're also
[29:30] going to give it just the tools that it needs like least privileged level of access for permissions, but also just the minimum amount it needs to do that it's function that you've defined. Now, the good thing about having very
[29:42] specific agents that are specialized is that evaluating them becomes very simple because they've got a very well- definfined task. You can understand how they should behave. If you have something that's very broad and wide
[29:54] ranging, it becomes very difficult to define a very good eval. So you'll find that you're probably going to need to do a lot more work on that if you've got something that's general. Specialization doesn't need to be task
[30:06] specific. So there are a lot of ways you could break this down. These are just examples. You could do it by domain, by the type of signal that you're looking at. You could do it by different types of entities in the data. And you could do it by various stages of the workflow.
[30:19] And you're ultimately going to have probably a mix of these. You might have one that writes case notes, but also one that looks very specifically at how cloud trail behaves or particular types of MITER attacks. At data bricks, we like to look at the security domain uh
[30:33] quite a lot, but we do a mix of everything. And the topology of how your agents fit together is going to evolve over time. You're probably going to start very simply and have one maybe broad agent
[30:45] and very quickly probably decide to split that. And that's probably very natural, but also it's something to be aware of that you probably do want to consolidate where it makes sense. And you will have a lot of the data based on
[30:57] that telemetry I gave you that you can mine and you can use Genie for that or any other tool and you can ask it to look at patterns of how things are working and you can say, "Oh, maybe there's a clear set of tools that are
[31:09] shared between these specialized agents and there's not too much dissimilar between their context. Maybe we should consolidate." uh there are various signals that you can do that. Ultimately, the one takeaway here is that you're going to usually end up with
[31:22] a supervisor and a bunch of specialized agents in various ways and you will need a catchall generalized agent for when there isn't the ability to rely on a specialized agent for a task at hand. So, what are your options to build an
[31:35] agent now that we've gone through all of this? Uh you can just go and get one of these frameworks that exist and are very popular. Um, OpenAI and Anthropic have their own and then you've got other open source options like Langchain. Uh, but
[31:48] there is a whole suite of these and there's one that comes out probably every other week or we are at a data bricks conference after all and we do work for data bricks. We could just use agent bricks. Um, so what does that look like? I will kind of walk us through a
[32:01] little bit of a very quick demo on how we can use agent bricks to build these agents. So, let me switch in here and I'm going to go into agents on the nav here at the bottom left. And we're going
[32:15] to see that I've got two already here. And if I go click create agent, I can see there's an option to create a supervisor, a knowledge assistant. We've also got these options on the left to do like parsing, etc. And we can just do BYO uh with code. If I click into this
[32:29] one here, it is a knowledge assistant. And what this essentially means is I can give it a whole bunch of documents or vector search indexes and it will allow me to kind of have that kind of materially uh accessible and it'll do a
[32:42] whole bunch of nice things. I haven't given it many instructions if none none instructions. I'm just going to ask a very basic question. How do we assess um let's talk about kind of if people have
[32:54] been subject to fishing attacks? So how do we respond to users being subject to a fishing attack? and let's see what it responds with. Now, all I'm expecting here is it to reference the existing runbooks that we have in the volume that
[33:07] I've attached. And we're going to then later take this knowledge assistant and combine it with an existing agent. And I can show you how easy and simple that is. So, it's responded here. We don't need to look too much at the content
[33:19] other than you can see it's clearly referenced and you can see what it's talking about. And if I click on the sources, you can see which documents and which chunks that it's pulled that from automatically. We haven't really had to do anything other than link those PDFs to the knowledge assistant. Now, I can
[33:32] further improve the quality of this and agent bricks takes a lot of that out of the kind of your hands and what you have to do is you say, I've got some examples of questions that I would like to provide and here are the guidelines that
[33:45] I would expect of a response and then there are some very clever techniques that we use from our research team that have inbuilt into the product. So what you can do is you can also open up the experiment view in MLflow and we've got a whole bunch of stuff here
[33:58] and I'm going to show you the telemetry that these agents emit. So this is uh open telemetry trace and I'm going to click on one of these here. Uh this is when I asked a question how do I respond to data Xfill events. And if I click on
[34:10] the timeline we can see kind of in this case it's a very simple trace. I can click on the documents. I can see all of those outputs that were returned. I can see the final response and it is basically a nested uh structure. Now
[34:23] this is just data that we can mine and use for quality and evaluation. But at the moment I'm going to go back into my supervisor agent here and you can see that this has been combined with a bunch
[34:35] of tools. It's got access to Genie uh and a few other things. I'm going to add access to search the web and you can see it's already got GitHub and enabled. And I'm also going to give it the knowledge assistant that we just were playing
[34:47] with. And it's going to ask my permissions to run as myself. So I'm going to authorize it to run on behalf of my identity. And it's also got the ability to kind of just explain what it's got the ability to do. Let's just
[34:59] retry that. And it's just going to be able to tell us, hey, yeah, I've got access to a whole bunch of UC functions. I know how to look at Jira. I know how to look at Confluence and GitHub. And I know how to
[35:11] search the web. Now this again emits that same open telemetry trace uh and you can use that to kind of mine information and also assess how it behaves. And again if we
[35:24] go to the experiment UI, I want to point out a few other things as well. So just to show you that every time you invoke it, if I go to the traces, you can see um they're all here. I just
[35:36] asked it the basic question twice. And again the the trace details are all there. And you can see in this case it's got tool evaluation calls. So one of the things I mentioned were eval. So you can use MLflow's native
[35:49] capability to create judges and define either functions or kind of LLM calls to do that. You can also do labeling sessions which is a very powerful way to involve experts that have been dealing
[36:01] with this kind of data for a long time and have them evaluate the outputs of these sessions and questions and then you can use that to feed back into the quality. So now that we've seen how to build the
[36:14] agents we should talk about how do you actually trigger these? I've shown you how you can work in Lake Flow, sorry, Lake Watch, and it does all the nice work with Genie in the loop. I've shown you
[36:26] how I can build an agent with agent bricks, but I actually haven't shown you how you can scale this out at all. So, Lakewatch actually has all of the data that's available accessible in tables. So, there's an internal table called
[36:38] Notables. It's just there and you can access it and do whatever you'd like with it and it's updated in real time. So, you can use what's called a table trigger in LakeFlow. So every time that table was updated, you can have it trigger an event and you can have
[36:50] backoff periods if you want to trigger it only every five minutes. But this means you're not doing it on an hourly schedule. You're waiting for data to arrive and you're doing it when it makes sense. And then you can just write those results out anywhere. So for example, to
[37:02] a table and that workflow will just communicate with the agent and it's very very straightforward to set this up. Alternatively, you saw that we've got the ability to push this into Jira automatically.
[37:14] So Lakewatch will push that into Jira and you could use an automation on the Jira side whether that's through rovo or something else. It can communicate with the agent and then you can also just talk to it yourself which doesn't scale
[37:26] very well but you can still do that. So I've shown you a bunch of capabilities with the agents and kind of this is a slide that talks more broadly about the existing capability that we've been talking about at the conference. So
[37:39] you'll see here that the sandboxes are new. We've got a whole bunch of basic capabilities with memory as well. All of this is in the platform to make it very simple and straightforward to build very powerful agents and it can just be a
[37:51] simple few clicks to get started and as you want to mature a strategy and get more and more serious you'll find you'll spend more of your time on like the evals and the observability and the monitoring side which we've got labeled as optimize.
[38:04] So I'll hand it back over to Allison to close us out. >> Yeah, thanks Zach. Uh I'll a quick recap. I know we covered a lot. We wanted to give an overview of the end to end flow. So we started with the loading the data foundation creating the lake
[38:18] putting it to a normalized format. Then now that we had the data in, we showed you how you can start creating detections and doing analysis on the data directly using the using the assistance. Finally, we use the agents
[38:32] to help us investigate and find more patterns, finding new things and even exported to an external system. The last step was now that I have all my data in the lake, I have all the capabilities in lake watch, but I'm in a very powerful
[38:45] platform like data bricks, how can I extend it? How can I augment it using my own deployments, creating my own agents for my workflows? So, we started talking about challenges. So, by building an open security
[38:57] lakehouse, uh you address that by analyzing all of your data. You're not constrained by expensive storage and ingestion fees and licensing models. You bring all of that into the lake. You
[39:09] start fighting agents with agents. You have your automation agents. You can create your own. You can customize them and you enable an open ecosystem. Of course, uh as I said, we're a data bricks conference. We think data bricks is the best place for you to run all of
[39:22] these, but anything that reads iceberg, anything that reads delta can connect into that and build your your own outside as well. So that's the beauty of an open ecosystem. Uh yeah, thank you for your time. That
[39:34] was the last slide. I hope you use it you was useful and yeah thanks

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.