Mo’ Agents Mo’ Problems: How We Simplified Lakehouse Threat Hunting as a Service at Scale

Summary

Booz Allen built an agentic threat hunting platform on Databricks that normalizes 20-plus data sources via Lakeflow DLT into OCSF, with specialized AI agents that autonomously generate hypotheses, investigate the lakehouse, and enrich indicators of compromise via MCP tools.
A critical safety constraint requires agents to query pre-aggregated views rather than raw telemetry directly, keeping billion-row datasets out of the LLM context window while preserving full investigative depth.
After scaling from one agent to 12, the team encountered coordination overhead, context fragmentation, and inconsistent findings—lessons that led to the orchestration patterns now used in production at petabyte scale.

Mo’ Agents Mo’ Problems: How We Simplified Lakehouse Threat Hunting as a Service at Scale

Security teams face an impossible ratio of millions of events, thousands of alerts, and a handful of analysts. Most hunt reactively or not at all. We built a threat hunting platform on Databricks that flips this equation. Our architecture spans the full stack: Lakeflow DLT normalizes 20+ sources into OCSF, billions of events are governed in Unity Catalog, and detection pipelines form the foundation for specialized AI agents that hunt autonomously. These agents generate hypotheses, investigate the lakehouse, enrich IOCs via MCP tools, and produce findings without prompting. The critical constraint: agents never see raw telemetry. They query pre-aggregated views, keeping billion-row datasets out of the LLM context while preserving investigative depth. You'll learn how we architected safe agent-to-lakehouse communication, orchestration patterns for autonomous hunt cycles, and detection at scale, while hearing about our service delivery at petabyte scale.
Talk By: Mike Saxton, Director Cyber Defense, Booz Allen ;

Chapters

FAQs

How does Booz Allen's agentic threat hunting platform work?

The platform normalizes over 20 telemetry sources via Lakeflow DLT into the OCSF standard, with billions of events governed in Unity Catalog. Specialized AI agents then autonomously generate threat hypotheses, investigate the lakehouse, and enrich indicators of compromise using MCP tools, producing findings without prompting.

Why are agents prevented from querying raw telemetry directly?

Sending billions of raw telemetry rows into an LLM context window is impractical and expensive. Instead, agents query pre-aggregated views, keeping the LLM context focused on relevant summaries while preserving the investigative depth needed to identify real threats.

What challenges did the team encounter when scaling to 12 agents?

Scaling from one agent to 12 introduced coordination overhead, context fragmentation across concurrent agents, and inconsistent findings that were difficult to debug. The team found they had effectively built a distributed system of microservices, leading them to adopt structured multi-agent orchestration patterns.

How does Lakebase support long-running threat hunting agents?

Lakebase provides persistent memory for agents conducting multi-step hunts that span hours or days, storing intermediate findings and context so agents can resume investigations without losing state. This is critical for threat hunts that require correlating signals across a large timeframe.

Full transcript

[00:08] Good afternoon everybody. Thanks for uh being here today and um for some of you taking a a step aside from the the France Sagal game. Um please feel free to watch it on your phones if you're streaming it. But um thanks so much for
[00:20] taking the time to be here. My name is Mike Saxton. I work at a company called Booze Allen. I'm our director of intelligent defense um in our commercial business. So, for those of you that may have been around the federal landscape, you might know Booze Allen as an
[00:33] advanced tech company that's building solutions for the federal government. What a lot of people don't know is we have a commercial team of a few hundred rock stars, which I have the privilege to work with. And we operate about 1,000
[00:45] incident responses and threat hunts a year in that commercial team. So today this is really a talk on our journey to building Agentic threat hunting uh within the data bricks platform and a service that we're delivering to our
[00:58] customers um and some of the things that we learned along the way. I hope that some of you can take back to your organizations and maybe implement some of this um if you're looking to automate your threat hunting capabilities. But
[01:11] ultimately we'll talk a little bit about our journey through this. all centralized or standardized kind of on how we think about threat hunting and then we'll talk a little bit more about the platform frameworks and how we built it. Um
[01:24] agents uh to everything nowadays. Um and like the title says, mo agents, mo problems is not just uh an ode to a great rapper, but also some of the challenges that we found um when we were
[01:37] building a threat hunting platforms and some of the challenges that come from agents uh at large. So in doing this, these were some of the challenges that we first started off with. We ran into issues with coordination overhead,
[01:50] verification problems, context fragmentation. We had multiple tools calling multiple or multiple agents calling multiple tools. Um it was really hard to debug and we had a lot of inconsistent findings. I will say this
[02:02] started about 16 to 18 months ago when we began building out these agents. But ultimately um what we realized is we started from one agent which we outgrew very quickly into multiple agents and
[02:15] eventually I think we got up to 12 total and things really started the wheels kind of started to fall off and we realized what we had actually built was just a distributed system of microservices. It was really hard to debug and keep running and have the same
[02:28] quality that we would want as threat hunters or security practitioners into understanding what the agents are putting out. And so the problem isn't necessarily that agents are bad. The problem is that if you don't manage and
[02:40] govern these agents in a way, you just end up with a distributed system that is hard to run. And for those of you that are working in socks today that already have to maintain a SIM and a forwarder and a collector and everything else that
[02:53] goes along with it, the last thing you want to do is debug some agent application microservices on top of that. But also we did some research in looking at what is the best way to do this. agents are really popular. They're
[03:05] growing. Everybody's running them in their business to like their open claw instances at home. And they're they're kind of changing the way that we're doing a lot of things just outside of security. And there's some standard buzzwords and marketing stuff that we
[03:18] always hear about hundreds and thousands of agents and multi- aent, etc. But there's also some really interesting emerging research that's coming out on how agents are being deployed. Um and primarily when we look back through this
[03:31] we found that um multi- aent systems are really come from taskspecific challenges that you're trying to solve. So if you need to parallelize a task, you can have a swarm of agents really
[03:44] handle it and it does a great job at that. When you start to say in like a threat hunting context of I need to build a hypothesis, I need to conduct and build all of my queries. I need to search for the data and now I need to give a detection at the end of this over
[03:57] to my incident response team. Uh that's where the challenges start to come in and I'll talk a little bit about that. But ultimately, did I lose the mic? Okay, cool. Um but ultimately um a lot
[04:10] of the failures that we realized and we initially went through came from the coordination and the verification breakdowns. And so there's this garbage in garbage out approach where we had some bad hypotheses that were created that ultimately found their way
[04:23] downstream into bad detections. And so this is a story and a journey about how we we built u what we did to help our customers. But first let's talk about threat hunting. Um I'm sure many of you if you come to a talk on threat hunting
[04:36] you know what thread hunting is. But no matter how many times I talk to different customers or different people in the industry about threat hunting, I always get you it's one of those ask people get seven opinions on what threat
[04:48] hunting is kind of perspective. So I'll give you our impression of what threat hunting is. Whether you choose to borrow it, that's great. Um if you all are doing a form of threat hunting that you feel is great today, that's fantastic,
[05:00] too. But I think this is relevant into how we talk about what we're going to go into uh with the rest of the agentic approach today. So threat hunting for us is a very behavioral driven process. Uh
[05:12] we do have we work with a lot of people that think that taking IP8.8.8.8 and running it through their SIM is a threat hunt and if that's what they feel comfortable that's great and so we don't want to derail that. But our focus is a
[05:25] lot more on the behavioral side of things. So not going to give a talk today on the pyramid of pain, but if you can think about at the top of the pyramid of pain, we're really focused on behaviors. Things are hard things that are hard to represent in a single detection or a um a single join or some
[05:40] sort of SQL query is really what we're focused on. Uh but more so it's understanding normal verse change as well. Every network is different. Every environment is different. And so we don't focus so much on trying to find
[05:53] behaviors that are unique within we we focus on trying to find behaviors that are unique within that specific environment. And I'll go through our methodology here in just a little bit. But ultimately we really want to know
[06:05] and understand um what normal looks like on the network before we even try to get into abnormal. Testing hypotheses and then turning unknowns unknowns into knowledge. And finally we really view
[06:18] this as a threat hunt. asks, "What could be happening that we do not already know how to detect?" So, some folks like to drive their threat hunting practices based on the MITER attack matrix or detection coverage or anything else
[06:31] along the way. Whatever method you choose, I think the goal at the end of the day is how can you get better detections and better visibility into your systems um in a in a threat hunting process.
[06:44] So, we use the the peak model for those that are familiar with it. uh prepare, execute, act and knowledge. Um there's other frameworks out there like the Tahiti framework uh or various other
[06:57] frameworks. We like the peak model and it's really central to our entire threat hunting philosophy on data bricks today. So more so than just being a framework and an acronym, we actually built agents
[07:10] directly for each of these functions instead of trying to have uh either a multiple agent handling multiple things or a variety of these dozens of agents which I'll talk about on the next slide.
[07:22] But this is how we have built our our threat hunting capability is around each of these. So there's only four agents, technically five if you consider agent bricks. And I'll get to that in uh towards the end of the presentation, but agent bricks runs the schedule and
[07:35] coordination layer for an orchestration layer for all of the agents. And then we have these four agents. We start with prepare. And for those threat hunters in the room or those that are familiar with hypothesis-based threat hunting, uh prepare is really where we start. We
[07:48] identify the threat hunt we want to do. We build the hypothesis, the necessary data sources that are associated with it, and we set expectations for what uh success looks like. From there, we go into execute where we run the hunt by
[08:01] analyzing data. We test hypotheses. We explore patterns uh and we document findings as evidence emerges. Uh and ACT, we turn all of this into things like detections or response actions or
[08:13] tuning or any number of outcomes. And then we have knowledge which is essentially a knowledge store or repository where all of this data goes from past hunts and we keep this as a thing to track uh for the purpose of the
[08:28] agents. It's like a rag layer that can pull previous events in. So it kind of keeps everything really structured. Um but in terms of agentic parlance there's really three big things we hear about a
[08:40] lot. We hear about agents, we hear about skills, and we hear about tools. I'm not going to go over tools today, but I will talk about agents and skills. So, these are our agents and how we define that is basically a semi-autonomous or
[08:52] autonomous loop of code that tries to achieve an objective. So, these are all various uh Python capabilities that are written and they all have a very specific focus and they all are governed
[09:04] very specifically which is how we apply a lot of the lessons learned that we have through a governance process. Um so these are the agent layers and as I mentioned each of them operate individually of each other but they also
[09:17] operate in succession. So an act agent cannot kick off unless it has previously pulled information from an execute uh agent as well. Um and so we use things
[09:29] like the lineage tracker and data bricks to make sure that there's always been some kind of past activity. But we also built them separately because they need to operate independently. And that was one of the first things that we found out when we started this process was
[09:42] that um as an example, the ACT agent does really well at creating sigma rules and detections. A lot of that stuff is open source. The model LLM models have been trained on it and they are really good at writing rules and detections.
[09:56] Where the challenge for us came in was honestly in the prepare phase like getting an agent to stay within the guard rails and the theory of threat hunting without hallucinating and
[10:08] developing a wide hypothesis like a traditional threat hunter would was a challenging task. So what we did is we can release all of these as like version one and act will stay as version one and prepare might be version 1.2.3.4
[10:22] for and grow over time as we refine this process. And so that's how all the agents come together. The next is skills. So this is how we look at our each skill. When we started this, as I
[10:34] mentioned, we had one agent that was kind of responsible for all of this and it did really well on our laptops. Um it challenged though when we got up to using some really big data sets specifically in con the context layer
[10:47] was the big challenge that we had. So, we decided to break this apart. And the first slide I showed showed all of the problems that we were having when we were thinking about threat hunting with agents. And the problems that we were
[10:59] having came because we had four different agents. And then we also tacked on each of these different uh threat hunt capabilities or or processes as another agent. So essentially we had 12 agents like a a prepare baseline, a
[11:13] prepare hypothesis, a prepare model assisted hunt. And that just became really challenging. So we stripped everything down to the four agent approach that we followed and now each of these are skills within the agent
[11:26] process. So the great thing that I love about skills is that they're markdown files that anybody can really update. Um so the more junior analysts that you have if they need to update uh their their markdown files or the skills they
[11:38] have that capability as well. But now the agents can pick the appropriate hunt that they want to follow. And so I'll walk briefly through all three of these before we jump off the slide. But uh
[11:50] when we execute the peak process uh the skills are clearly defined in the background and the agent pulls from these skills. So we'll start with a baseline hunt. And I mentioned everything that we do has to be based off what is normal or abnormal. And
[12:03] traditionally baseline hunting has been really difficult for us in traditional SIM platforms. They're slow. They take forever to return queries. you can't really write data science type capabilities directly in the sim. You
[12:16] can run a Jupyter notebook, but um there's just other challenges that come with with the slowness of the system. And so when we're looking at a baseline hunt, we don't want 30 days of DNS traffic or 30 days of authentication
[12:29] traffic. I want a real sample size of what baseline looks like. So I'm already thinking I want 6 months of DNS data and I want it broken out by who it's calling, who it's talking to, what services are coming from so that we
[12:42] understand overall what the network looks like. The next one is hypothesis based oh and sorry those stay uh baselined on an every 7day window. So agent bricks and lakeflow processes will
[12:55] update the baseline hunts about every seven days um making sure that the data is fresh. So I like to always start with baseline but sometimes we can't but uh then we go into the hypothesis based
[13:07] threat hunt and this is where a lot of people traditionally think of threat hunting is I have this gap in my detection. I'm going to go on thread intelligence uh websites. I'm going to look about how threat actors are exploiting this. I'm going to come up
[13:19] with a thought in my head and then I'm going to go through the process of trying to find uh conduct my hunt and try to find if an adversary has made it into the network. Uh we call this process h the process behind that is haystacking. So we'll go out and we'll
[13:32] grab six months worth of data. Uh we'll get down from millions of events to thousands of events to hundreds of events finally to 10 10 hopefully not tens of events but ultimately this filtration process through the loop is
[13:45] is what we follow in a hypothesis-based hunt. Along the way we'll refine the hypothesis with everything that we find as well. Um and that gets driven back into the overall hunt package. And then finally, the modelass assisted hunt. So,
[13:58] historically, we've used analytics, statistics, machine learning, um, and now AI to really drive this modelass assisted hunt. One of the things that I I like about this model that PE had put together is it aligns really well to our
[14:10] detection engineering philosophy. So, we start with very deterministic, which is atomic rules, sigma rules, things like that. We then move into probabilistic, which is our machine learning capabilities and our our notebooks that
[14:23] run on the network. And then we have non-deterministic with the large language models. And really those help drive the other two. So those are really good as I mentioned at building detections. Sometimes not as great at building open-ended hypotheses. Um but
[14:36] in a balanced approach we really use all three for detect for threat hunting purposes. Um but this is the overall framework that we we use starting with peak and then these three types of hunt. So we use it as an operating model uh
[14:49] not just uh giving us the ability to break this down easily. But what becomes really important when we talk about things like how this links in with agent bricks and how this links in with unity catalog is that these essentially become
[15:02] govern agents with very specific job tasks and grants on the network that they can operate in different levels. It also gives the platform a life cycle. So so many threat hunts have been started,
[15:14] hypothesized, carried out and then a really long word document has been written and sometimes people never revisit them. Um the great thing about using uh platforms uh like data bricks
[15:26] is that we can express these as notebooks. We can have agents edit the notebooks. We can have agents edit workflows uh etc. Um and before I go on from this uh so all of our agents are
[15:38] primarily customuilt. if you're looking to adopt and build your own agents. Um I will say that the recently released uh data bricks AI devkit um has been phenomenal for helping us accelerate a
[15:51] lot of this uh we're primarily a cloud code shop. So just a forward slash I think it's datab bricks aagent and you can basically just ask the skill directly of the data bicks AI kit to really help you build and tune your
[16:03] agents and make sure that they're customized and that's accelerated uh the agent development and tuning process significantly. So if you're interested in building your own on a something like a a lang framework or agent bricks or
[16:16] you know anything else definitely download that and and use it. It helps with um more than just the building of the agents but also the authentication the lakeflow processes etc. So uh making log data useful for AI um
[16:33] this has been a big challenge that we have run into um in handling various types of data and what's interesting is um I probably read more research than I should but there's been some really
[16:45] interesting conversations or research done recently on how we're expanding context windows for agents but they're not necessarily getting better and one of the reasons is that there's so much bad data in threat hunting that we found
[16:58] was that there's so much bad data up front that causes what they're referring to as noise within these context windows. So, um, when we first started this, I would have told you just let the agent worry about ECS versus OCSF or
[17:12] anything else. The agent can do it really well. And I proved myself wrong with that. And the team was right that we did need to normalize on a common schema. We needed to unnst JSON files. we need to just deal with really good
[17:24] data sanitation because not only does it create noise within your context window of when you're hunting. Um, it also is really expensive as well to have the agent do everything for you at once. So,
[17:38] um, we'll go into a little bit about how we prep the data and how we look at it. So, there's more steps than this, but this is really at a high level. So, we use the lakehouse to do the heavy lifting for everything that we do. uh data flows in via zero bus connector and
[17:52] it lands in unity catalog and S3 bronze. We do use the medallion architecture for our threat hunting platform. We use lake flow, spark declarative pipelines and or data live tables. I can never remember what the new term is. Um to handle our
[18:05] medallion promotion process from bronze, silver, gold um and so all of that is built into lakeflow and and traditionally handled directly through that. The next thing we do is an OCSF
[18:17] activity stream. So I'll get to this. I think it's on the next slide, but um events on a network are really hard to just generally work with. And I kind of equating it to if you were to read a
[18:29] book that had just a bunch of random sentences in it. That's kind of what dealing with logs are like. You need to turn it into something digestible like entities which make paragraphs. And I'll talk about that on the next part. But to
[18:42] start there, we use this concept of what's called an activity stream. Activity streams are essentially taking all of these different log types. Um, moving them into entity relations and then putting that all in one table. So
[18:55] instead of having 18 different tables about hosts and authentication and IP addresses and source and destination and all of that, we actually start stitching together back the logs so that they can give us a good view of how things look
[19:08] like from an entire life cycle. Um, funny story. When I first started this presentation, I was going to go really in depth on like ontology and how this is all important. Um, if you're familiar
[19:21] with the meme of like I've been working on this and he just tweeted it out. Uh, data bricks actually released a thing called onto bricks. Um, it's on their GitHub page. Uh, it's not built for cyber security but using something like a defend framework or owl models.
[19:34] There's actually an ontology that you can use and put directly into uh, data bricks. now onto Bricks on the GitHub page. Um, so I had to go back and redo the first part of this because it it exists now in the the open source space,
[19:47] but ontology is really big for us as well and that helps drive this activity stream, but basically uh it helps ground our agents in a better way. And that's certainly a different conversation that we can have. Once we build the activity
[19:59] stream, everything is governed by the Unity catalog in the activity stream. So I'll talk about this in a couple slides, but basically this is where we allow permissions and authentication across the data. We never let agents have
[20:11] access to raw data. One of the biggest conversations we've had with sysos has been I do not want my raw security data going into some company's LLM and I can respect that. So what we have done is
[20:23] instead say we don't really need raw logs for threat hunting. We need aggregates, we need statistics, we need trends and change over time and none of that really requires the raw data. So that is all heavily governed through the unity catalog. And then one of the great
[20:37] things about this is that uh we have gold entity views. So and again this will be on the next slide but what this means is that if you look at the medallion architecture the traditional sim will bring you maybe bronze and then
[20:51] to silver and it kind of drops you off in this data lake data swamp of all of your data organized by maybe its source into a table or something very similar. Um, we instead like to take that a step
[21:03] further and in gold we build what are called entity views and these are basically essent essentially materialized views if you're familiar or precomputed views based on entities within the network. So that is something
[21:16] that we cannot do in traditional platforms. Historically we've had to export all of this data into another database and do it. But now we've built it directly in so that all data is set up uh to come in. It goes through this
[21:30] entire process. We have a YAML metadata driven for kind of what gets built in these tables. But these entities, if you're familiar with machine learning or you're a machine learning engineer, this is heavily borrowed from things like feature engineering. So the the gold
[21:44] entities are broken down into um the number of failed loginins over the past hour or the number of failed loginins over the past 24hour or authentication burst around two failed loginins with a successful login and a unique IP. So all
[21:59] of this happens automatically and our gold layer for precomputing these views. We don't have to build necessarily the detections over them. Uh they just happen as the tables build. Agent bricks is our supervisor and our orchestrator
[22:11] as I mentioned. Uh that is a key piece of keeping the agents governed in how we look at applying permissions into the agents. And then finally findings and detections which is basically um uh
[22:24] sigma rules or any number of things. Ultimately, all of this gets stored in lakebase though, so that it's always accessible by other agents as well. So, I talked a little bit earlier about entity or events verse entity. And the
[22:38] way that I traditionally view this is trying to read logs in a sim is a really difficult process because you're just shoving a bunch of data in and it's getting indexed and now we have to go
[22:51] through and figure all of these things out. And I I used it before, but the analogy that I use is like basically somebody wrote a book and it's just a mixture of sentences and you need to figure out how to build the paragraphs for that. So entities are our approach
[23:04] to doing that. We use, if you want to look up like entity resolution, that's what's done here. But basically, we break these into knowledgeable things that we want to know and that agents would want to know for threat hunting. And then we just allow them to query and
[23:17] understand the entity layer uh within within data bricks. So events are going to tell you something like what is on your left hand side. Uh Bob had a failed login, had another failed login,
[23:29] successfully logged in, command.exe was started, who am I was run, DNS query, VPN uh log failure for authentication happened. In a traditional event-based model, this is all stuff that we would have to track through, write queries is
[23:42] that haststack process I was talking about where we'd probably need to pull in a whole bunch of authentication logs, filter for failed loginins, write for failed loginins at two and then a successful one, rewrite the query for
[23:55] okay, now that we have this filtered out, uh what are we looking for for command.exe? Instead, we break this into the entity model where we start looking at what is our first scene IP address, where are uh
[24:07] rare tenant sources and what are some unusual processes that are associated with authentication b uh burst. And so when I started at the very beginning, I talked about the baseline. This is what helps us build all of that out. So we
[24:19] build all of the things that we care about in the gold layer for querying and then the the tables get populated over time and ultimately this story on the other side of the slide turns into a paragraph that helps you understand that
[24:32] not only did Bob have two failed loginins and then a successful login um but also who am I was run on the network if that's a table that you want to mention or any of the other um service to activity processes that can be built
[24:45] as well. From there uh we look at other things within entities that change over time. So I mentioned the first scene IP. Then we go into unusual laptop or host calling out uh no or no successful login
[24:57] from a a rare IP etc. Ultimately what we end up with is when the agents go into hunt we end up with five leaked linked events three sources and then one investigation package. So this will all
[25:10] get put together the go over to the incident response team or open up a ticket in Jira or some sort of uh other process. But this moves us from trying to understand millions and millions of
[25:22] events that happened within the network and to actually contextualize things that are going on within the network and that is the area that our agents live is in this entity layer. And so I mentioned that a agents never
[25:34] see raw telemetry and there's there's really two reasons for this. The first one is that we found out that most people are uh or a lot of sisos I should say are still uncomfortable with sending
[25:46] raw telemetry to somebody that isn't like a managed security service provider or just kind of a SAS provider. I'll just put it that way. Um so that was number one. Number two is that when we first started this process, we found
[25:59] that running threat hunts in an environment that is ingesting about 20 terabytes of data a day filled up context really, really, really fast. And so we were consistently compacting where
[26:11] we were consistently drifting off of things. And so we really wanted to find a better way to do this. And we started looking at threat hunting. And rarely in threat hunting do we actually need or look at the raw logs. Instead, we're writing data frames that are pulling
[26:24] information together. We're ranking them by, you know, if you're looking at DNS logs, whether it's top talkers or bottom talkers on the network, you're somehow filtering and sorting sorting them. But
[26:36] rarely did we ever really need to look at like the individual host IP address that was doing that until much further into the investigation. And sometimes that was actually the IR team's process to handle that. So we instead said let's
[26:50] take this materialized view in the gold layer or this entity view and instead make that the governance layer for what our agents are querying. And so what that means is basically within Unity catalog our agents have an identity that
[27:04] they are able to access and a service principle but they are never given grant access to any other table than gold. So as you can see on the screen in this query if it tries to run select from
[27:16] silver O raw or even just silver O it will be denied because it is a boundary violation that it is now trying to access another row of data that it cannot access to. So these are governed non-human identities that we are
[27:28] applying directly in the datab bricks platform through agent bricks and unity catalog. uh because no matter how many times we try, we still see agents sort of jumping the gun on what they can
[27:40] access and kind of thinking for their own. Um and so when we start to get into more of the semi-autonomous nature of this, that has been the challenge. But basically the platform is the guard rails for um what can happen. The other
[27:53] thing is that when we we run the uh we run it this way, the context window stays small. As I mentioned um billion row data never enters context instead it's usually something wrapped up or aggregated and then we can break it
[28:06] apart from there. So four agents, one supervisor as I mentioned, agent bricks is up top is our supervisor, our orchestrator, the main sort of plat the main sort of thing driving our agents that exist for what
[28:20] they can access, how they access it, the governance associated with using um uh AI models so that we can track callouts to LLMs and then ultimately we manage performance flow within like ML flow to
[28:34] understand how all of these agents are working. Um, and that really becomes sort of our metric is we have these benchmarks built in. But MLflow is where we really start to track things. But if you layer it over in the the four layers
[28:46] that we started with, we end up with these again these prepare, execute, and act agents. And so we start with ingesting the objectives and selecting the hunt type over in the prepare phase.
[28:58] And we go through the execute, act, and knowledge phase. Each of these operate independently. Each of these are governed independently. And each of them are released independently. So as I mentioned early on, LLMs are really good
[29:10] at writing sigma rules or spark queries or spark jobs or um data bricks notebooks. Um they're pretty good at that. But getting them to stay settled with building a hypothesis can be a
[29:23] unique challenge. So this gives us two things. It gives us the ability to ship agents individually. So as I mentioned at can stay on like version one. Meanwhile, we may have a whole bunch of releases of the prepare agent because we just need to consistently uh work with
[29:37] tuning it and getting it to operate in the way that we need. Uh and then the other thing is that each of these can hold their own context window as well. So we're not trying to hold context window for all four agents across the entire hunt. We're actually giving
[29:50] prepare its own context window and when it's done, it writes to lakebase. Execute will then know based on the UU ID from the hunt that was assigned that it needs to go pull this information. it pulls that context down. So instead of
[30:02] managing like markdown files, we're actually just writing directly to lakebase. And then they all know which tables they should be operating and accessing and and pulling from. Um each agent has its own job as I mentioned. And then finally, these isolated context
[30:15] windows keep everything small. So if you're interested in moving into this way and building something, I mentioned the data bricks AI dev toolkit. It's phenomenal for helping accelerate a lot of this development.
[30:27] Um, but if you want to get into like agentic threat hunting and building your own capability, learn from us. Don't start with a gazillion agents to to start with. You'll just get frustrated. Um, don't start with every log source.
[30:39] Pick a specific threat hunt that you're interested in. Um, build in human in the loop processes. Uh, and then finally, don't do any really complex hunts. Maybe just focus on one hunt. If you have hunt playbooks already, pick one from that
[30:52] and and put it into action. Um if there's not, you can you can Google, you know, multiple different uh playbooks that exist as a starting place. Um but do start with one highvalue hunt. So maybe you have something that you just
[31:04] haven't been able to really figure out or write a detection on or any number of things. Um start there. Don't make it too complex, but make it high value. Um the other piece is start with probably two entity types. That's where you're
[31:16] going to get the most the most bang for your buck in implementing these. Um focus on clear evidence requirements and human in the loop. So don't focus on running these semi-autonomously quite yet. Just focus really on leveraging um
[31:29] the platform to to guide the way. I think if you were to use this and leverage things like building uh lakeflow pipelines or writing this new notebook to get through our data, that's
[31:41] a really good starting place to to start with. And then also make sure that you're focusing on building the governance layer within Unity Catalog so that your agents aren't going out and doing a bunch of random things. And finally, and I'll open for questions if
[31:55] there are any, uh, three key takeaways. Really focus on tuning the system before you scale the agents. Um, threat hunting needs entity context, not just event search. So, we have found, as I mentioned, just trying to get it to
[32:08] crank through log data over and over again. It's just not a really successful process, but leverage the power of data bricks into this gold layer that that's really where you're going to get your biggest bang uh, for what you're trying
[32:20] to do. Um, agent should investigate govern views. Again, as I mentioned, hold uh your authentication and your governance within Unity catalog. Uh, and don't let it have access to the raw telemetry. If you want to, that's cool,
[32:32] but don't start directly with having it query like a silver layer as an example. And then finally, really focus on leveraging agents for the purpose of orchestration, enrichment, evidence, and guard rails. Um, you don't need to have
[32:45] the agents uh handle workflow, processes, and everything else. just have it use LakeFlow to build the workflow process. Or if you have a really bad data set that uh you know the
[32:57] application team has been hanging on to this application for 20 years and they won't get rid of it and you have to write a bunch of manipulation to query that. Um have the agent do that have it build up and collapse the lakeflow
[33:10] pipelines to handle the ETL of all of your data. Um but ultimately let the agent be the orchestrator and not the actual builder builder of a lot of these different capabilities.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.