From Petabytes to Proactive Defense: Powering AI Swarms With Databricks

Summary

Arctic Wolf rebuilt its security data platform on the Databricks Data and AI platform to process petabytes of telemetry across 10,000 organizations, replacing traditional partitioning with liquid clustering and predictive optimization to eliminate data skew, reduce file counts by 50%, and accelerate 90-day queries from 51 seconds to 6.6 seconds.
Continuous pipelines replaced hourly batch jobs to achieve minute-level data freshness, while gold tables trimmed AI agent context windows from 2,000 to 300 columns and metric views provide consistent semantic understanding that allows multiple agents to reason over the same data without ambiguity.
Multi-agent orchestration based on the OODA loop—Observe, Orient, Decide, Act—coordinates threat hunting across email, identity, cloud, and endpoint systems, enabling proactive security defense at petabyte scale across Arctic Wolf's monitored customer base.

From Petabytes to Proactive Defense: Powering AI Swarms With Databricks

When security operations process trillions of events daily, the bottleneck is not data, it's performance. Arctic Wolf analyzed petabytes of telemetry across 10,000 organizations and discovered that partitioning alone created small files, data skew, and costly acceleration layers that pushed query latency to unacceptable levels. This talk reveals how intelligent data organization unlocks real-time threat detection at massive scale.
You'll learn how Arctic Wolf architected a Databricks platform using liquid clustering and predictive optimization to eliminate data skew, reduce file counts by 50%, and accelerate 90-day queries from 51 seconds to 6.6 seconds. See how continuous pipelines replaced hourly batches to achieve minute-level freshness, how gold tables reduce AI agent context windows from 2000 to 300 columns, and how metric views enable consistent semantic understanding across agents. Discover how multi-agent orchestration using the OODA loop coordinates threat hunting across email, identity, cloud, and endpoint systems, enabling proactive defense at petabyte scale.
🤝

Chapters

FAQs

What is liquid clustering and why did Arctic Wolf adopt it?

Liquid clustering is a Databricks feature that organizes data files without fixed partition boundaries, automatically balancing file sizes and co-locating related data. Arctic Wolf adopted it because traditional partitioning created small files and data skew that pushed query latency to unacceptable levels when processing 90 days of security telemetry.

What performance improvements did Arctic Wolf achieve with liquid clustering?

Liquid clustering reduced file counts by 50% and accelerated 90-day security queries from 51 seconds to 6.6 seconds—approximately an 8x speedup. Combined with predictive optimization and continuous pipelines, the platform now delivers minute-level data freshness compared to the previous hourly batch schedule.

How does Arctic Wolf use gold tables to improve AI agent performance?

Gold tables are pre-aggregated, denormalized datasets that distill raw telemetry into analyst-relevant columns. By reducing the column count from 2,000 to 300, gold tables significantly shrink the context window that AI agents must process, improving both speed and accuracy of agent-driven threat detection.

What is the OODA loop and how does Arctic Wolf apply it to security AI?

The OODA loop—Observe, Orient, Decide, Act—is a decision-making framework Arctic Wolf uses to structure multi-agent threat orchestration. Each stage maps to a workflow component: ingestion pipelines observe the environment, enrichment and metric views orient the data for analysis, severity scoring decides the response priority, and automated workflows act by routing threats to the appropriate team.

Full transcript

[00:08] Hi everyone. I hope everyone is excited after the keynotes. Um my name is Justin. I'm a distinguished data engineer at Arctic Wolf. Today my co-presenter and Raj Nish and I are going to talk walk you through how we rebuilt our data platform from the ground up to power AIdriven security
[00:24] operation. We're talking pabyte of telemetry, liquid clustering, continuous pipeline, and eventually agentic swarm doing proactive threat detection. Let's get to it. Go.
[00:39] Quick intro. I spent the past 10 year in the cyber security space using data bricks to collect and process telemetry at scale. I'll hand it to Rashnish to introduce himself. Thanks, Justin. Hey, I'm Rashnish. I'm a delivery solutions architect at data bricks. been with data bricks for more
[00:55] than two years and in this industry for more than two decades. Uh I have been helping hundreds of customers land and optimize their data estate. Before we get started in the technical detail, I want to set the stage with a problem statement. I used this in my
[01:12] previous pre presentation last year and I feel it's just as relevant today. The vast amount of log data engineer generated makes it challenging to identify potential security threats and incident in a timely manner. This is the core tension that we live in that we
[01:29] live with every day where we are building when we're building our data team at Arctic Wolf. We're constantly dealing with this problem space. Everybody wants everything ingest all the data in real time with infinite data retention pre-enriched and ready for
[01:44] analysis. And most of all they want a single table for and every field searchable in millisecond. And ideally they want all the data in a single UI from all the different geographical region where we collect data from. And
[02:00] oh by the way they want us to stay compliant with all the regional data regulation as well on top of that. And most importantly they want to do it for free but or cheap as cheap as possible. That's the world we operate in. Today I'm going to show you how we tackle it.
[02:20] So what did we actually settle to build? The goal was simple to say but hard to execute. Our end goal was to build a data platform with AI for AI and ML security operation. To get there, we need to remove two major blocker. First, we need to move the data accelerator.
[02:37] Second, we need to redesign a pipeline to support the ton of data freshness and scale that AI workload demands. Let me explain why. Here you go. When we first built our table, we used
[02:53] partitioning. It worked, but we couldn't quite hit our SLA with just partitioning. So, these are a number that our product manager wanted. Sorry. These are the number that Oh,
[03:09] technical difficulty. These are the numbers that our product manager wanted. Two second for small customer for three days of for three days of data. 3 second for medium, five for large. What we actually got for
[03:26] using our partition table was 7-second, 13 and 12. So we missed it by quite a bit of margin. So what did we do? We bolted on a data accelerator to bridge that data freshness gap and query performance gap.
[03:43] So you might think why are we removing a data accelerator now since we just we we just bolted it on right. The problem was we had two set of infrastructure to maintain essential and they essentially working on the same data set and that created four compounding issues.
[04:01] One, infrastructure complexity. We have two team managing the infrastructure, two system to process the data and it's the same data. Second, because of a data accelerator, we need to query blend. Every user query had to be split up and
[04:17] sent to two back end and the result is then stitched back together. That's fragile and it's slow. And lastly, and then cost. We have over a pab pabyte of data with the accelerator. We we are
[04:34] essentially storing the data twice and we're proc we're paying for the infrastructure process to query as well. And lastly, SQL interoperability, our data accelerator is not NCSQL
[04:51] compliant. We ended up having to rewrite a lot of the SQL statement. one for the back end and one for the one for the accelerator and one for data bricks. That's just gets really tedious really fast.
[05:08] For phase one, we work with data bricks to leverage liquid clustering to reduce our data accelerator usage from days to hours. We ran multiple experiments to create a framework to understand our data in multiple dimension. First volume by customer
[05:24] volume by telemetry type and then query patterns for internal external use cases. And along the way we we actually did an additional experiment table composition by telemetry type and what that means is based on certain telemetry type what are the which fields are null
[05:42] which are not null and what are the possible value of it based on that telemetry type. Let me walk you through what we were dealing with before. The three core problem with our partition table small file which is a
[05:58] direct consequences of our partition column choice. We partition by date hours and rely heavily on Zordering against customer ID at telemetry type to keep the data grouped together which means during low volume hours we
[06:15] generate really tiny files. Data skewess security telemetry is inherently uneven. A firewall generate very different volume than endpoint agent and different customer has different sensor profile. So they're are
[06:30] very uneven. And lastly arriving data just as we are tackling this problem later arriving we're we have a real life system data comes in late endpoints customer don't send the data all at the same time. So these are all real world ingestion problems.
[06:50] Look at the small file problem. Here's what it looks like in practice. You can see in this visualization, none of our parquet file that makes up our delta table ever rarely exceeded a thousand meg in size. And you can see this probably is worse on the recent
[07:06] data where most of them actually are below 200 megs in interest. Ideally, we want to target larger file size because that has because the larger the file size, the better when you run query, the less file IO it needs to do,
[07:21] which leads to better query performance. And here's the skew endpoint agent dominate our telemetry data followed by DNS and a firewall firewall and IDS alert which are more refined alert. when I'm searching for IDS alert
[07:38] and I have to go through all these endpoint data it makes the query slow and then on top of that every customer has a unique data profile some might just have an endpoint and no firewall some may have DNS but no endpoint they all configure very differently which
[07:53] makes trying to tailor it for a particular try to tailor it very difficult and finally the late arriving data It comes from three places. Unreliable network going offline and less machine
[08:11] going offline and let's be honest gremlin and goblins in our own infrastructure AWS outage data brick outage bugs. It happens and when it does show up it lands outside of the data we already optimize and degrades the query performance over time.
[08:34] Now here's the thing. When we actually looked at the data, late arriving data only makes up around 0.5% of our total volume. The vast majority of our data actually lands in less than five minute. So it's not a volume problem. But that 0.5% arrogated
[08:50] day over day was quietly degrading our query performance by 30 or 40%. Because every time a query runs, it has to scan those late arriving data in those partition to ensure completeness. Small problem, big impact over time. I'll now
[09:07] hand it over to Raj to talk about how liquid clustering helps us tackle this problem and enable us to use to leverage AI against our data set. Thanks Justin. All right. So Justin spoke about three problems
[09:27] small files data skewess and late arriving data. I have worked with hundreds of customers and I've hardly seen any customer who does not have any of these problems. So it's not specific to Arctic Wolf but a widespread problem being t being used by different customers. Now let me give you an
[09:42] illustration by using customer and date as an example uh to show you. Now your scenario could be different but this is just an illustration to say how with partitioning you can see these small files and data skewess problems. As you
[09:57] can see these small square boxes represent the size of the file and as you can see for customer C there are lot of these small files and we can also see there's lot of data skewess for customer C there are a lot of files but customer D and E hardly have any files right uh
[10:15] uh each day now with these small files there is a lot of metadata operation involved and hence the reads are slower now one thing that might come up in your mind is okay there are small files. What if I compact these files and make them bigger? All right, let's do it. All
[10:32] right, when we compact these files, here is what happens. As you can see, for customer A, you compacted three files. For customer C, you compacted so many files. But for customer D and E, there was hardly any compaction because
[10:47] there was nothing to compact. Right? So you run optimize which is how you compact files but that does not go beyond partition boundaries and this is how after compaction data looks like. Now this is my target file size which is
[11:03] what I aspire to have for every customer. I was able to achieve that for customer C well but nothing for customer D and E. their file size continued to remain the same which means compaction did not solve my small files problem. It
[11:20] also did not solve data skewess right as you can see lot of data for customer C but D and D there is hardly any uh hardly any data right uh so how do I so in a sense if you have
[11:37] partitioning you will you will continue to have the small file problem as well as data skewess problem and you might think okay I have partitioned it by day what if I partitioned by week I can also aggregate these small files again into bigger chunk but then beware
[11:53] partitioning is static which means if you want to now partition by by week you have to rewrite the whole data what if in future you think as my data patterns evolve I might want to partition by month this can keep evolving and you will have to rewrite every time all your data so it's not dynamic now what if
[12:13] these partition boundaries are blurred or what if they don't even exist and a system can intelligently identify the set of customers and dates to combine together to compact files in an intelligent way and that's liquid clustering for you. Let me demonstrate.
[12:30] Go back to the same example. All right, let me go back to the same example and this time I will do cluster by date and customer ID. Now, same set of data, customer C has lot of data. D and E hardly has any data. But now I
[12:46] will run optimize which will compact data intelligently in a way that it combines let's say customer A and B and few dates together for smaller customers it might combine three customers or or many customers. This is just an illustration but to let you know that
[13:03] with liquid clustering there are no boundaries as was there with partitioning right and this is what it looks like after compaction an intelligently balanced well balanced size file sizes across customers so my small file
[13:18] problems gone how about data skewess do you see any data skewess I can see very balanced kind of data all throughout different customers so I don't see any data skewess no I is just very not symmetric. I don't like it. But wait a second. Did you also not talk
[13:35] about late arriving data? I did. All right. Let's see. How do you handle late arriving data? Now, let's say a new injection comes in on a day. What happens with liquid clustering? It will now only look at the newly ingested data. It will not touch the already well
[13:52] balanced cluster. So, for example, 15th and 14th, they're already wellclustered. it will not touch any of that data and it'll only look at that incremental clustering. It's like full refresh versus incremental refresh. So liquid clustering when clusters new data it
[14:08] uses incremental injection or in incremental clustering. That way there is very less right amplification. Right? And this is how this is how it looks like when it compacts the data. What if the data
[14:24] instead of coming on 16th would have come on 14th and 15th. Well, same result. You see, whenever data comes in, whether it is late arriving or current continuous data, it intelligently clusters data in an incremental way without any right amplification. So, you
[14:41] get seamlessly balanced data files across customers with no data skewess. All right, how do I turn it on? Well, pretty simple. All you have to do is just use either cluster by auto keyword
[14:56] or cluster by specific column list. If let's say you do not have very fixed query patterns with specific columns being used in the in the filters, joins, aggregations etc. Use cluster by auto which means system will identify the set of clust columns to cluster by and these
[15:13] can keep evolving or changing as the system identifies different query patterns over a period of time. So you do not need to worry about it. So just use cluster by auto in case let's say your 90 or 95% of the queries have fixed query patterns you can use those fields in cluster by those specific columns.
[15:31] Now um there is a limitation of up to four columns uh what you can put in the column list but with arctic wool for example we had around three or four pabytes of of a single table and we were able to cluster it efficiently with just two columns. So don't think four column
[15:48] is a limitation. The more the number of columns, the more dilution of the cluster happens. So keep it simple, concise. My recommendation would be to use start with cluster by auto. That should give you a really good performance off the mark. All right. Now when you when you think about liquid
[16:06] clustering, you know the benefits of it. Now how do you enable it and what all aspects do you need to think about? There are four pillars when you think about liquid clustering. liquid clustering of course predictive optimization that is an engine that runs
[16:22] compaction and clustering which is what optimize does. It collects statistics which is what analyze does and it also removes the old files or stale files which are no longer needed which means my read operations have to now scan lesser number of files that that's what
[16:38] vacuum runs. Eagle clustering which is clustering on right which means as soon as I I ingest my data at the same time I cluster my data so I don't have the small file problem to start with itself right I
[16:55] don't even get that creep up right so that's eager clustering for you you write you cluster the data as soon as you ingest it and lastly query optimization please don't ignore query optimization when you enable liquid clustering so these are the four pillars that you should think about when you
[17:11] talk when we think about liquid clustering. All right. Now I'll talk about best practices against each. For liquid clustering I have just couple of best practices. Used manage tables. Managed tables are a doubledigit percentage
[17:26] significantly improved than external tables. The performance is much better with managed tables compared to external tables and it unlocks lot of other features like predictive optimization which would otherwise not be possible. So use manage tables
[17:42] and pick the right clustering keys. Again if you have uh if you have chosen cluster by auto, you do not need to worry about it. But in case you have chosen cluster by specific columns, pick the right clustering clustering keys which happen to be in your filter clause in your joins etc. With predictive
[18:00] optimization, the only feedback that I have or recommendation that I have for you is please enable it. That's all what you need to do. enable PO and it will run optimize, analyze and vacuum for you behind the scenes. You do not need to worry and it'll only run those
[18:16] operations when it thinks they are it'll give you benefits by running those operations based on the query patterns. So there's no fixed pattern to when it'll run depending on the query patterns it'll identify when to run these operations. Clustering on right like I said as soon
[18:32] as you're clustering you're ingesting the data you want to cluster it right then and there. Now only couple of recommendations here. Process things, process your injection streams in batches. Uh now you could do eager clustering with both streaming as well
[18:48] as batch pipelines. Yes, you heard it right. It works with streaming as well. In fact, we made it work with Arctic Wolf on streaming. So with streaming workloads or with batch try to process your data in batches so that you have sufficient chunk of data to cluster and think of eager clustering as your local
[19:04] cluster while your optimize is your global cluster with and the last thing is query optimization. Again, couple of things here. F filter on clustering keys. Again, if
[19:22] you're using cluster by auto, you do not need to worry about it. But again, if you're not using cluster by auto, try as much as possible to keep those keys in in your queries in the filter clause or joins or aggregates. And lastly, leverage data skipping
[19:39] stats. If you have used predictive optimization, again, you do not need to worry about it. it'll be taken care by itself. In case you are running manual optimizes by yourself, then utilize this key feature of data skipping stats column that will collect statistics
[19:56] beyond 32. So in general for let's say external tables and regular tables first 32 column stats is collected for data skipping. So all of those column stats of min and max is kept kept in the files. If you want more than 32 column stats to be collected because they might
[20:12] be appearing in your filter or join clauses, then use data skipping stats column and we used it efficiently here with actic world to make things process better. All right, over to you. Thank you. Well, Raj kind of let the cat
[20:27] out of the bag early, but we did settle on organization ID and date as our clustering key. After running some analysis based on the internal query and external query um we noticed hey most 98% of our queries had those few in it that's why we decided to do that so with
[20:45] the organization and ID and date set we apply eager and we start implementing and after much tuning and custom DBR image we are able to get eager clustering using for each batch and we are able to then set a target file size
[21:01] around five to 600 GB and here's the result and here is the result. Oh no, this clicker is tricky. Um the result very clear the majority of our parquet file are now over 500 meg. You can see even
[21:17] the most recent data at the far right hand corner we're getting close to majority is over a,000 meg 2,000 meg. We even got some around 4,000 meg. And that means fewer file read operation which improved query performance.
[21:39] Here's a sideby-side comparison. For a 3day small customer query, we went from 6.5 to 3.2 seconds. And in the net 90th percentile, we went down from 7.8 to four. And of course, like all features, there were a handful of cases where clustering resulted in slightly slower
[21:54] query, but as a whole, it was a clear improvement across the board. There we go. So, this is a summary of what we have done. Now I'm going to talk
[22:10] about the next step which is to continuously ingest data in so that we get that low latency low data freshness data.
[22:27] So phase one got us to an hour. Phase two will now we're now talking about minutes. So little history, our pipeline was running on an hourly batch schedule, which meant our data accelerator was still in play. We needed to fill that
[22:42] gap between the hourly runs to fully eliminate the data accelerator. We needed to move it to continuous pipeline. So working with our PM, our PM wanted data freshness from minute to seconds. Well, little challenging to get
[22:58] low second on a pedibyte scale. So we got them down to low minutes, five minutes ideally. Um they wanted query time to be better, but with data freshness, we're processing data faster. There's challenges with that. We we feel that's a separate problem to tackle at a
[23:14] later point. And then finally, with all these in place, we could finally kick a data accelerator, remove it completely. But as we made the pipeline continuous, old challenges came back. The small file problem came back because we're
[23:30] processing smaller batches of data which because we're processing less data and processing smaller files, eager clustering effectiveness became was impacted because we're processing less data. So we're clustering on less data which then
[23:47] ultimately led to more optimize because we're writing these small smaller files and optimize needs to work extra hard to recompact them into larger file size that we saw earlier. These it wasn't going to work.
[24:03] It was very expensive. So as we work through this process we actually our team actually identified that eager clustering became our bottleneck. We're continuously stuck between seven to 12 minute per batch even as we scale our compute and our
[24:20] cost actually increased it 54% compared to our hourly batch which wasn't what we expected. Typically it should be roughly about the same or slightly cheaper. Um so we escalated to data brick engineer and they helped us with two things. a specialized spark
[24:37] configuration tuning and a custom data data data bricks release image which I feel was very important um is based on an ingestion profile and it adjusts how aggressively eager cluster runs the idea is trade a little clustering quality at
[24:54] right time write faster cluster less aggressively and let optimize and po handle the rest asynchronously So here are the settings that work for us. I want to be clear, these are tuned
[25:10] for art table specific workload. Don't copy and paste them and expect the same result. These are more meant as a conversation starter with your account team and solution architect. Um in the interest of time I'm going to skip go into detail but basically
[25:28] the result we achieved 6 to9 minute at peak 5 minute off peak and overall our cost decreased by 7.8%. Why is that important? This allows us to give us h room to scale up additional compute resource to
[25:44] improve the performance in the further. So we ended up with this. This is the same uh delta table breakdown on the parquet file. You could see there's less large parquet file on the recently
[26:02] ingested data but overall the time to create these were still relatively short compared to partition. So we ended up achieving our goal.
[26:19] So what did we learn? The most important aspect is we learned that low minute freshness most likely is achievable with our current design. But to get it down to seconds, we feel redesign is needed. And we are looking to explore real-time mode on a
[26:35] smaller table with just two or three hour retention or real-time mode using our analytic that we collected on columns that are important for internal and external customer. And most importantly, by switching to continuous, our data is fresh enough now for us to
[26:51] run ML use cases directly on data bricks built on top of our data. We truly got an AI and data platform that we could work with. Okay, refine silver to go. I'm not going to go over this. We both agree everyone
[27:06] knows about gold table. Been talked about enough already. Um the highlight reel is we our silver table is around over a thousand column wide closer to 2,000 and we're able to using analytics drop that down based on telemetry type
[27:23] to roughly two to 300 column on average for per gold table. And why is that important? Because LM agent when working with the silver table has to figure have to go through all these column to figure out which one are
[27:39] relevant to a security question and that takes up a lot of context window. With the go table we're able to shrink that down to around two to 300 columns and provide better description and along with a table description so that they could just hone in on a specific goal
[27:55] table they're interested in without going through the entire table. And you can see in the results AI agent spiked up. Um our internal user loved it because a wide column you see so many nulls and our user hated it. It's like I have to scroll through like pages of
[28:11] nulls before I see the value I want. I have to type I have to disconnect from my workflow and search against the field I'm actually interested in. So no one really liked it. So to the this is the fun part. uh with a solid platform underneath our AI team
[28:27] started building agentic workflow on top of it. So what is an agentic swarm? It is a collection of autonomous AI agent and I use the term autonomous loosely. They're more like teenagers. They need some handholding. They control their own
[28:44] flow. They collectively converge on an output. Well, sometimes they do, but in theory they should. And when you run them on the same data, you should get consistent repeatable result. But they do need a supervisor, either a human in the loop or another AI agent to kind of
[28:59] supervise them and correct them from time to time. Before I go further, I want to share a quote from my former colleague. In a brave new world where vulnerability are increasing 40% year-over-year and
[29:15] the cost to identify and exploit vulnerabilities are falling dramatically. Proactive defense is imperative. And this align a lot with what was discussed by Alli during the keynotes on auto detection with lakewatch.
[29:30] But what this quote means in practice is as large language models such as mythos or opus gets more powerful the time from CVE publication the active exploitation is shrinking. It used to be months but now we're seeing it goes from weeks days
[29:46] to hour and the rate of CVE discovery is also accelerating. We simply cannot respond to this at human speed anymore. The red team is using all these AI tool to constantly attack our infrastructure
[30:02] and as a group the blue team needs help and Arctic Wolf CTI team and our agentic human stock is here to help. Well, that that was my marketing speech.
[30:21] So again common theme core problem linear triage cannot scale our sock is one of the world's largest little bit of marketing and we hit the ceiling on what can be accomplished by adding more analysis at some point people just doesn't buy you more efficiency I like to do a comparison it's like you have a
[30:39] feature that takes 80 man hour to do in the past a product manager or a manager may say if I gave you 80 resources it would take you an hour, right? That's simply not true. So, the solution is a parallel multi-
[30:54] aent system grounded in up-to-date Fred intel and a corpus of telemetry, ticket history, and resolution data provided by our sock. Agents that automatically use that data to detect, triage threats from trillions of observation as they come
[31:10] in. Machine speed at scale. So with our agent we adopted the UDA loop and what is UDA? It is observe, orient, decide and act. It is a
[31:28] decision-making framework originally developed by military strategist John Boy to describe how a fighter pilot process and respond to rapid changing situation in a dog fight. The idea is whoever could cycle through this loop
[31:44] faster than the adversary wins. In in cyber security, we borrow this because it maps perfectly on how a sock should operate. Continuously gathering signal, make sense of them, decide on a response, act before the attacker does.
[32:01] And we apply this loop to our swarm architecture. Observe we we observe we collect a telemetry orient we collect a telemetry ask question against it how did this happen is this data relevant
[32:17] are there IOC in there that allows us to connect it to our XTI system once we get that result we make this decision based on the ILC that was detected which tier this what level of response should we suggest
[32:33] autoblock ticket view, monitor or suppress. Then we pass the data to our act agent who is trained on a collection of our customer engagement runbook on what to do when certain cyber security event occur and then provide action that
[32:51] is consistent. And of course, we need to ground all of this with the data and also put in the semantic information on when to use the runbook and and how to measure the severity of a cyber security incident in
[33:10] into the data and we want that to live close to the data. My co-presenter Raj will now talk about metric view and how that helps us. Thanks Justin. Now if you have a lot of uh downstream applications, it becomes imperative to use a semantic layer that
[33:28] works seamlessly across all of those downstream applications including AI agents, Genie, BI tools, etc. Now what that metric the semantic layer provides is is the is the context the business
[33:43] context that these downstream applications can understand, right? and metric view is the answer for semantic layer from data brick side. Now what does it provide? It gives you a single source of truth which means you define the metric ones and everybody is going to use it multiple times which means you
[33:59] also get higher LLM accuracy because the same the all the AI agents are going to use the same exact definition of your metric view which means they don't go discover different different columns to to answer your questions. They're built-in business semantics because your
[34:14] display name, synonyms, etc. are all part of that business semantics and yes it is governed by unity catalog. So you get all the permissions, lineage, auditability etc. right out of the box and you have performance built in because right like I said you create metric view once you create the measure
[34:31] once it is being read or used by multiple agents. So every agent is not creating their own calculations again and again. So it is performant and you also reduce the prompt engineering because now the business context is already embedded. You don't need to provide that business context and you get um a a seamless governed and metric
[34:50] view semantic layer which is consistent and secure. Thank you. This works. So what does it look like in Arctic Wolf in our platform? So this is a high level picture. Uh we have a hive mind architecture. A supervisor agent
[35:08] controls the overall flow and a memory agent enables learning from previous run so that the supervisor could learn from the past and learn from when to ignore false positive and get smarter over time. I'm I won't start talking about the flow
[35:25] because we kind of ran over it. Um but we do have a demo that will show how the agent goes through the flow and then I think it's more interesting to see what our metric view look like. This is a identity offmetric view. Um obviously
[35:40] it's much bigger than this but unfortunately I didn't want to record a video. We only included event count the stink host. These are all measures that our agents could then use in a consistent manner. And the nice thing is because this is not part of the prompt,
[35:55] we could have a separate infrastructure that continuously update these metric view as our internal user build additional metrics to basically support their own hunt. So we're actually learning from our internal threat researcher and even from external
[36:11] customer command when they're doing analysis. So demo time. Let's see this in action. We're going to show you a concept of an identity agent implemented in a swarm format running using genie and agent brick.
[36:29] Hi everyone. I'm stepping in today because Justin has been talking for 40 minutes straight and his voice gave up before he did. So he handed me the mic. Let's get into it. What you're about to see is an identity agent built using something called the udal loop. Observe, orient, decide, act. Each stage is its
[36:46] own AI agent coordinated by a central supervisor. Think of it as a team of specialists, each with one job, working together to analyze a security environment. And if you don't know what an udaloop is at this point, Justin really didn't do his job well today. Let's kick it off. I'll ask the agent to
[37:02] analyze our internal test tenant, Bacon Security. Watch what happens first. Before the agent does anything, it goes to memory. It's recalling what it learned from previous sessions, past mistakes, past findings, context about this environment. This is how the agent gets smarter over time, not just faster.
[37:19] Now memory is loaded. The agent moves into the observe stage. The observe stage is gathering raw signal. It's pulling in data, events, logs, activity, anything relevant to Bacon security. No analysis yet, just aggregation and collection. Think of it as the agent
[37:35] opening its eyes. Those collected and aggregated observations are now handed off to the Orient stage. This is where things get interesting. The agent starts correlating what it found against threat intelligence data, known attack patterns, adversary behavior, indicators of compromise. It's
[37:52] building a picture of what's actually happening, not just what was logged. Orient hands its results to the decide stage. Here, the agent assigns a priority. How serious is this? How urgent does it need to be treated? It reviews everything and makes a call.
[38:12] The agent generates a report and serializes the findings. Now, here's where it gets honest. The agent flagged this environment as production and rated priority 1. That's wrong. Bacon security is our internal tenant where purple teaming can occur. It's where we test, not where we operate. A priority one on a test environment would have people
[38:28] waking up at 2 a.m. for no reason. So, as the human in the loop, we flag the mistake. Watch what happens next. The agent doesn't just accept the correction. It reasons through it. It asks itself, "Why did I get this wrong? What should I have known? And how do I make sure I don't do this again?" It
[38:45] writes that reasoning to memory and reerializes the corrected output. This is the feedback loop that makes the agent better. Fast forward through the UDA loop. Observe, orient, decide, and look at the output. Correct environment classification. Appropriate priority. No
[39:00] false alarm. Same agent, same question, better answer because it remembered. Good. That's a wrap on the topic of agent. There's obviously more to share but not of not enough time. To conclude,
[39:16] here are the setting that work for Arctic Wolf. Again, these are conversations started for your data team with your data bricks account team, not as a copy and paste recipe.
[39:33] Okay, thank you for all your time. Um to learn more about Arctic Wolf, um scan the QR code. Um your feedback and also fill out your session survey. Your feedback directly shape what get presented in future summit. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.