Skip to main content

From SIEM to Lakehouse: A Practical Path to SOC Modernization on Databricks

Summary

  • Traditional SIEM architectures force security teams to sample and filter telemetry due to high ingestion costs — one cost comparison in this video shows $4 million for Sentinel versus $700,000 on Databricks for the same 20TB of data.
  • Anvilogic integrates with Databricks to provide a detection-as-code unified detect framework, running correlation detections natively on lakehouse data while AI agents automate detection engineering, alert tuning, and data onboarding for new telemetry sources.
  • This video outlines a phased path from SIEM-centric operations to a full lakehouse-driven SOC, covering automated alert classification, AI-driven false-positive reduction, and a realistic roadmap to SIEM replacement.

From SIEM to Lakehouse: A Practical Path to SOC Modernization on Databricks

Watch: From SIEM to Lakehouse: A Practical Path to SOC Modernization on Databricks
Security teams are hitting the limits of traditional SIEM architecture. Rising ingestion costs, fragmented telemetry and brittle detections force tradeoffs between visibility and budget. this video outlines a practical, phased path from SIEM-centric operations to lakehouse-driven security analytics. We will explain how SOC teams have cut costs and boosted performance by offloading telemetry from legacy SIEMs to Databricks, without breaking detections. We’ll also show how security teams run and correlate detections natively across their data sources, and use AI agents to automate detection engineering, tuning, and triage while reducing noise and vendor lock-in. Attendees will leave with a clear roadmap to building a modern, AI-driven SOC on Databricks, and how they can unlock broader detection coverage, lower costs at scale, improved triage efficiency and a realistic path to full SIEM replacement.
Talk By: Alan Mazankiewicz, Sr. Solution Architect, Databricks ; Jacques Reynaud, Security Field Engineer, Anvilogic ;

Chapters

FAQs

What is a security lakehouse and how does it differ from a traditional SIEM?

A security lakehouse uses a platform like Databricks to store all security telemetry at low cost in Delta Lake, decoupling storage from compute so teams can retain full-fidelity data rather than sampling it due to SIEM ingestion pricing. The cost difference can be dramatic — this video cites an example of $4 million for Sentinel versus $700,000 on Databricks for the same 20TB dataset.

How does Anvilogic integrate with Databricks for SOC operations?

Anvilogic provides a detection-as-code framework called the unified detect framework that allows security engineers to write, manage, and version-control detections that run natively against data in the Databricks lakehouse. The integration supports both a SIEM-plus-lakehouse transition model and a full SIEM replacement path, depending on the organization's migration stage.

How does AI reduce false positives in security alert workflows?

Anvilogic's AI-powered alert tuning analyzes historical alert patterns to identify detections that consistently produce false positives and recommends tuning adjustments or suppression rules. Alert classification into malicious, suspicious, and benign categories further reduces noise for SOC analysts, allowing them to focus on genuine threats.

What is the phased path to SIEM modernization on Databricks?

The phased approach starts with offloading high-volume telemetry from a legacy SIEM to Databricks to reduce ingestion costs, then running correlations natively on the lakehouse, and finally using AI agents to automate detection engineering, onboard new data sources, and perform triage. The end state is a modern AI-driven SOC with broader detection coverage and lower total cost than a SIEM-only architecture.

Full transcript

[00:08] Good afternoon everyone. Uh thank you for having the patience and coming over here on a late afternoon session on the third day. So how's the conference going for all of you?
[00:20] Awesome. Awesome. So here today we are going to be talking about um
[00:34] on lakehouses and uh how we are seeing the the whole space around the cyber analytics as a data problem and
[00:46] data analytics and agentic data problem. So just go through uh my name is Anand Rao. I am subbing in for Alan my colleague
[01:00] who was supposed to be here on stage. Unfortunately he is not doing well. So uh I'm subbing in. So big caveat there saying I'm these are his slides and his talk track. I'm trying to uh see if I
[01:14] can make the best uh for him and his absence here. So
[01:27] and we can even talk a little bit about what do we mean by a security lakehouse and then how an logic our partner uh for the databicks environment uh enables
[01:39] seams on a security lakehouse right I'm an from the I'm a field engineer on the databicks uh public sector sector team. So I work with a lot of the public sector uh defense industrial base and
[01:52] federal system integrator customers. So I have a lot of u deep understanding around the cyber analytics needs especially around the public sector world and also I am part of the theme
[02:05] team for our newly announced lakewatch team lakewatch product.
[02:17] I'm one of the field engineers uh working with many customers to integrate and on board them onto the platform. Um I've got a quite extensive background in sock and intelligence um parts of being
[02:29] in the sock environment. Um and yeah, so today we'll be talking about how Amplogic enables um you know customers to to leverage things like a security lakehouse to um implement their their end-to-end secoffs um infrastructure environment.
[02:42] Fantastic. Thanks Jacqu. So let's just jump in right you know before we jump into this slide here as I said like you know how many of you are databicks
[02:54] practitioners people who use databicks today almost all of you uh we do have a couple of people who are not using databicks but databicks is a data and AI platform by itself right you know we are no our
[03:07] our strength at databix is high-end data processing which means high scale and speed right so That's been our core focus. And then as we start seeing uh
[03:19] the cyber security space as a data problem, what does it even mean by saying that? And what do we even mean by saying we we are a security lakehouse? Uh the concept is pretty simple, right? You know, you bring in all of your data
[03:32] into the lakehouse and uh have them available for an analysis uh from a cyber security perspective. And uh one of the key things that we are seeing among our customers is uh non-organic
[03:47] decision making in terms of how much data they're actually pushing uh into the cyber security analysis tooling platforms if you speak primarily on a cost perspective. Uh cost makes it
[04:00] prohibitive for them to bring their entirety of the data set into their platforms for analysis. They're sometimes saying I'm going to move 30% of my data 40% of my data. I'm going to move data from these envir these
[04:14] platforms into the splunks and central ones and cyber rocks of the world and then not doing the others because that's only things that we can afford in terms of licensing cost whatever else right
[04:26] that and then that for me that is like I'm going to go make sure my front door and the front porch is fully uh stable and managed and controlled and I'm going to open the back door, right? You might
[04:38] as well not do anything at that point. So for us, we want to enable customers to bring all of their data into the lakehouse because we are uh a lowcost environment when it comes to data
[04:51] processing and data data handling and data uh you know as a data lakehouse, right? And also we have the flexibility for customers to bring structured data, unstructured data, semistructured data, all of them into the lakehouse. And
[05:03] that's what we call in in my world security lakehouse. And security lakehouse brings all of this data. It could be JSON's logs and any of those things into the lakehouse so that it can
[05:15] get analyzed. But then here is a quick comparison of uh you know we did uh a simple comparison around uh x amount of data on Sentinel 1 and
[05:27] compared it with the data lakehouse. Right? So we brought in like uh this was a comparison that was done with two terabytes of data three months and then 20 terabytes of so three months worth of
[05:39] data for analysis 2 terabytes into Sentinel and then 20 terabytes into lakeoffs and we brought in 18 months of data. So just a comparison, we're almost talking about 10 times the data. And
[05:53] then the annual cost there was around $4 million and this is all based on pure list price. So no discounts and all that juggling around. But then we brought
[06:05] comparisons straight up to data bricks and then bringing 20 uh terabytes of data which is 18 months worth of data 100 queries daily 5 terabytes each query
[06:17] uh from a processing perspective annual cost is around 700,000. So I'm going to let you analyze and be the judge of what it is. I'm not going to say anything. But these are the kind of things that
[06:29] will enable you to say I'm not going to make bad decisions. I'm going to make sure I'm protecting every asset that I have because your data is your asset and you want to protect it. And that's the
[06:41] core goal for all of you uh folks who are sitting here and talking about and listening to us. So taking a little bit of a step back for people who have not who are not familiar with databicks databicks is a at a core
[06:56] our uh the reason why we are able to bring that level of total cost of ownership reduction is purely because we've separated compute and storage and we use object storage for all of our storage needs and we bring in data at
[07:11] high scale. We have customers who are talking about tens of pabytes of data being loaded onto database uh processed for uh uh you know different needs
[07:23] building out medallion architectures curating data and all of that and then from a pricing perspective I'm not going to read every line of this but you start seeing a clear comparison here you know when we're talking about uh Microsoft
[07:36] Sentinel versus data bricks uh what's the tier what's the data you know like the data here as well as you know from a databix comparison what is the kind of compute we are talking about trying to
[07:49] match apples to apples and sometimes even better right so that's the idea here pricing structure for 1 tabyte of data uh like in this example we are storing data in ADLS buckets which is
[08:01] the object store in Azure and uh the injection and the query costs are right up there for you I'm not going to read it for you but uh you start to get a feel for why you want to start thinking
[08:13] about the lakehouse as your store for all of your data for doing cyber analytics and that's what we call a security lakehouse and this is an example of one of our
[08:25] customers uh start chartered who had similar challenges right you know expensive uh seam products that they were into difficult to manage them because then they have to decide what
[08:37] part of the data am I going to send am I going to send my right hand or the left hand to protect That's not a decision you want to do, right? You want to be able to protect yourself across the board. So that's uh a common problem. All of you are facing that I'm sure and
[08:51] some of our key customers are facing it. And the solution and the impact it's obvious, you know, I don't have to go into the details there. So that's where I'm going to introduce my uh friend
[09:03] Jacques from Animal Logic. uh because we are all solving similar problems because we want our customers all of you to get the best benefits out of all of the technology that we are developing. So we
[09:17] are we at data bricks are building our own products and you know building solutions that can help you solve these problems and make your life better and then we have partners who are going along right with us to help you as well.
[09:30] So handing it off to uh Jacques to take it through for the next slides. >> Thank you Nan. I appreciate it. So I just want to spend some time today uh talking about Applogic and how we as a company and and a platform are trying to
[09:42] modernize the sock and part of that strategy is also leveraging things like data bricks as a lakehouse. Um so what oh we're going backwards.
[09:56] There we go. Okay. So what is the benefit of uh leveraging a uh lakehouse such as data bricks? Right. So as an already mentioned that total cost of ownership is massively reduced. Not only are you moving away from uh the legacy um per gigabyte uh ingestion costs u
[10:10] from seams such as like Splunk, but also the underlying data store is much much cheaper. You can leverage things like uh Amazon S3 to store your data as a you know data platform. Secondly, we've got a much faster time to detect and
[10:23] respond. So, not only are you able to stream data in near real time using things like cribble and cafka and some of uh the data bricks technology that they've recently released this week um but also the detections actually run in
[10:36] data bricks run incredibly efficiently with with the architecture we've implemented um and you can see near real-time alerting right so uh detections running in seconds rather than minutes or hours you know in legacy seams you'd see um traditionally and the
[10:50] third uh sort of benefit you're seeing here is that modularization and open um approach where you can swap out these technologies as you do as your business requirements change. You can change your streaming um technology, you can change your your data store um and you can move
[11:03] as your business requirements change and as they adapt over time.
[11:15] it's a quite a complic complicated uh diagram but I'll try and simplify it keep it high level. Um on the left you see your data storage platform. So we support things like S3, Azure blob, GCP and what we do from AML logic is
[11:27] actually on board that that that data from where it sits. We don't move it. We we bring it into data bricks. It's within your environment and we bring it into uh the bronze table. So we leverage the medallion framework for your data pipelining within data bricks. So we
[11:40] bring that raw data into bronze. We store it there and we push it through the the DT pipeline through silver to gold. And gold is our like normalized and schematized uh version of your data.
[11:52] And that's where our detections run on top, right? So having your data normalized and schematized in in gold enables you to create very efficient queries um and run them incredibly quickly. And so we deploy our detections on top of that gold layer that that uh
[12:05] DT table and uh those detections then feed the alerts that they produce then feed our alert lake and that's that EI table. It stays all within your data bricks environment. So it's all local. Um so talking about detections, how do
[12:17] we build and deploy detections? Well, we have um our unified detect framework that we uh implement for all of our detections and that framework leverages uh detection as code, right? So we
[12:31] identified a number of issues when it comes to detection engineering as a whole. There was a SANS detection engineering report uh produced in 2025 that highlighted three kind of key issues that engineers face. Um the first of which is resource and time constraints. So it's traditionally quite
[12:44] a slow process to build out these detections. Um second one is the complexity of the threat landscape. So being able to uh accurately uh sort of map these detections to tactics and techniques that are seen in the threat landscape. And third one is also the
[12:56] difficulty in validating the effectiveness of these detections. So with our platform and the uh detection engineering aspect of it with the unified detect framework, we can modul modulize your detections into into uh
[13:08] code reusable code that you can use across multiple detections which massively reduces the time and effort required to build new detections. And as part of the framework, we also um automatically map these detections to
[13:20] the microtack framework. And so that you have that tactics and techniques mapping already built into the to the detection use cases that you're building out in the platform. And this um overall brings down that uh that time to build, time to
[13:33] deploy um and increases your precision and it also increases your your coverage because you're mapping directly to MITER. So number of benefits here from this framework um and these are for our atomic and signature based detections and what we do from there is we uh do
[13:48] our we have a correlation framework, right? So we are able to build correlated detections that that bring together multiple uh signature based detections in in groups in stages um and
[14:00] correlate them across multiple asset classes like users, IPs, hosts, domains. And what that does is it paints a picture of this attack has happened. These these different atomic detections have fired in this sequence in this
[14:13] order which is has been seen before in a realworld attack. Um and so what you're doing here is building much higher fidelity alerting using um this correlation framework for your detections and within the platform is
[14:26] incredibly simple process. It's almost like a drag drop exercise where you can group detections together those atomic detections but and you can time frame them and you can um you know stage them out into multiple stages quite easily.
[14:38] And the last kind of core component of the platform um when it comes to the detection engineering life cycle is tuning. So we also have a machine learning based and AI based um tuning model which continuously tries to identify tuning opportunities of the
[14:51] detections you have deployed in in your data bricks environment. So it uses that alert lake that EOI table to understand any correlations or patterns um of detections that have been firing frequently. So those high volume um
[15:04] alerts and identifies opportunities to potentially tune those alerts down using the model um that that's powering the back end. And this is continuously being run. Uh it's not there's no user input to to trigger these these workflows. And
[15:17] what we're seeing is with the um with the with the tuning insights like a 90% reduction in in volume of alerts which is massive when it comes to alert fatigue and um your analysts on a day-to-day basis looking at these these detections. Um
[15:30] so that's kind of like the core components of detection during life cycle from a manual perspective. But the um sec this next section I want to kind of dive into the secop agent secops capabilities that we've been building
[15:42] out over the last few years. And just to start off with I just want to kind of go over the the high level architecture of what they what they look like. So we've we've been building agents and automated workflows to help multiple aspects of the secops life cycle. So we've got our
[15:55] data on boarding uh data normalization aspect. You've got the search and detect which is more the engineering uh side of things. You've got the alerts. your lriage, investigation, um, and remediation steps. And then underneath
[16:07] all that, we've built an AIOS, which is like a fabric that pulls together all these agents. So, you can build and orchestrate multi-stage um, automation routines that leverage these different agents together to perform um, like sort
[16:19] of like highcale um, lengthy workflows that require multiple steps and decisions to be made. So just to go through some of these agents, the first one I want to just hit up is the data
[16:31] boarding data on boarding agent. This is quite obviously one of the first steps in getting your data in from uh in data bricks and being able to actually deploy detections. We have built out a um an agent that essentially takes the the the
[16:43] data that you've got in bronze. So you've not normalized it yet. You've not fully onboarded it. And it will build out that medallion framework pipeline by uh contextually understanding what is that bronze table? What data do you have in it? what data source is it? Um, and
[16:56] it will build the PI Spark code to actually deploy in your data bricks environment and normalize that entire uh data set into gold. And that's then makes it ready for detections to be uh deployed against it. And we've been
[17:08] seeing um our customers leveraging this data on boarding agent seeing like massive benefits when it comes to that on boarding process. We had one customer move from Splunk to data bricks in three months uh with 71 data feeds and it took
[17:21] on an average uh to do the normalization and data on boarding process about eight minutes per feed which is incredible. Um and as a result they saved nearly a half million dollars in um analyst time and effort.
[17:34] Uh the next agent is the search agent. So this is like our uh search capability that takes natural language queries. you can talk to the you can prompt the agent um with natural language to build out queries that search across your entire
[17:46] data set uh in in multiple environments. So we um have a federated approach where we can search multiple different data platforms. We support Splunk, data bricks, uh, Azour and and Snowflake. And in within this, um, this search
[18:00] agent, you can search across all of your data anywhere it sits. Um, and the search agent will build the queries and know where the tables are, know where the data sources are, understand the the syntax and the language it's got to use
[18:12] and build out complex queries to look for whatever thing that you want to find. Um, and so this is a really powerful feature. really brings down that time to create searches and and and following on from that detections themselves, but it also lowers the bar
[18:25] to entry when it comes to your analysts that are trying to leverage um it's you know some analysts might know SQL if they're moving to data bricks or snowflake and so this really helps a lot of like junior analysts to learn not only learn the language but also be able
[18:37] to create queries um without necessarily having to know exactly how to create a search. So say you've built a search and you're looking to document this and build this into a detection. Uh we have our build
[18:49] agent and this takes with a click of a button the the search query that you've built and documents and does all the audit around building out a detection on the platform itself. So it'll um format the query into our unifi detect
[19:01] framework um format. It'll uh add the use case description. It'll have all the tagging all the MITER framework mappings with the tactics and techniques that it's identified in in the actual search itself. and then obviously build out
[19:14] that detection and and its order history. So you've got all that there in the platform with the click of a button. And this again reduces that t that overhead that time needed to build detections on the platform and deploy them. And so when it comes to okay, we
[19:26] want to deploy this detection. Well, before we do that, we actually want to tune it. Um having detections deployed without any
[19:38] peak in alerts when you initially deploy a new detection. That's because it hasn't been tuned yet. And so what we have is our tuning agent and this enables analysts before they deploy a new detection uh to actually tune it using uh the data that's already in the
[19:50] in in in data bricks for that data source. They can run that detection over the last 90 days and and find all the events that would have triggered and the tuning agent will go through that data and actually analyze opportunities and
[20:02] correlate the data sets to to figure out okay well actually you've got multiple opportunities here to tune out um all these events that would have fired otherwise. Um and it'll do that in a smart manner. It'll do it has contextual understanding of the data that it's
[20:15] looking at, the events that it's looking at and also it will do research as to okay is this type of activity that I'm seeing as a result of this alert good or bad and it'll understand whether the opportunities that it's finding are known good or known bad behavior. So it
[20:28] won't provide you with a tuning opportunity that will actually tune out malicious behavior which is obviously you don't want to do. Um, so this is a really powerful way in reducing again that alert volume um by uh pre-tuning it
[20:40] before you deploy. And again, we've had customers reduce their alert uh volume by 80% using this uh this agent and saving multiple hours per detection uh in doing so. The last agent that I kind of want to talk about today is our alert
[20:53] analyzer agent. And this is like when the alerts have come through, they fired on the dashboard, we want to do like an initial triage or uh on that alert and figure out what's going on. And what that agent does is just that it will it will take that alert or the correlation
[21:06] based alert in this example you see on the screen and understand exactly what's happened in this chain of events um and it will document everything. It will pro provide a report and more importantly um it actually does a verdict of what this
[21:19] alert is. Is it malicious? Is it suspicious, benign or or just unusual? And that um classification is really important when it comes to triage analysts. They need to know what's important to look at. And by by enabling
[21:31] this this alert analyzer, it really helps drive efficiency in only looking at the alerts that are important. And not only has it really documented everything that's happened in that alert, but it's pushed up and prioritized the ones that the analysts
[21:44] need to look at first. So, as a result, you you reduce the number of alerts you you triage because you're only focusing on that are important and that saves hundreds of hours of daily, especially in large um large organizations where they have hundreds of alerts on a daily
[21:56] basis. So, I mentioned blueprints earlier that kind of sits on top of our AIOS um for fabric and I just wanted to dive in as to you know what is a blueprint Rob? What do they do? Um and this is a new
[22:09] feature we've been building out so we're trying to you know showcase it to everyone that we see. Um so a blueprint what is it? It's a multi-step um uh orchestration routine that you can build
[22:21] um using an analyst knowledge and context to uh enable entire workflows to be automated at scale. So we can build out um entire workflows that can be
[22:33] applied to things like data on boarding, detection, engineering, maintenance and health, investigate and respond these areas of the whole secops life cycle and we can automate entire workflows. So what does that look like? Well, we had a
[22:46] use case with a Fortune 200 uh company, the oil oil and gas industry, and they were looking to on board a number hundreds of clouds uh cloud detections into their um production environment. But they had the issue where they
[22:59] couldn't uh you know, increase the headcount of the analysts looking at these alerts because, you know, traditionally as more alerts come in, you need more people to look at them. Um so, how did we find a way to um help them out with that? So, this customer was a multiplatform customer. They had
[23:11] both uh Splunk and data bricks. So what we did was we built a end-to-end uh L1 um triage blueprint. And what that what that did essentially was um triage and
[23:23] investigate the alerts that were coming through uh from these cloud detections and automate that entire triage and investigation process so that these these analysts weren't having to manually do it themselves. And the result well they were able to on board
[23:36] these detections actually drive down their MTR because obviously these blueprints were automated. So they uh were running very very quickly and coming to conclusions on these detections very quickly and their
[23:48] accuracy actually improved. So um number of benefits and there was actually an an unexpected uh benefit as well where analysts were being upskilled as a result of being exposed to the blueprint
[24:00] that was running across these detections. They could see the logic that's being applied the the steps they had to do to triage these alerts um effectively and as a result were being um sort of educated in that process as well.
[24:12] So this is just an example of what that blueprint looks like. You can see that multi-step um sort of workflow. You got initial alert context. So putting in that triage alert, it then does um additional contextual analysis. So it
[24:24] looks for uh it does applies additional like sub searches into your data sets to pull in more information about what was going on around the event. And then the third step or doing third party enrichment. So pulling in things like
[24:36] showdown um virus total information to further enrich that that um information on that alert and then coming to a decision of like is this malicious? Is it benign? Do I need to do any further steps and if so if it is malicious then
[24:48] doing that additional um sort of like uh exposure checks where you're trying to figure out is anything else impacted and then finally logging that report uh logging that report and investigation into a ticket and sending it off to your
[25:00] s or your forensics teams. So it's kind of like the general layout. quite a simple one. Um they can be very very complicated and that's completely down to the use case you're applying it to. So you you might be wondering okay so
[25:12] I'm I'm on Splunk for example or on on a legacy team that I want to migrate away from and and start leveraging data bricks and all these capabilities and these features and all these things all the time I could and money I could save as a result. We actually will support
[25:25] you in that process because we have the ability to support multiple multiple data platforms at the same time. So we can have you on Splunk and data bricks throughout your migration process and that enables you to follow a very uh
[25:38] simple and uh repeatable process of onboarding your data into data bricks um ensuring you've got detection parity you got the same number of alerts coming through and once you're happy with that whole onboarding process for that
[25:50] particular data set or that data source it's Windows logs you can then turn off the feed in your legacy seam and we will support you throughout the entire chain of process so um you will never lose any of your production alerts
[26:02] you there'll be no gaps in your monitoring um during the whole process. So it's something we support and it's just uh yeah a great feature of having that federated search and that is everything I wanted to go over today. I think we'll open up for questions,
[26:27] problem and how you can use the platforms like databicks uh and uh platforms that are built on top of databicks uh as a data platform to solve your cyber security needs. And uh
[26:40] there's a neat solution that uh you know shows you how to do that. So appreciate the time Zach and uh we're

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.