Adobe Security Lakehouse: From Splunk SIEM to Databricks LakeWatch

Summary

Adobe built a security data platform on Databricks to address limitations of their traditional SIEM, including telemetry volumes that exceeded SIEM capacity, performance bottlenecks, detections locked in proprietary models, and AI data set requirements the SIEM could not meet.
Logs are routed through Cribl into both Splunk and Amazon S3, where Spark pipelines and Databricks Asset Bundles process data through a medallion architecture with OCSF normalization and YAML-based detection-as-code with CI/CD for scalable detection engineering.
An AI-assisted workflow migrated more than 500 Splunk detection rules to LakeWatch using two-stage backtesting for reliability validation, with MLflow managing ML models, sliding-window validation, and drift monitoring in a closed-loop detection engineering workflow.

Adobe Security Lakehouse: From Splunk SIEM to Databricks LakeWatch

Adobe's security lakehouse extends threat detection beyond the limits of a traditional SIEM using Databricks LakeWatch, Unity Catalog, and AI. Adobe explains how its security data platform supports governed telemetry, scalable analytics, detection engineering, hunting, and machine learning on a common foundation.
Learn how Cribl routes logs to Splunk and Amazon S3, where Spark pipelines and Databricks Asset Bundles process data through a medallion architecture. The session covers OCSF normalization, YAML-based detection as code, CI/CD, Databricks Genie, and AI-assisted migration of 500+ Splunk rules. It also examines two-stage backtesting, MLflow model management, sliding-window validation, drift monitoring, and closed-loop detection engineering.

Chapters

FAQs

Why did Adobe build a security lakehouse instead of relying only on their existing SIEM?

Adobe's security telemetry grew beyond what their SIEM could handle, and detections were locked into proprietary SIEM models that caused performance bottlenecks impacting detection observability. AI workloads also required data sets larger than the SIEM could support, leading Adobe to build a unified data platform for detections, hunting, and analytics.

What is Databricks LakeWatch and how does Adobe plan to use it?

LakeWatch is a Databricks capability for scaling threat detection beyond a traditional SIEM on a lakehouse foundation. Adobe is using it as the destination for their security data pipeline, processing OCSF-normalized logs through a medallion architecture and running YAML-based detection rules managed as code.

How did Adobe migrate over 500 Splunk detection rules to Databricks?

Adobe used an AI-assisted workflow to translate more than 500 Splunk rules into LakeWatch-compatible detections, with engineers reviewing and validating AI-generated translations. A two-stage backtesting approach then verifies detection reliability before rules are promoted to production.

How does Adobe manage ML models for security detection on Databricks?

Adobe uses MLflow for model management and lifecycle tracking, with sliding-window backtesting to validate model performance over time and drift monitoring to catch degradation. The platform also supports closed-loop detection engineering where AI agents help continuously improve detection rules.

Full transcript

[00:08] All right. Thank you everybody for coming. We're really excited for our first break out of the day here at Cyber Day. I mentioned it earlier in our cyber security forum, the customer Adobe who won our excellence in cyber security award. They've got a tremendous use case on Databricks building their own security
[00:23] lakehouse. You're going to hear that and then they've also been an awesome design partner for LakeWatch. So they really helped us along our journey. So I'm very excited to introduce Bharat and Robert.
[00:46] >> Good afternoon everyone. Thank you for joining us today and it's great to be here at Databricks AI Summit and I hope everyone is having a wonderful time. We are here to discuss about Adobe's journey from seam to data platform, how we are planning to use LakeWatch, Unity Catalog and AI to scale our threat detections.
[01:03] Quick introductions. My name is Bharat Gamini. I'm a lead security data engineer at Adobe. Over the past few years at Adobe, I've been working on designing and developing >>
[01:19] Robert. Robert. >> Hi everyone. I'm Robert Annen. I'm a senior security AI engineer.
[01:35] we'll discuss about what is Adobe's security data platform, the challenges we are facing with seam and how we are thinking LakeWatch can solve some of these problems. Later we will discuss about detection as a code, how we are approaching detection as a code for our seam as well as our data lake and how we
[01:53] are thinking that AI can uh accelerate our detection engineering program, and we'll take questions at the end. Uh feel free to ask ask us any questions you would like.
[02:12] So, security data platform at Adobe, um this journey wasn't about replacing seam. It was about addressing limitations we were hitting as our data analytics and AI needs continued to grow. Here are the key challenges we have faced. Security telemetry grew beyond seams can handle, and detections logged
[02:30] into proprietary seam models, and performance bottlenecks impa- impacted detection performance and observability. And AI required data sets that uh more than seams can handle. These challenges led us to build next generation of our
[02:47] security architecture with unified governance across data, AI, and detections, and detection engineering that is built like software, uh along with open standard security schema framework using OCSF. >>
[03:02] >> This provides us uh scalable analytics platform which can be queried across years of telemetry, and the outcome is a unified data platform that can be run for detections, hunting, and analytics using the same foundation.
[03:19] So, uh using the data platform, we are able to scale beyond the seam-centric detections, which is a evolution for us. Let's look at Adobe security data platform at high level. So, starting from the left, we have
[03:35] numerous sources we collect uh logs from and send the send these logs to our data routing tool Cribl. Cribl then sends these logs to both our seam and data lake. As soon as we get the logs into our data lake storage, which is S3, we process
[03:50] these logs using batch and streaming Spark jobs, which are orchestrated using Databricks jobs deployed via Databricks asset bundles. All the data flows through our medallion architecture from bronze to gold, uh stored in our
[04:06] Unity Catalog. All over end users consume this data from Unity Catalog be using SQL queries, complex notebooks, or using ML and AI workloads. We also have different user personas starting with our SOC analysts
[04:22] to our hunt teams and all over product and software security teams. All teams use this data differently, but they all use same common foundation managed using Unity Catalog.
[04:40] As our detection program started to scale, we started facing operational challenges with SIEM-based detections. Visibility into detection pipelines became as important as detections themselves, and we need to understand not just what the alerts were fired, but also the end-to-end pipeline telemetry.
[04:58] If we can't see how the detection pipelines are running, we don't know when they are failing. So, that's the that's something that we are facing issues with the SIEM where we are running blind, we don't know when the searches are failing or when the searches are running, but it's not producing the results either due to the
[05:15] vendor changing some fields or the schema has changed. So, whatever the rule was created against, we are not getting the results. So, we need this end-to-end telemetry to identify how the detections are performing. And detection failures are difficult to diagnose without the execution level
[05:31] visibility, which is very important for us to scale our detection program. And to operate detections at scale, we need better observability and better diagnosis and detections that are built like modern software.
[05:50] To address these challenges, we are looking LakeWatch as a potential platform to bring better observability, scalability, and engineering discipline to our detection program. As telemetry continued to grow, we need a platform that is scalable without having to compromise between cost and
[06:07] performance. Detection engineering should look like software engineering which which able to code, test, automate, and do rapid iteration. And LakeWatch provides that flexibility by allowing us to create detections using SQL, notebooks, and
[06:24] using ML models. Also, we need operational visibility into end-to-end pipelines. Without that, we cannot operate at at this scale. And by running detections closer to the data, we are able to run historic and real-time detections and correlation
[06:41] searches easily using the LakeWatch and Databricks platform. Through our evolution, LakeWatch has shown potential to help us operate detections with more engineering driven approach, which we are looking as we are continue to scale our detection operations.
[07:06] Yeah, this slide illustrates how detection engineering could operate on LakeWatch and the capabilities it could bring to our teams. Detections can be developed using LakeWatch, which is what it is mainly meant for. But as you can see from the left, we have different log
[07:22] sources. Using LakeFlow Connect, we can collect these logs and ingest the data into Unity Catalog. Along with that, LakeWatch also provides OCSF normalization. We have out-of-the-box marketplace presets that we can use to
[07:38] uh normalize these logs in OCSF bottom Or you can use your custom presets to build these custom log normalizations and enrich the data and store the data in Unity Catalog. Then you can build the detections defining those detection rules as YAML
[07:54] files and deploy these YAML files using Lake Watch APIs. You can create multiple detection type using the same CI/CD process and build these Lake Watch detections. As you can see, there are custom detections like IOC
[08:10] based detections, risk based detections. Everything can be done using the same infrastructure and follow the same CI/CD process. These detections can be scheduled either batch or real-time, and you can continuously find the detections as the data arrives.
[08:27] Then the detections will create output as notables and observables. So, as a SOC analyst, you can use these notables, analyze the notable, and expand the notable with the more information using Databricks Genie. As an analyst, you can come to
[08:43] Databricks Genie, open the notable, and get the insights of the notable by analyzing the data across your data platform and get those outcomes. So, overall, this is what the Lake Watch providing us the type of engineering driven approach we are looking as we
[09:00] scale beyond seam centric solutions. And security teams spend good amount of time uh correlating and translating data across different tools. A common data model can help eliminate
[09:16] that problem and create a standard data model that can be used for detections, analytics, and AI. So, Lake Watch provides this ability to normalize the data in OCSF format. Instead of building custom logic for each individual data set, you can rely
[09:33] on the Databricks uh presets to normalize the data. and cross course correlation is easier with the normalized security data and building and scaling of detections become very simple using normalized OCSF data. So,
[09:48] the idea is simple. So, normalize the data once and then use this data across your detection analytics and AI which will simplify the data analytics for end user. So, this is an example of how detection rule could look like on Lakewatch
[10:05] platform. So, it provides centralized view for detection rule and rule metadata along with consistent rule management across different detection type. As I mentioned before, you can deploy multiple detection types using the same model. Be it a simple SQL query
[10:23] or your complex pipeline that correlates data across multiple sources or batch and streaming detection rules, all can be deployed using the same model. And you get the visibility into severity, fidelity, as well as a detection status and the ownership using the same UI and
[10:40] the platform. This also provides a simple life cycle management across your environments from deploying your rules to the dev and then promoting it to prod, you can follow the same life cycle strategy for rule deployment. So, the
[10:56] goal is to make detection management more scalable, observable, and operationally excellent. So, as our detection program scale, it's not enough to know what the detections
[11:11] are firing, but we need to understand end-to-end telemetry So, end-to-end visibility into the detection platform is provided by Lakewatch by default. By using system tables along with the Lakewatch APIs, we can get this end-to-end telemetry.
[11:28] Uh understanding the rule performance and runtime is also achievable using the Lakewatch. Using Lakewatch you can see how each rule is running along with how much each rule is costing us along with how much how much time it is taking to run the
[11:44] rule. So, we can improve the detection reliability using these metrics and continue to continue to improve our detection processes. So, using uh using this operational data we can continuously improve our engineering
[11:59] decisions. And the objective is to manage the detection engineering as a software and software product service with measurable performance, impact, and reliability. So, that is the whole idea about why we are planning to use
[12:16] Lakewatch and how we think that it can help and scale our detection program and build our detections like a software service. Now, Robert will talk about detection as a code and how we are using AI to scale our detection program. Robert.
[12:34] >> Thank you, Bharat. Uh so, Bharat showed you the platform and I'm going to walk you through a little bit about an origin story, where we're currently at today, some lessons learned, um some self-inflicted gunshot wounds, and where we're going. But before I dive
[12:49] in, are there any detection engineers in the audience? Okay, great. We got a couple. Um anybody working on detection as code? Oh, sweet. Okay, love to talk to you all afterwards and hear how you're all building it.
[13:08] So, a little bit background of myself. I've been working on building machine learning systems for the last 15 years. And one of the things that I learned is building models or coming up with a model is pretty easy, deploying it, but getting trust or building that trust,
[13:24] getting the business to trust those models is the hardest part. And so that's going to be a common theme throughout this presentation. So where we started, probably like a lot of people, is we have over 500 detection rules in Splunk, all handcrafted,
[13:40] battle-tested, uh but no version control system. You know, we have everything in Jira, Slack, email, uh human memory, which is not a great governance model. So we modern we modernized that recently, and we developed detection as
[13:56] code. And one of the things that we bet on from the beginning was to make it platform agnostic. So that way, no matter what platform we use, seem, we can integrate it seamlessly.
[14:15] So what does that look like? So this is just a very simple model. We chose YAML as our kind of single source of truth file for our rules. Uh YAML allows us to pretty much do whatever we want, which is fantastic. So half of it is uh you know, platform specific, and the
[14:32] other half is just uh you know, platform agnostic. So all of our MITRE attack information, references, resources, notes, etc. is first half, and that's agnostic. And then the second half is, you know, our Splunk correlation
[14:47] searches, or our pipe SQL, or even pointing to references for Databricks custom notebooks for Lake Watch. So we're able to author once, and then deploy anywhere.
[15:05] So this is our process that we kind of came up with. We started working on this November of last year, and it went into production in February. So for those who are familiar with the software engineering life cycle, we essentially just ported that over to detection as code. So anytime an engineer, an analyst
[15:21] creates a new rule, and it starts with a PR branch. I mean, sorry, feature branch. And then we have a whole bunch of CLI tools that we built. Jira, Slack, Wiki or Confluence. We have a bunch of MCP
[15:38] servers and skills that help guide the analyst or the engineer on creating those rules or tuning them. So we have all the business logic, testing in there, and then we also have schema validation and we're using Pydantic data models for that. And that
[15:55] also helps us deploy to different platforms cuz every platform has different APIs, and so we need to be able to transform our YAML files to the platform of choice. We also have a bunch of testing. So we
[16:10] have style guides, additional business logic that, you know, over the years, so we've kind of taken that institutional knowledge and put those into tests. So that way before a rule ever gets deployed, it needs to make needs to pass all those tests. And
[16:28] we also introduce back testing and I'll talk about that later. So once a rule has gone through this process, a PR is created. Analysts can then go back and forth. All of that conversation is now stored in one place,
[16:43] and then we have a custom CI/CD pipeline that we then push those rules to production. So one of the things that we're trying to prevent is somebody just going into the UI, making some changes, they didn't document it anywhere. We can't quite
[16:59] stop that yet, but we at least have a process in place and we also have drift detection. So if somebody does need to make a change in the UI, we're constantly monitoring what's going on there, comparing it to what's in our Git repo, and then we automatically open a PR in
[17:15] case somebody does need to make a change in the UI. Okay, so that's a little bit about where we were. Now, we're moving over towards Lake Watch. So, we have 500 plus rules in
[17:31] Splunk. How do we get that over to Lake Watch? That takes a whole team, dedicated, you know, time and money, which we don't really have. And I'm sure a lot of cybersecurity teams don't have have that. So, taking years worth of Splunk content
[17:48] and trying to migrate that or convert it to Lake Watch could take a long time. So, we're leveraging AI, as well as our detections code repo to kind of help speed that up. And this is where the cautionary tale
[18:03] comes in. Uh so, Mitchell Hashimoto, creator of Ghosty, any Ghosty fans in the audience? Oh, sweet. All right. We're best friends already. So, he posted something on X recently that I thought was brilliant. And
[18:19] I don't know if any of you've been following him, but he's talking about AI psychosis. And he wrote a custom renderer, and he decided to redo it in Go and let AI take a shot at it. So, the first iteration in Go
[18:36] uh it was about 85 milliseconds. Then, after I think it was like 4 hours, AI got it down to 2 milliseconds. Sounds impressive. So, if you just blindly accept that, you're like, "Oh man, this is great." But, Mitchell Hashimoto's
[18:53] handwritten code was actually 75 times faster still than what AI wrote for him. So, his big philosophy is think, analyze, learn. And so, that's something that we've started adopting here at
[19:08] Adobe when it comes to detection engineering working with AI. So, how do we go from Splunk to Lake Watch? So, we already have our detection as code repo. So, that has all of our contacts, all of our lessons learned,
[19:24] our skills, how to use the different CLI tools that we have available. And then, we also have a great um in repository of information on these rules in Wiki. So, the agent can actually once it analyzes the
[19:39] the repo and understands the task, it can then go get additional context from the Wiki page. That's currently where we say whether a rule is active or not. And then, if it's not active, you know, it just stops the process and doesn't go any further.
[19:54] Then, the next is, you know, Splunk rules are pretty um there's a lot to unpack. You have macros, you have lookups, you have a whole bunch of other content. So, instead of just saying, "Hey Claude, convert this rule, make no mistakes."
[20:10] That doesn't really work when you have macros and information that's hidden from it. So, we have some custom-built classifiers and scripts to kind of to extract these for us to then give the agent more context when it's come when it's comes time to create the new rule.
[20:26] And then, we generate the initial SQL or a custom notebook if we need to do something beyond just pipe SQL. And I'll talk a little bit more about that later. Then, we have a backtesting gate. And so, this is what allows the agent to
[20:43] just continue to cycle through and get, you know, improve itself. So, it's kind of self-healing when it's writing the rule. And then, once it passes the backtesting gate, and I'll talk more I'll break that more down in the next slide, then a PR is created for a human to
[21:00] review and to scrutinize. But before I go into the back testing, this is one of our one of many AI psychosis moments. So, we were working on an ngrok tunneling rule.
[21:16] And as you see on the left, that's our Splunk, SPL, and then on the right is our pipe sequel. So, when we first wrote this without back testing, you know, the our agent did a fantastic job. All of our tests passed, schemas
[21:32] were validated, everything looks good. Let's deploy. An hour later, I went to go check Lakewatch, and we had over 3,000 notables created. So, a lot of false positives. And then when I went back and looked into to see what actually happened, why is this
[21:48] happening? Was our agent decided to take all those different term statements from Splunk and just collapse it into domain name like ngrok. But we're actually interested in those specific indicators. And so, it just made it too broad. But everything looked
[22:05] great. So, we went back in, cleaned it up. Obviously, this is much smaller compared to Hashimoto Hashimoto's example. But this is one of the many uh you know, AI psychosis moments that we encountered on using AI to help us
[22:22] convert uh SPL to Lakewatch sequel. So, then the real question is, you know, if it can run, does it actually detect? So,
[22:39] you a lot of times in software engineering, just because something deploys, runs, doesn't mean that the outcome is correct. So, how do we build that trust in what the agent did? So, one of those things is our two-stage
[22:55] back testing gate. So, the first, we create a very lightweight kind of recall. So, for the ML people in the room, you know, we're just trying to see is the agent in the ballpark. So, we just set a threshold of 80% for recall, and this allows the agent to just have
[23:12] some sort of benchmark to compare it against what's happening in Splunk. So, we're not worried about like if in Splunk we had 15 notables yesterday, we're just like did Lakewatch even detect anything yesterday? And then if not, you know, okay, agent
[23:29] goes back to the drawing board, and it keeps doing this until it reaches that recall. And this just helps speed it up for us. Then the next stage is we're now much more interested are we detecting it the same way and in the same time window as Splunk?
[23:45] Because ultimately what we need is to build that trust with the detection engineering team. And we don't want to come in and change how they've been doing things. So, we need to show yes, we can replicate these same detections in
[24:02] Lakewatch that you have in Splunk. So, that helps build that trust. And so, if both of these fail, then uh the rule's not created. Sometimes we need to go in. There's obviously more edge cases than just this
[24:17] um for backtesting. So, I'll talk about model-based detections next. Um but we do have uh you know, some different strategies for handling that. Uh sometimes there's some rules that haven't been active in a while, and we might not have the
[24:33] historical data in Splunk to test those. So, we borrowed some things from the ML community for anomaly detection, where we actually try and force the model to generate false positives. So, not just you know,
[24:49] sometimes you know, the Lakewatch rule will also say, "Yeah, we we detected nothing." But that's not helpful uh especially when we don't have historical information to validate against. So, then we came up with a sampling technique and some false positive uh, algorithms to kind of push that
[25:07] to help us with those rules that haven't fired in a while. So, now the interesting problem or challenge that we faced was uh, for those of you familiar with uh, the MLT toolkit from Splunk, it is their machine learning library that allows you
[25:24] to build uh, you know, you know, density functions, anomaly detection, or even behavioral-based models. So, those don't really translate well to you know, SQL or even a custom notebook. But, that's where we have the whole
[25:40] Databricks ecosystem. So, this is as of today, this is kind of how everything is set up for us within our Databricks environment. So, we're able to take advantage of the Databricks
[25:55] asset bundles, our the Lakewatch CLI, the Databricks CLI, their MCP server, and what we do is we use that to actually create the model, deploy it, train it, schedule it, sync it with
[26:11] Lakewatch, even run some tests, set up MLflow. And then once we have that model, then we deploy it to our serving endpoint in Databricks. We create tags so that way the endpoint you know, we always are working with the
[26:27] latest model. Uh, we can also, if we need some custom enrichment scripts, we can put those in our Unity Catalog volume. So, that way the endpoint can reach it as well as Lakewatch. Uh, all of our EDR logs are available. We have
[26:43] our lookups as well. And then the Splunk macros, we also converted those to uh, SQL UDFs. And so, then all of that gets through the AI gateway. We have Lakehouse monitoring. And then, Lakehouse, or Lake Watch,
[27:01] takes it from there. You know, it generates the notable for us and sets it up for the detection engineering team and the analyst to to work with. So, one of the things, too, that we had to come up with for those
[27:18] uh, MLTK models was how do we backtest those? So, if we need to backtest over the last 90 days, uh, you can't just take all of that, build a single model, and test it. So, those familiar with machine learning, you know, that's we call that target
[27:34] leakage, where we're taking information the model wouldn't know at prediction time, and we're kind of giving it the answers to the test. So, we needed to come up with a sliding window. Uh, you'll see this a lot in time series modeling. And so, we came up with
[27:51] this to kind of help us plug that gap with those model-based detections. So, and then, not saying that a daily retraining, or this is the way to do it. Again, we're building that trust. This is how these models were
[28:06] created in Splunk by our detection engineering team. And so, we're migrating this over, showing that we can get the same results. But now that we have MLflow and all these other tools, and we have access to uh, you know, drift detection and those
[28:21] things, we can now make more informed decisions on how much data do we need to train these models? How often do we need to train these models? As well as put some additional guardrails. So, what happens if the model sees new information that wasn't in a the training set before? So, we can then
[28:40] uh, you know, put these little guardrails in place to say, "Okay, if we see something that came out of bounds, you know, this is the default value for that. Or, you know, if you get all null values, what do we do? So, those are some of the cool things that we're able to now bring
[28:55] to our detection engineering team with Lake Watch and Databricks. So, where we're going is trying to put ourselves out of a job with closed loop. So, we're leveraging a lot of And some
[29:11] of these are internal tools that we're building at Adobe for cybersecurity. We already have LID, which is our log discovery and query helper. So, this builds our data maps for us. And this was critical in converting those uh
[29:26] Splunk rules to Lake Watch. So, all of the data that we have in Splunk we can easily map to what's in Databricks or in our Unity Catalog. Radar is our uh detection gap coverage tool. So, if
[29:42] leadership or an analyst or an engineer is just like, "Hey, this new TTP that came out, do we have coverage on that?" Radar will analyze all of the logs and our detections to actually see, "Hey, do we have something similar? If not, open
[29:59] a PR." What we want to get to is actually use Radar to open up a PR in detection as code. And then the next thing is Akasha, which is our agentic analyst. And so, that will actually run investigations itself.
[30:14] What we want to do is use Akasha to kind of help us fine-tune those detection rules. So, as it's analyzing detections, if it's seeing the same thing over and over again, can it help us improve our existing detections
[30:30] or recommend, you know, any additional automations that we might want to run on top of it? And so, this is the closed-loop system. All these tools are built and in place today. The next hard part is actually getting
[30:45] it to all work together as well as building that trust. So, thanks. I'm Robert Annan. We're going to open up for questions now. Please fire away or if you want to connect afterwards, I'll see if I can stand out somewhere. Pretty easy to find
[31:03] obviously, so come grab me. Love to chat with you all. But yeah, if you have questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.