Lakehouse-Native Cybersecurity: Multi-Agent Simulation and Adversary Exposure Management

Summary

This video presents a Lakehouse-native cybersecurity architecture featuring an Agentic SIEM that normalizes data from seven native sensors and more than 900 third-party sources into a governed Lakehouse schema for continuous threat simulation.
Eight specialized sub-agents orchestrate the full attack simulation lifecycle — including reconnaissance, path prediction, strategy selection, script generation, and success condition evaluation — to verify real exploitability with confidence scoring based on data coverage and fidelity.
The digital twin approach transforms the Lakehouse from a passive data store into an active validation engine, verifying whether attack paths are actually exploitable rather than estimating cyber risk from static assessments.

Lakehouse-Native Cybersecurity: Multi-Agent Simulation and Adversary Exposure Management

Modern security platforms collect massive telemetry but remain reactive to threats already in progress. this video presents a Lakehouse-native cybersecurity architecture powered by an Agentic SIEM that transforms raw logs into simulation-ready intelligence. An AI-driven SIEM normalizes heterogeneous security data from seven native sensors and 900+ third-party sources into a governed Lakehouse schema with Universal Data Pipelines and common data schemas.
From that unified foundation, a multi-agent system simulates realistic attack chains including privilege escalation, lateral movement, and credential harvesting, validating each step against Lakehouse-derived state. Rather than estimating cyber risk, this digital twin verifies exploitability with confidence scoring based on data coverage and fidelity. Discover how eight specialized sub-agents orchestrate reconnaissance, path prediction, strategy selection, script generation, success condition evaluation, and judgment across a Continuous Threat Exposure Management lifecycle. Learn why the Lakehouse becomes an active validation engine for proactive, scalable, and continuous adversary exposure management rather than a passive data store.

Chapters

FAQs

What is Continuous Threat Exposure Management and how does the digital twin support it?

Continuous Threat Exposure Management is a framework for continuously validating and managing an organization's exposure to cyber threats rather than conducting periodic point-in-time assessments. The digital twin simulates realistic attack chains against Lakehouse-derived state, providing ongoing verification of which attack paths are actually exploitable at any given moment.

How do the eight sub-agents divide work in the attack simulation system?

The eight specialized sub-agents each own a distinct phase of the attack simulation: reconnaissance gathers environment state, path prediction models possible attack routes, strategy selection chooses the approach, script generation creates the attack code, success condition evaluation assesses outcomes, and additional agents handle coordination and judgment across the pipeline.

What data sources does the Agentic SIEM normalize for threat simulation?

The Agentic SIEM normalizes heterogeneous security data from seven native sensors and more than 900 third-party sources into a common data schema stored in the governed Lakehouse. This unified schema enables the multi-agent system to reason across all security telemetry consistently.

Why is validating attack paths better than estimating cyber risk?

Risk estimation relies on assumptions about what an attacker could do, which can miss real vulnerabilities or overstate theoretical ones. The digital twin validates actual exploitability by simulating attack chains against the real Lakehouse-derived environment state, providing confidence scores based on actual data coverage and fidelity rather than expert judgment alone.

Full transcript

[00:08] Hello everyone. Welcome to the session. I know I'm a spark and this is Dennis. We know that we are the things block between you and the lunch. And also between you and the weekend. But I really appreciate you can join this
[00:23] session. We believe it will be very interesting. And our topic is lighthouse native cybersecurity rating simulation. We'll cover something about the agency theme and digital twin. They are all big word, right? And so I will efficiently use the time.
[00:39] So I need to define what is the rating operation in this session. I will tell you what and why. And the secondary Dennis will help us to try to define our definition about the agency theme and cybersecurity digital
[00:55] twin. And we'll come out some live demo. And finally we will come out some lesson learned and take away. Okay, that is agenda. So
[01:10] every rating operation or every rating story have a start. has an end. About the start can can be very simple like maybe just your environment. Any of your employee just clicks a
[01:26] phishing email. So your credential is stolen. Or it can be very advanced or very complicated like we know the mythos like the LNM that is so advanced. It can just found a unknown
[01:42] vulnerability in your machine. So it can just enter your machine. Either way the right right now the the hackers or the rating had a full hole in your environment. And the right side actually might be very different.
[01:59] Maybe just your credential is stolen. Or maybe your customer data is is exposed. Or maybe just your entire infrastructure is compromised. So the hacker is using your GPU happily. But maybe none of them happened because
[02:16] your security features, your defense take the role and nothing happens. So the hacker didn't go that far. But today, I didn't talk about the start and I also didn't talk about the end. I want to talk about everything between.
[02:32] I want to try to convince you that in the between actually we are talking about how the hacker collect information. That is data, right? And how the hackers do the decisions. That is the reasoning, right? And then how
[02:48] the environment validate this attack path is success or not. Why this rating operation is success? Why this operation is fail? So we want to turn the middle part that I want to try to say that the rating
[03:04] operation actually is a agent AI agent reasoning in our lighthouse query problem. So um allow to me to do a little deeper that about the between the start. In the between, the the rating rating
[03:21] model will do a number of discovery things. They might use a lot of rating tool like SharpHound, like the BloodHound, like like a lot of the legitimate tools to do the to talk to your AD server, to talk to your network to collect some information.
[03:38] And then you would using this information to building some map. And then to do the execution. I think the whole process is not much different from a James Bond movie, right? Every time you you go to
[03:53] this movie, you know James Bond go to the front desk talking to to the lady, and then he he is building a map, and he know who which account is in which room. So, about execution might be just depends on the situation. Maybe the key
[04:09] is just on the wall. On the wall is just like the the your credential actually is in the Windows LSASS memory, so you need to to get that key. Or maybe sometimes it is in the safe, or maybe it's in the lady. So, you you you need to talk to that lady. So, it's it's always based on
[04:25] the data to make the decisions. So, um let me let me just take a a little example further that for example uh in the current workflow that you will do a number of reconnaissance, that is data
[04:41] collection. And then you will pick out one of the candidate strategy. Here we always use the cybersecurity common language is a MITRE. MITRE define defines the uh tactic, technique, and what's the procedures. And then about the digital twin side,
[04:57] because we collect all the data. So, actually we we need to reasoning of the data that hey, what are those success condition? Do we collect those from our sensors? And finally we collect all the the the things and provide to the judge, and the judge will say hey, this success
[05:12] and this is fail. So, uh um I I I believe that everyone go to the keynote and when we go to a keynote, we know that hey, the whole uh Databricks ecosystem will collect our data. And we also heard about the Lay
[05:28] Waze Lay Waze story, right? Right? We will have the Lay Lay Waze story. It mostly talk about the detection. And but detection is something we call is a reactive, because some bad guy is already do something in your environment, so we do the reactive
[05:45] detection. What we want to present here is we want to push from the reaction to the to the pro- proactive that we believe that every enterprise customer want to continuous validate exploitable attack path in their environment. But there is
[06:02] a big security gap here we want to address here. Because it's really hard for three reasons. The first one because traditional pen testing or rating they they are uh quite a lot of the time. Human teams have limited of
[06:19] hours. They might pull a handful of the path, but they only have the time to validate the most critical one. So the the coverage is partial. The second one is because the live rating mean actually exercise the real
[06:34] system, carry the real risk. So for for many industry we when we facing those customer, they told me that for example in the health care, the in the financing in the transportation system if this kind of thing the rating mean brings out disruption, actually it have
[06:52] a serious business safety concerns. So most of the rating they told me that in the real situation, the enterprise just circle a very small area to let the pen test to do the to their their job. So uh it it is the
[07:08] second reason. And the third one is because it take time. So it it bring the blind between test because your environment change every day. Uh new new employee, new settings. How do we know that something between there are some risk?
[07:24] So what's our proposal is maybe we can do this kind of rating operation over we call the digital twin. So we can use the AI to validate every attack path. We can find the all the path that matters and we validate everything safely because we
[07:39] actually didn't run those script. And finally we can do those continuous validation, so you won't have no problem between between the things. So, uh about the technology foundation, actually there are four important part.
[07:55] The first part is about agentic thing. The agentic thing is try to collect all the security data telemetry at scale. So, of course, everyone in this room might know how to use a Spark for your ETL, how to define your data format, how to make it really efficient.
[08:12] And the second component is the digital twin, that how you can make your data can talk, can interact with you for your use cases. For this one, we start from first, you need to have your data in control, so you see my EP your
[08:28] candidate. You need to have a lot of computation to build application layer, how to make it scale out and scale up using GPU or that kind of clusters. And then, when we do things, we need to think about the agent harness. So, of course,
[08:44] you learn a lot about the data bricks in the keynote and the section, right? And how to use the lay base to build your agent's memory in efficient. Or some of you, maybe you are right now using lay graph or maybe you use agent SDK, that's your options.
[09:00] And then, in the end, the validation, when we to validate those rating simulation things, not only validate the security part, we need to validate the agent. So, we actually need a lot of telemetry to see things. You can use the MLflow, you can use Grafana, lay fuse,
[09:17] that kind of things. So, these are the technology four layer things. So, I I I hope that everyone have a clear picture what we are doing. Then, I will hand over to Dennis to talk about the digital twin. Thank you.
[09:43] Okay. Uh thanks, Buck. Uh as Buck mentioned, cybersecurity is the data problem. It is an art to connect data into insight and actions. So,
[10:01] So, uh before talking about the HDA thing, let's start from the CTM. Cyber uh the CTM is the continuous threat exposure management. Uh named by the Gartner, which is the risk the modern risk management framework.
[10:19] CTM is covered five phases, five stages. Uh the the discovery, the scoping, discovery, validation, validation, and the mobilization. For scoping, first, you need to scope the critical
[10:36] assets, such as your CEO account, such as the domain controller, which uh you need to understand the risk of loss critical assets. Then, you need to uh discover discover the
[10:51] critical assets discover the attack surface. Uh for example, uh the email, what kind of email will send to your CEO account, might be the phishing email. And uh which endpoint will connect to
[11:07] your domain controller, maybe it is the compromised end endpoint. And uh next, to prioritize the risk in your organizations through the attack path prediction.
[11:22] The most critical one is the validation. Uh we use the digital twin technology to build up our adversary exposure management. Uh the difference between the traditional rating mean and the
[11:39] adversary exposure management is that the rating mean uh it might break your business if it is too aggressive. But the risk simulation will not. Second,
[11:54] uh the traditional rating mean is very expensive. So, it can run once or twice a year. However, the uh adversary exposure management can run at any time without cost concern.
[12:10] Last but not least, the uh automated mitigation and the remediation uh through the result of the validation. So, this is the whole life cycle of the C-TAM framework, which is include in the
[12:26] cyber risk exposure management. So, let's talk about the agent thing. It is the foundation uh of the Vision One Vision One platform. And uh we have seven native sensors such
[12:43] as the endpoint, email, cloud, network, identity, AI, and the data security. And uh we also support more than 900 plus data sources, third-party data sources, such as the Palo Alto network
[12:59] firewall, such as the Cloudflare, such as the Microsoft Defender, and so on. And through the three layers, the data lake, the threat intelligence, and the agent thing layer, attacks cannot hide in our
[13:16] platform. So, take this attack scenario uh as the example. So, the host was compromised by the account takeover. Uh through the email attack. This is the
[13:32] detection from the endpoint. And by adding more third-party data logs, uh for example, the firewall logs, uh the router and the switch data into the agent and SIM,
[13:50] then uh we can understand the attack, how the lateral movement did and uh how the payload was downloaded from the CNC callback server. So, by combining the native sensor and
[14:06] also the third-party logs, uh we discover the attack attempts. And uh moreover, for the attack post petition, uh we help and also uh guide user how to
[14:22] fix the critical issues and uh prevent the similar attacks uh happening in the future. So, how how can we achieve this? First, consider the data through the universal data pipeline.
[14:41] So, uh we have the data the universal data collector, retrieve the data from diverse data sources. And uh uh the data sources uh we have the we will extract the data
[14:58] and uh we will transfer data and uh we will load data into the unified data lake. So, once the data has been used by the security modules in the the the Vision One platform,
[15:14] then uh the security outcome, different security outcomes can offered, such as the detection and the response and the cyber risk exposure management. Second, the common language of the AI agents.
[15:31] Uh the common data schema. So, we have the AI schema agent can continuously turn data into insight and it connect the different agents in our platform such as the triage agent,
[15:47] uh the investigation agent, the auto response agent. So, they can talk to each other and then they can work together. And third, there's a powerful threat intelligence agent behind the scene. Uh Trend Micro AI
[16:03] uh has a worldwide uh loan DDI program, zero-day initiative, which covers uh 73% zero-day vulnerabilities around the world in 2025.
[16:18] And this is because we have the AI enhanced uh AI enhanced security intelligence and research. We call it ACER, A E S I R. Uh with the ACER, it not only discovered vulnerabilities,
[16:35] but also it can abstract and uh also prioritize the mitigations and remediations. So, with the ACER and the threat intelligence agent, uh
[16:50] we we can have the progressive data refinement, refine data with a meaningful correlations with the full threat context. So, uh it turns the data from brown data to the parent data, which is the AI
[17:06] ready data. So, with these foundations, the unified data lake, the threat intelligence, and the AI agents, uh the twin is impossible. So, So, Spark, so can you give more
[17:22] technical details on how the digital twin run in our platform? Yes. Thank you, Dennis. So, I will go through a little about the agent harness and I will do the demo. I think I actually joined the training
[17:38] session in Monday. The agent harness evaluation actually a long day uh talk. Today, I just want to talk one thing. The rating operation actually is not a single agent problem. It's a multiple uh agent problems. So, you can see that
[17:55] from the left side is a rating. The first guy actually it it do the lighthouse query and create a map. And the second guy here
[18:12] Oh, it actually reasoning about the the the the entire ontology graph and build build the path. Maybe you will think about what's the shortest path. You will do some algorithm search. And the third one is doing the strategy. You're thinking about which minor tactic technique will use.
[18:28] And the fourth guy is the operator. He generate the actual scripts. However, we didn't really run this script in the custom environment because that will bring some harm. So, this kind of script will hand over to the the the other side the the the booting guy.
[18:45] The first one is the resolver. He reasoning about the script and thinking about what's the success conditions related if we will judge this script will success or not. Then it will hand over to our we call the digital twin. He will do the
[19:00] lighthouse query to collect all the data. And we we put all the collect evidence and the contest and intention and hand over to the judge. And judge will make the decision if this rating operation success or not. And then do the world update.
[19:17] So actually we mapping this uh eight different kind of sub agent concept to the agent framework. So we you can implement it with LangGraph, you can using uh uh Genie, you can using Agent SDK. But in the end, you you you just can simply to
[19:35] talk to your AI model that hey, you are the experience rating expert. I provide this uh tool like MCP, I provide this skill. What's your strategy and what's your your next move? So every of the agent actually we do very similar uh manner.
[19:54] So allow me to switch or to our demo. Th- This demo is uh might be uh a little complicated, but I th- this is a design to let everyone know what's happening. So the So the first screen is the dash dashboard. I I want to deliver three
[20:10] message. Uh the first one is actually the whole digital twin is a data pipeline things, right? Like uh Dennis mentioned, we need to have a data ingest. And then we build the ontology. And then we build the attack graph and attack
[20:25] path. Then we do the simulation. So it's actually a streamline pipeline. The second message I want to deliver is it actually cannot be achieved by one step, right? It actually require a number of iteration.
[20:41] For example, data. Maybe we notice that hey, there are some uh devices in your environment didn't deploy the sensor. So actually we don't have the visibility about what happening there. So there are a number of action need to do or to enhance in the data perspective.
[20:58] The second thing is when we have the data, we build the ontology, but some of the ages it actually cannot be just achieved by just a very simple rule. We we need some kind of expert to review. And for the attack path and the simulation, they all will already
[21:14] generate some kind of risk and provide actions. So, the second one is that continuously in each of the category, there will be some tasks being assigned. That's the second message you want I want to deliver. The third message I want to deliver is actually the entire process
[21:31] is actually a teamwork. Some of tasks we will assign to the executive. Some of the tasks we will assign to the CISO. Some of the tasks we will assign to the IT 19 do some installation, and some of the tasks we will assign to the SOC. So, that is the dashboard. Dashboard is
[21:48] talking about action. And the second tab here is about the data injection. Just like you you you we talk about if we just do the detection, maybe we just need a few of logs. However, if you want to do the digital twin, we actually need to involve with a
[22:03] number of different domains. So, from left to right, you can see we need to collect data from user, endpoint, email, cloud, application, code, data, network, and more. And from top to down, you can see that
[22:19] for each of category, we have a lot of data. For example, for the user, we integrate with the AD, Azure, or maybe other kind of identity servers. So, we need to know the account list, the group list, the GPO, ACO, these kind of things.
[22:35] And for the endpoint part, we need to have some sensors there. We need to know what kind of process is there, file, registry, network configurations. And also, we don't only care about what data we already have. We only care about
[22:50] what we aware of in the environment and what data is needed. However, having data it's just a database, right? It's not digital twin. For example, if you collect all the sparks, email, chat message, your writing. This are all logs, not the digital twin.
[23:07] Right? So, what we Why we need to build become a digital twin is we need to build some other layers. It start from ontology. After we have ontology, we know that hey, we know there's account here who is Jason. There is a machine here. What's
[23:24] their relation? So, the ontology the first step is we need to identify what is the vertex, what's the age, what's the meaning. However, tell the story have a lot of different representations. Maybe we didn't want to look the ontology. We want to know which account
[23:39] and which computer actually in the same building or in the same subnet. So, we actually need to transform the data in a different way. For example, I I want to know hey, this machine and this machine and this machine are they are in the subnet. So, they might able to have more
[23:55] easy to connect with each other. But, the account and machine you want to go from this subnet to the other, we need to have a firewall ACL or another different settings. So, I I know this kind of connectivity. That would be the critical part as I mentioned, the
[24:10] success condition if the rating can go from here to there. And then if we have this kind of information, then we can go further that hey, what's the attack path? For example, if this account is be compromised, just I mentioned, it just accidentally click a phishing email. Can
[24:26] it be a rating's entry point and it can do the full hole and do the lateral movement, do a number of jump after the in the end, can you reach your SQL server, can you access your data lake, or it can compromise your entire AD?
[24:42] So, that is the question we want to ask to ourselves and to provide to our customer. So, in this case, maybe I can just run one simulation. I just pick up one of the cases and I would just start.
[24:57] So, uh the rating star is just like the maybe the James Bond or the Mission Impossible that you are Ethan Hunt. You give you some contest. I can give you the first key. And where is the entry point? What is a goal? And the the first the first two AI
[25:14] agent I just mentioned, you will just do the query, collect information, build a graph, and and build a path. After the path, then you will hand over to the the rest six different agent. Uh for example, it is it start to
[25:30] reasoning about what's my first step I want to do lateral movement. And after I lateral movement from from this machine to the other machine, I want to do the credential access. But because the the credential maybe in the memory, I need to work on the tool. Let
[25:46] me dive in a little bit to let it feel what's happening. For example, the first one, this one, it is the strategy builder. So, it will choose the MITRE tactic is credential access, and you will still to choose the MITRE technique that which which things I I want to do because
[26:03] the the AI agent notice that, "Hey, there are other kind of the user looking to this machine, so your credential might be in the LSASS memory. So, I want to use the the the some kind of mechanism to read the credential in your LSASS memory." So,
[26:19] after this strategy is built, then the script resolver the generator actually will generate the script. What kind of tool you will use? What kind of parameter you use? It it will be the script generator's responsibility. And
[26:34] it will hand over the resolver that, "Hey, hey, based on the threat knowledge, what kind of the success condition there?" So, it will list the success condition. And then uh it hand over the we call the digital twin. So, digital twin will reasoning
[26:50] over all the success condition and generate the query. If the data is on the graph, it may make generate a site site for graph query syntax. If data is the in the lay base or in the tables, I will generate the sequel. So,
[27:06] based on this sequel, collect the data and we will collect the value. So, the value will hand over hand over to the judge. The judge will say, "Hey, this one is a success and maybe the other one is fail." So, you can see that several step is be composed and some of uh fail
[27:24] one will will will become the red and some of part is green. So, in the end, um I Yeah, so it it it just it just every time I do the prompt, I I use using the agent harness, talk to your agent, provide the API, provide the data and
[27:42] let AI agent reason in the process, think of what data you need. So, we we issue this kind of lay house query. What kind of things I I I might need need other things I I call the other API. So, uh in the end, in all of this very
[27:57] simple demo that, hey, in the entire process, we we say the rating already uh do a a number of the the probing, collect information and finally the goal is achieved. And when the goal is achieved, actually you it might take some minutes, but I can just uh go over
[28:15] the the other one I already uh finished earlier.
[28:31] Yeah. So, in in in in the end, you will you will generate a rating report that you have the ex executive summary, the what is a tap path, issue path, what kind of data we collect, why I say it's success or and why I say it's fail.
[28:51] So, um that is a demo part. So, let me go back to my slide. I think the entire story is because this is a Databricks Summit, you already have the data. So, actually you can build all this story over the Databricks. But, what's the difference? Maybe you
[29:07] can use other framework, right? I think if you use other kind of framework, you you you need to move your data to your agent. But, if you build over Databricks, you can actually bring your computation to the data, right? Data is the key for cyber security.
[29:25] Yeah. So, finally, everyone can build, but we want to show more lesson learned. Actually, there are some detail here. The first one is actually we claim ourselves do the digital twin, but most of the customer first chat question is, "Hey, can you simulate everything?" Actually, no.
[29:42] We didn't simulate everything. We only we talk to the customer to to know what they care. We talk to the rating that what actually they want to prove. So, actually what we prove to you is actually what the most important part is the success condition. That require
[29:58] knowledge and require the data. So, we are not pursue for perfect simulation. We just want to prove for uh perfect accurate validation. That is just Dennis mentioned about the C10. Where Gartner mentioned that, "Hey, all
[30:14] your all your uh company are invest cyber security, you you purchase a lot of EDR, firewall, a lot of money, right? But, how you prove you are safe from those rating or uh APT attack?" And uh
[30:34] the second lesson is in this case outdated data is more dangerous than missing data. Because if you missing data, you know you We have this data. However, the entire data lake actually collect data from sensors. Sensors have might have a different kind of feedback mechanism. Maybe some of
[30:49] some of data feedback timely, some of data just maybe send back data once a day. So, sometimes they are out of data. They will create some false alarm to our customers. So, we we actually suffer a lot from that. And the second one is actually a test success condition is much harder than
[31:05] building a test script, right? Right now you use a fable, maybe you use in the office 4.8. Actually, they are very powerful to generate those scripts. But success condition is hard. The third one is actually prompt is easy. We we actually didn't spend much time on
[31:20] prompt, but Agent Honesty actually spend more efforts. And the final one is where will be the differentiation comes from? I think the entire architecture what we we we demo to you actually now is really new era that you see our website,
[31:37] you can instantly to copy you you it with the cow code, right? So, what's the most part? Actually, it's the validation knowledge and the feedback loop. What is that? The validation knowledge is that just I mentioned, generate script is easy. However, how do you validate
[31:54] the vulnerabilities is really exploitable? When we talk to our customer, kind of say, I'll show you, hey, there's a tap path, your the data is danger. That's it. They and they respond to us that, why do I care? I have a firewall, right? How we validate if that vulnerability is really
[32:10] exploitable is what customer really care. And the And the the other thing is the feedback loop that, you know, hackers change their strategy daily. And that will be have have a corresponding success condition. So, we need to have a
[32:26] capability to fine-tune your sensor, what data need need need to be collected. We need to have a very a uh agile behavior that I I can tweak my sensor collect different kind of data for us. So, um
[32:44] the takeaway is you actually see what's possible, right? And you are almost finished your Data Bricks Summit. You have all the power of how to use the data, what's the tool there. And Data Bricks also provide you
[32:59] a very strong foundation. And you are the data expert of your your company. So, you are possibly just snapshot today's uh video and talk to your AI and you can just do your own AI
[33:16] power rating simulation in your own version of the security digital twin. So, I hope you can be your James Bond in your organizations. And uh
[33:32] if you need any help, we are your Q branch. Yeah. So, uh thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.