Replace your Legacy SIEM with Lakewatch: Build Your Own Custom Apps with Lakehouse & Lakebase

Summary

Databricks Lakewatch is presented as an Open Security Lakehouse that lands 100% of telemetry in open Delta Lake formats, eliminating the ingestion-based pricing and proprietary data lock-in of legacy SIEMs.
A production customer example demonstrates ingesting 13TB per day across 22 sources at up to 80% lower cost, with OCSF-normalized Spark Declarative Pipelines delivering sub-15-minute threat detection and 94% MITRE ATT&CK coverage.
This video shows how to build custom security applications on the same data using a React and FastAPI stack on Databricks Apps backed by Lakebase, enabling AI-powered threat hunting workflows that prebuilt SIEMs cannot surface.

Replace your Legacy SIEM with Lakewatch: Build Your Own Custom Apps with Lakehouse & Lakebase

Many legacy SIEMs charge on ingestion, lock data behind proprietary formats, and limit AI customization. Databricks Lakewatch changes that. As the Open Security Lakehouse, Lakewatch lands 100% of your telemetry in open formats on Delta Lake — and because the data is open, security teams can pull any signal into custom apps built natively on Databricks. this video walks through a production SecOps platform on Lakewatch, ingesting 13TB/day across 22 sources at up to 80% lower cost. OCSF-normalized SDP pipelines deliver sub-15-minute detection with 94% MITRE ATT&CK coverage. A React + FastAPI app on Databricks Apps, backed by Lakebase, shows the art of the possible — AI threat hunting with Claude and custom analyst workflows no prebuilt SIEM could surface.
Talk By: Anand Rao, Sr. Solution Architect, Databricks ; Surya Sai Turaga, Field Engineering, Databricks ;

Chapters

FAQs

What is Databricks Lakewatch and how does it differ from a traditional SIEM?

Lakewatch is described as an Open Security Lakehouse that stores 100% of security telemetry in open Delta Lake formats under the organization's own control. Unlike traditional SIEMs that charge based on ingestion volume and lock data in proprietary formats, Lakewatch allows security teams to run any workload—including custom AI applications—directly on their raw data.

What cost and performance results does a Lakewatch customer achieve in this video?

The customer example described ingests 13TB of security data per day across 22 sources at up to 80% lower cost than their previous SIEM. OCSF-normalized Spark Declarative Pipelines deliver sub-15-minute detection latency with 94% MITRE ATT&CK framework coverage.

How are custom security applications built on top of Lakewatch?

Custom applications are built using a React and FastAPI stack deployed as Databricks Apps, with Lakebase serving as the application backend database. Because the underlying telemetry is in open Delta Lake format, security analysts can surface custom threat hunting workflows and MITRE ATT&CK coverage gap detection that would not be possible in a closed SIEM product.

What role does Genie play in security analytics on Lakewatch?

Genie provides natural-language querying capabilities over security data stored in Lakewatch, allowing analysts to ask threat-hunting questions in plain English without writing SQL. The video also covers agentic detection rules and automated response capabilities as extensions of this natural-language interface.

Full transcript

[00:08] Thank you very much. Um very good morning to all of you. Gosh, it's the third day, fourth day for some of you to attend the training on Monday. So I really really appreciate um coming in. Uh so topic for today is um some of the
[00:20] guidelines, some of the uh approaches how you could replace your legacy theme uh with some of the stacks, some of the announcements that we probably have gone through yesterday um as well as day
[00:32] before on the keynote. Um so what we're going to pretty much cover um is we would not assume that you were able to attend all the announcements. So I'm going to give you a lay of land a little bit of recap um and then we'll fast
[00:46] forward to actual implementation how um the world would look like with data bricks apps and using the technology underneath which is lakebase uh and some of the lakehouse architectures and so on and so forth. So that's our agenda. Um
[00:59] for introductions u my name is Surya. I'm going to cover the first part of uh today's session. Um I'm joined I'm an on lake base. Uh and we are passionate about building apps um on on data
[01:11] bricks. Um so that's where um I I earn my right to be on the stage but I would also give introduce Anand. Why don't you >> how's it going everyone? Good morning. Have you all already uh gotten into the
[01:25] conference fatigue? you know how many sessions you attend every day and after the whole concert yesterday how many of you attended the concert yesterday was that big or what you know that just
[01:38] looking at the size of people there like the number of people there like I was talking to some of the folks saying that just tells you so many people are using databicks and datab technologies it's it's fascinating for us at databicks to
[01:51] see all of that and having all of you here so thank you for coming in on a Thursday morning uh and trying to listen to us. So as Surya said uh I'm a uh theme on the lake
[02:03] watch product and a cyber security at data bricks as well as a huge uh I hold a huge passion for building apps uh apps on all sorts of different things on data bricks and outside of databicks as well.
[02:16] So today we're going to be talking about um the lakewatch product and then the second part we are going to cover how do you build custom apps and the other facets to the same data that's being
[02:28] used by lakewatch. So let's get started. >> Awesome. Thank you An. Um so just to um kick off the talk um so here are some of your seen challenges you pretty much are
[02:40] aware of if you are in this um industry and been for there and operating in this industry for a long time. Uh the key things you would really um uh hit home with this is we all are aware that um
[02:53] there is a clear problem of unsustainable costs when you try to scale in this area. uh because the just the amount the velocity of the data the variety of the data that comes in um you
[03:06] actually take a conscious decision on what is that subset of data that you would like to deal process and see and look for insight uh and that's actually
[03:19] a clear uh escape route um for for um security loopholes right because purely because of the volume you cannot just look for a needle in a hay stack uh in
[03:32] that current setup. So it's really uns unsustainable to look uh when you have the sheer amount of data that is flowing through especially in the security space. The second thing that really prohibits um looking for uh some of
[03:46] those vulnerabilities in your data logs is the data silos and vendor lockins itself. You would like to do more but it's just the way the data is stored, the way that the data is filtered and
[03:59] you take a constant decision of not looking into 80% of the data that is available maybe some vulnerability is hiding uh in in that plain sight uh is probably the other uh element. Third,
[04:11] the moment your data is locked in the the moment the data is locked into the specific vendor's proprietary format, you're not agile. you are not able to um pretty much use any of the latest and
[04:24] greatest technology on top of that subset of the data that you uh filter to work off of. Finally, um this is very very real. In fact, the very recent news
[04:36] about the Linux thread which was there for decades, right? I see nodding heads. Uh and and why did we miss that is uh pretty much why did it surface first of all because of AI accelerated threats,
[04:48] right? That was clear in plain sight. Like believe it or not, agents are acting on the software. They are basically acting on your data. That means they can they can use the power of these machines to do more and they don't
[05:02] know agents don't know what is right what is not and hence these um accelerated AI accelerated threats are surfaced and believe it's just just a just a starting point. It's just a tipping point, right? So you're going to
[05:14] see a whole more um uh threats being surfaced in the market as you all could the so the other element is and in this space we have always been reactive right
[05:26] um and and um reactive not when I say uh in a sense that can we compete with these agents how fast they are building this software with all the volume of um
[05:40] the vulnerabilities that are hiding right so human can really not see and look at every software that everybody's writing and releasing and publishing. So we really need somebody to have another
[05:52] AI agentic system to combat AI developed software. So that's where uh we have seen a little bit of success. We will see some of these in action today. Anan is actually going to show up um the whole portal and how we built and how we
[06:04] have tackled some of those. Um and then obviously we talked at length about limited data collection, right? Um so that's really um the basis and and the areas we will cover in this talk. Um but
[06:16] here is one of the examples I was mentioning earlier AI accelerated threats right um you can see supply chain very very popular uh open AI zero day threat um and how do you combat that because um really the tentacles of these
[06:30] threats have actually gone way far before we actually realize that how do you stop it right the the clear is zero day right it's zero hour now so how do you stop all of these threats is the is the another example um so identifying is
[06:43] one aspect the other aspect is how easy it is to pull the plug off. Um so so that's the the aspect we will also see in action there. Now um a show of hands very very quickly where um there was an
[06:56] era where you set your password and you change it once in two to three years and then there was this whole scramble of like oh no you need to change your password every 60 to 90 days right that's the timeline and and why and and
[07:10] when I said we are reactive about some of these things we are implementing these solutions to combat such things because the sheer rate of vulnerabilities that are weaponized is changing. It used to take 60 to 90 days.
[07:24] I mean, let's go further back. It used to take years. Went to 60 to 90 days. No, you got to change your password every every three months. Now, quasi should have all these combinations to the era where we are going passwordless,
[07:36] right? Almost feels like an era where you had no phones to dials to mobiles, right? So, that's how fast things have changed uh in this area. Now uh I only I only mentioned up to the timeline of
[07:49] 2022 which was about 9 months. Can somebody take a guess in the room how much time it takes today um to weaponize a vulnerability? Can take a somebody take a guess
[08:12] 16 minutes. Um it's less than 0.00 5 seconds. That's how fast um these threats are being released. That's how fast. Yes. Uh it's a stat and we'll share that study as well. Um so it is faster than the blink of an eye essentially. Um so anyway, so now fast
[08:27] forward to to today. Um the agents are developing it. Somebody said minutes and hours. Uh but today it it is in the in the in the area of hours. uh but the vulnerabilities are being tried out
[08:40] every few few milliseconds sub milliseconds that's how fast they're trying where is there a loophole that I can go after uh because it's it's machine it's machine that'sing there was they were always there but they have now
[08:53] become smart they're learning from each other they're talking to each other um so they can just like skip a whole whole things it's not no longer round robin uh where they are trying and the wish luck no they are they have become much more
[09:06] smarter Um Ali also announced uh on the first day of keynote that agentic AI is already there and what does that mean? Um the AGI artificial general intelligence what does that mean? That means is it is learning it exactly knows
[09:20] where humans could make a mistake and it is also learning from other machines that what they have tried. So uh that's the area uh and that's how fast it is and exactly what just moving on to the next slide where you'll see how what is
[09:34] the scale of data that it is emitting uh what is going on and so on and so forth. Um and and basically this slide sends us a message that uh it is really not a
[09:46] security problem. It's a data problem. Right? It's actually a data problem. How would you store all of this data, all the logs and how would you act on top of it? Do you have that technology? And you
[09:58] have this clear need that a machine looks for that because human speed cannot compete with what agents are trying and they are learning from each other and so on and so forth, right? Uh
[10:10] I can name some of those open source initiatives um where you know they are out there already but uh for the for this scope of this presentation uh what we really wanted to do is um we want to
[10:22] eliminate the blind spots. Um and how would you do it is when you are able to operate on the entire data estate. Um so how and how we can do that how we can design that uh is one aspect. The other
[10:36] aspect is how fast you could see it uh in your own dashboards, your reporting systems and so on and so forth. But that is also critical. It's not just the data gets available into storage. It is also
[10:49] how fast the data can be reported where you can take an action on it or a machine can take action on it. Right? Therefore the technology leg base and you probably have heard some of the announcements with Raiden which is as fast as 0.007 seconds. So we're trying
[11:03] to get there right? We're trying to get there where an agent sends something and it is able to pick it up. The systems are fast enough to read that data and showcase to you like how many it has combed over hours, right? Um so that's
[11:16] really the second aspect. Now when I said it is actually a data problem, we're going to use AI capabilities to tackle those um data problems as well. So that's a little bit of recap on how
[11:29] an announcements that we've done over the couple of days tie together to this business problem um and how it um attaches specifically to seam right and now welcome to the world of agentic seam
[11:43] where we're trying to unify your security analytics platform your with the open lakehouse architecture and the defense mechanism is pretty much pivoting to be an AIdriven and AI first.
[11:57] Um, for the reasons we we talked about now, you really need not only an open lakehouse ecosystem or open ecosystem, but you also need something that would scale uh scale at that speed, scale at
[12:10] that velocity and variety, right? So, so that's uh really it. Um, here is actual stats from where one of our customers implemented this um into their own ecosystem. So these actually numbers are
[12:23] real from our customer who had ingested about 13 terabytes of a day u per day for about across the variety of data sources about 22 plus and we'll see a sample of that um how that they have
[12:36] designed. Now the reality is I I talked about the vendor lockin system. The vendor lockin system is not um the data itself. It is the accessibility to the variety. How you would how would you get
[12:50] insights from that data right? If that dashboard, that UI is very rigid, then you are not able to add or bring your own analytics and insights or surface them. Right? So can AI help you build those new kind of reporting mechanisms
[13:04] for you? And for AI to build these, what you need is a AI database, right? So you have heard about AI ready data, but I'm talking about AI ready databases. That is where lakebase comes into the
[13:16] picture. And with the announcement from yesterday without LTA the you would not need to ingest data in a specific like a database you would ingest that data into your regular cloud storage which is in
[13:28] your open ecosystem and the compute could be very much agile. So that basically takes back to this architecture pattern where you can see that there is you have data sources ingested into delta lake and then there
[13:41] is lakebased and why there is lakebased is because you need an AI ready database to surface and via a jar to pro to surface that information into dashboards in in the format that you would like.
[13:53] Right? So AI is actually trying to surface that in a very very quick manner. So if you are to ready after this talk go back and implement this really you can combine that delta lake and lake base as one tile uh because now
[14:06] it's all under lap but this is how the customer was able to build over the year right so uh that said um you this is why lakebase will power your app layer because you would like to surface this
[14:18] information and AI would like to surface this information it is not humans um so you would have an AI AIdriven defense mechanism in place all All right. So, um that said, um I talked about how uh
[14:31] these databases and this data needs to be surfaced. Now, let's see some things in action. Right? So, if you have a data set, if you have pabytes of data that is landed in your lakehouse, uh it is it needs to be powered by an AI to surface
[14:44] this into lakebase without writing any pipelines. Uh just automatically it surfaces that information for you. Uh and how would that what is the extent? what is how far you could go and slice and dice that information. That said, I
[14:58] will hand it over to Anand and hope that sets the stage for you um on on how what you can accomplish. >> Thank you, Surya.
[15:10] discussions around uh why we built this whole platform. uh we want to make sure customers are not making non-organic decisions around uh what data they're going to put through this rigor and what
[15:24] they're not going to put through these riggers. I'm a big fan of talking about like if you are protecting your front door and all of the things the front door and open your back door uh you know for anyone to come in that does not uh
[15:38] serve your purposes. It in fact beats the purpose of protecting your front door and all of that. So with that scale and uh agility and all of those things that Surya talked about on how we are
[15:51] getting attacked uh with all of the threats and all of the vulnerabilities that are being uh abused and misused. What we built was a platform that can take care of that. We had a full SIM and
[16:03] agenting SIM in that where we are having agents uh address those uh vulnerabilities and patch those vulnerabilities and all that. But then one of the things that he said should
[16:16] resonate well with you. We at data bricks what are we best at? We know how to manage data at scale. Like we know how to do data engineering. We know how to do data analytics. We know how to
[16:28] build agents that work with data. So we started looking at this cyber security space with a data lens. Like we looked at it as a data problem and sol we are solving this as a data problem. And that
[16:41] brings us to the next phase of our discussion today. When we say it's a data problem, when we having all of the data ingested into the platform, another key thing that we are good at at data bricks is keeping the data open. You own
[16:56] the data. It's your data. You have access to that data. It's not a black box where you ship uh your logs into a platform and then go to some screens and look at those screens for uh you know signals and other you know uh analysis
[17:11] and things like that. In fact we do provide a lot lot of those but along with that you have access to all of the data. So now you're able to look at the different facets like and say hey uh
[17:24] what is my data looking like? what is my analytics looking like and can I figure out other insights out of that data so that's where we are going to talk about today where can I build a custom app on
[17:36] it custom app on the entirety of my data that's backing lake watch and then how can I uh expose things that are important to me uh rather than using a generic uh interface that lakewatch does
[17:50] provide uh and like you know I don't know how many of you were able to see those sessions where you can interact with Genie to ask questions to debug issues that are being detected by the
[18:03] agents and all of that. It's fascinating how you can do things these days. You don't even need an interface. You can push it all aside and start talking to your uh agent to do all of the analysis and detections and all of that. But then
[18:17] what I uh started looking at was hey if I want to start pushing through uh this concept of figuring out other facets that is
[18:29] important to us. So I started looking at building an application that allows me to say I want to look at my security posture. I want to look at my uh
[18:41] incident review process. Where is it? I want to look at my threat intelligence. I want to go look at a global map and see how things are happening. Where are my threats coming from? Where is my attention needed and things like that.
[18:54] And having all of those at your fingertips is super important. And for me, a visual uh representation is uh uh it actually, you know, it has a 10 times
[19:06] positive impact than going through words and reading uh reports and all of that. So in that mindset I started building this app saying okay I'm going to start with a dashboard um where I'll say hey
[19:19] what is my if I first thing I'm doing is sipping my coffee in the morning uh and then I'm saying what where where are my attention needed like I'll go and say hey are there any alerts that need my
[19:31] attention are there any other security issues going on uh where's my sec how are my security metrics looking like and all of that and then I might even go into uh a global app and say, "Hey,
[19:43] where is just click out of this?" So, I'm looking at this uh in a picture and saying, "Where are my threats coming from?" In this example here, you're
[19:55] seeing that there's a whole bulk load of attacks happening uh or originating from the Far East. Uh and then they're targeting our data centers in the United States. So if I zoom in here and I'll
[20:08] say hey what am I what are my signals telling me and where am I getting my signals from I'm getting it from the data right I accumulated all the data I'm not choosing what data I'm going to
[20:20] bring to the system because databicks allows you to bring all your data you don't get charged for bringing additional data and then now you're curating all of that data through your medallion architecture and all of that we'll talk about that in a in a minute
[20:32] but now you've curated your data and you have the most latest data that needs your attention coming up in the screens and popping into a a view where you are
[20:44] saying okay now I know exactly what's happening I I know that there is a nuclear network that's getting attacked uh with a criticality of high you know high and then we in the last 24 hours
[20:56] there have been 342 uh attempts to break into that network and things like that so I got all of this from the data that was brought in and then analyzed by agents and then
[21:08] I've set aside a view and uh a window into that whole thing. So now once I found out that there is an attack happening I can go into uh the rest of the app here. This is a uh interesting
[21:22] way to look at uh the data that you have in the in the system. Right? So here if uh folks over here familiar with the miter attack matrix which tells you the different techniques that uh bad actors
[21:34] use to break into your platform break into your systems like I get in through a port that's open into the network you get into a Linux machine and then swap into something else. So this all of
[21:46] those are documented well by these uh what we call miter attack matrix and this screen here is showing you uh just the art of plausible here saying which of those miter attack matrix have coverage by your detection rules. So do
[22:00] you have rules to detect all of those techniques and then uh are there rules that are potentially loopholes in your systems and then things like that. So always constantly monitoring your coverage is super important. Why do I
[22:13] have some reds here? because this is an art of a possible demo where I want to show you that hey if there is a a hole or a gap it can be surfaced and then you can say uh now we're all working with
[22:25] agentic options I can go in and find one of those red techniques and say hey what is this technique uh I can click on it or you know I can I should be able to now say hey genie code go fix this gap
[22:37] create a detection rule such that I have coverage for this right so similar to that you uh you can also identify all of the gaps and then start working on all
[22:49] these gaps and say hey do these gaps matter to me like for example if I'm at Microsoft team shop and I have a slack uh technique doesn't matter to me so I'm going to say that does not apply to me
[23:01] and then move it aside so you'll be able to identify gaps uh on the all the techniques and uh you can you should be able to look at all of that and then the next thing I want to talk about is the analytics right so you have access to
[23:14] all of the data. You have access to uh the cases that came in. You have access to the vulnerabilities that are being found on a daily basis, hourly basis now. And then you can start doing threat hunting. You can do uh cybermetrics
[23:29] uh analysis and you can find out what are the uh destinations and the sources of all the attacks that are happening. You can do network analysis, risk analysis on the network uh perspective. You can do risk qualification and say
[23:42] hey are these really threats? How big are the threats? How small are the threats? Can I address them? Do I have agents that addressing them? All of these uh are possible now to be
[23:54] extracted as information so that now you will be fully aware of your security posture and be able to say am I protected? Am I having any vulnerabilities? Am I having any
[24:06] loopholes that I need to go plug? Right? So once again, this is giving you the ability to say I have access to the data. I own the data. I can build on top
[24:18] of everything that I'm getting as part of the SIM. Uh because now I can build custom apps. By the way, this is a databix app uh apps like uh basically an app running on databix apps. Um where you can deploy it. It has access to all
[24:32] of the data control data. you can govern and say decide who has access to what data only the uh security operations might have access to certain parts of the screens here and the data and you can control all of that so that's uh the
[24:45] advantage of you building custom apps on top of database but then I want to quickly take you through the architecture how did I build this right so this is uh the reference architecture for this app that I built I call it the
[24:58] cyber security uh the security lakehouse and um basically it's an app sitting on top of lakers, right? So, similar to a lot of the other applications uh on different aspects,
[25:11] right? You know, your own you know custom use cases that you have at your uh uh work. So we in this case I'm bringing in logs and uh other information from the cloud logs endpoint
[25:25] logs identity logs network logs business logs bringing all of those from those different data sources taking them through the rigors of a medallion architecture and saying I'm going to bring them as bronze layers and I'm
[25:37] going to convert them into silver and gold based on my requirement. So in this case, think about uh Lakewatch bringing all of the data. You have the bronze layer and then now that bronze layer is
[25:49] being used by lakewatch and the agentic systems in Lakewatch, but then you can always build your own silver and gold for your custom purposes, custom needs and then have them uh like in my case,
[26:02] I'm actually bringing them into lakebase. By the way, just a clarification. I want to make sure everyone knows lakewatch does not have a requirement of lakebase. So I don't want you to take home this mindset that uh
[26:14] you will have to use lake lakebase to use lakewatch. That's not in this app's perspective. I wanted this app to be more agile, quick, reactive and all of that. And that's why Surya talked about
[26:27] uh the database for the apps, the database for the agents, the database for AI. That's what leakbase is for us because it's fast and uh huh low latency. So that's why I'm moving my data automatically using the automatic
[26:40] sync uh capability within database into lakebased and then providing that to the AI agents that I've created uh the databix apps one of this uh this is a good example here uh to the databix app
[26:54] and then also being uh making that data available for ML models. I had a gentleman yesterday at the cyber security booth. He said, "Can I just use all of the data ingested by Lakewatch because I want to you build machine
[27:07] learning models, classic models on them. I want to do an anomaly detection and other things." I'm like exactly why we call it an open agent XM because you have access to the data. You own the data. You can curate whatever data you
[27:20] want out of the branch assuming you have the right access and then you can do anything to that. So that's the I wanted to make sure I paint this clear picture that this is how uh the reference architecture is going to look like. But
[27:32] then when what's the real architecture in this app? So here I'm actually bringing data from 22 different data sources like I'm bringing data source like you you probably see the usual suspects here like you know
[27:45] the cloud I'm talking about AWS cloud trail azure activity uh crowd strike identity uh sources like uh active directory enter ID uh similar ones and
[27:57] on the network side you have usual suspect you have palo alto you have uh cisco asa I'm bringing a variety of logs and var variety of uh you know data from the different data sources I'm ingesting
[28:09] them uh through the standard models that is being used by lake lake watch and then I'm curating once I'm curating that data I am bringing them into my uh what
[28:22] I call gold tables once I have them in the gold tables I'm syncing them back in lake base I'm using them right so if I look at this another screen I have here to showcase that flow of data right so
[28:34] I'm bringing the data into landing sp landing zones uh you know which are nothing but volumes UC manage volumes uh and then once I bring it into UC manage volumes I pull them into the bronze
[28:46] layer pull and you know and do all the cleaning up you know all of that for the data sets and then curate them into gold and UI tables. So just want to give you a feel for how the data is flowing for
[28:59] this app. So that all those screens that you saw is getting the right data uh at the right amount of curation and the right amount of detail. Right? So this is generally the architecture I
[29:13] want to show you. And another quick thing I wanted to talk to you about here was maybe I'll tap here. Sorry.
[29:39] >> So I have created a bunch of genie spaces on the same data sets. So you are how many of you are familiar with genie? So thank you. That's one of the best uh parts of our platform. Well, every one
[29:51] of our customers who I talk to are really enamored with Genie because of the value that they get out of it. Right? So, same way here, I brought in all the data and now I have unleashed
[30:03] Genie on all of that data set. Right? I get fresh data getting pushed in on a minute-by-minute basis, hourby- hour basis. Right? Now, I can start asking questions in English. I can say, "Hey,
[30:15] tell me the most recent threats that happened here." For example, in this case, let me see if I have some history where I can pull up example. So here I asked identify uh some
[30:28] interesting outliers in entity data like so this was data set entry uh where should I the distribution of confidence scores? So you can start asking questions saying what is happening
[30:40] today? You know where is my threat detection going? How how am I identifying threats? How am I identifying vulnerabilities and how am I going to address those like because Genie has access to the data set that's
[30:53] current and live and also has access to some of the data sets. You can attach uh other information that goes along as reference information to it. So now it can start making recommendations for you
[31:05] and say hey I did find out that someone changed the Slack configuration today that might have a global impact on your company. you want to go and re uh revert that for now and then talk about how to
[31:17] uh go identify why that person made that change for the slack and all that. So now you're you can start working with real data live data in English and then start working towards like you know what
[31:31] you want to achieve like you know let suda talked about all of those uh different four uh oh let me go to that slide in a I want to
[31:53] So this slide here right here if you look at it we are applying agents to find like detect the problems. We are applying agents uh like you were using agents to triage the problems. I can go into Genie code and you might have seen
[32:05] some examples on the main stage yesterday where they were talking about hey Genie uh tell me uh the most recent threats that are happening today most recent detections and then it'll say hey I found these four cases that were
[32:18] created thanks to this one detection rule and then the moment you find those cases that were created you can say can you triage this case for me it'll go do all the analysis for you find all the data that's relevant to that case and
[32:31] then come up and say this person did this, this is the reason why they did this apparently, blah blah blah. So, it'll give you all of those analytics, but also I want to apply the response, right? Finding what is the recommended
[32:44] response to that issue. It could be shutting down a network, isolating a network, it could be reverting a change that was done on Slack or Teams or some somewhere else. So, being able to do all of that agentically, how do you do that
[32:56] agentically? Because my agent might not have access to do all of that. it might have access to MCP servers which have tools to do that. Once again, it's all about governance at that point when you're taking responses because you want
[33:08] to make sure that not every agent is able to shut switches off or on, turn them on, things like that. You want to make sure agents are governed and that's where Unity catalog comes into picture where Unity catalog not only governs
[33:21] allows you to govern and control your uh table access or view access or you know volume access it also does that with agents and LLMs. So now you're able to say which agent is able to perform what
[33:35] which agent is able to reach a particular MCP server. Let's in in that Slack example I took uh talked about there might be a Slack MCP server which has tools that will revert a change that
[33:47] was done in a configuration if you have the right credentials. So you want to make sure those credential management and governance is done for those agents and now you should be able to add the response aspect to that flow. I I bring
[34:00] the data I apply my detections I find out the problems and then what do I do after that? Am I going to call someone and say Anan there's a problem come look at it. No that's not going to work. So how do you build those agentic
[34:12] responses? That's the whole full cycle that uh we are uh expecting the whole you know the industry to move towards and hopefully uh one thing that I show like hopefully some of the things that
[34:24] we showed today is the idea of you know the the agentic sim is taking care of a lot of these things for you but you have access to all that data. uh the enormity of data that's
[34:36] brought for you that's curated for you and then now how you can use th those data points and data sets for building your own custom facets like you know you might want to see uh different uh
[34:50] security posture view you might want to slice and dice that with different uh uh you know angles that's what your database brings for you where you're not locked into a black box it's almost like
[35:03] a glass box you can see everything going on. You can even go and access the data if you have the right credentials and build a custom app that's pretty extensive if you want. >> Awesome. >> We want to open up for questions.
[35:16] >> Yeah. So, I'll quickly summarize one getting data, right? We we looked at different sources. We know what uh standard data definition sources that you have, which is standard, right? So that means lakewatch can connect to those data sources that you have list 22
[35:28] plus um bring that because we know the data model we know how these the format the logs where it is um you could you could unify that in the data model and then have this custom app that you're
[35:41] seeing on the screen right now to surface that information in a in a consumable format. Right? So that's one aspect. The second aspect we talked about is how you can use Genie not to chat on the data but act on the data.
[35:54] That means if you have a vulnerability that is sitting on the database that we saw about was those red um highlights on the entire dashboard would able to schedule a task that hey whenever you see such a vulnerability
[36:07] act on it right you don't have to come and it doesn't need to wait for you so the agentic system within Genie would act on it and go and update that database right since this is AI agents they need to be very very fast the database needs to be spin up the
[36:19] database needs to be super responsive because you don't keep a log of what happen right um so that's the second aspect the third aspect is when gen is actually acting on it how is it acting on it it is acting on by researching
[36:33] what's going around how many people have surfaced that over the internet what were the fixes it scrapes through that through that and applies on on your platform on your uh security boundaries real time right so those are the three
[36:46] aspects so bring your data we know their formats standard data sources you need customization very well you could do it because data is in format. Second, using likewatch. Second, you use Genie systems to not only ask questions but act on it.
[37:00] Third, you would have a database where you're keeping a log of it. But the third aspect of it is being able to go out in the internet to look for these CVEes and automatically act on behalf of you. Hey, this was a CVE. This is where
[37:12] the fix was and this is what it was done. I tried it. It did not work. So, I shut off the network. Right? So, can you define u the overall flow? Can you govern the agent, how it acts, what it tries, how long it tries and so on and
[37:25] so forth. All those definitions are done um and acted upon. So that's the overall um area. And then if you want to customize, you get a new data source, you have a new ingestion pattern, how
[37:37] would you fit it in this standard data model, you don't need to worry about that because you actually don't need that data model. It's it's so agile that lakebase can accommodate a always changing set of data models. Um so
[37:49] that's where the ontology will come into the picture right the ontology will be able to identify um how this is being spread how this data model is expanding and how would you surface this um into a
[38:02] security lakehouse custom app. So that's um the overall summary of um how this entire system is designed. This is a plug-and-play uh application obviously it goes via that data model. So if you're leveraging lake base lake watch
[38:15] to too many lakes uh lake watch to ingest from the standard data sources um and then it would automatically fit in to this and you could surface this app. >> Come talk to us uh happy to share any of
[38:27] the insights and learnings like I always look at this as learnings uh for us because uh we are constantly learning from all of you on a daily basis and uh happy to share all of that.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.