Learn How We Secure and Govern Lakewatch on the Databricks Trusted Lakehouse

Summary

Lakewatch is an application layer built on top of the Databricks Data and AI platform and automatically inherits the platform's underlying security controls, including private link, IP allowlisting, and serverless compute isolation.
Unity Catalog provides Lakewatch with centralized governance across all security data assets, supporting table-, row-, and column-level security and a role-based access model with Viewer, Creator, and Manager permission scopes.
Lakewatch's unique inherited permission model automatically propagates access controls from parent resources—such as datasources and detection rules—down to investigation cases, streamlining SOC workflows while enforcing need-to-know access.

Learn How We Secure and Govern Lakewatch on the Databricks Trusted Lakehouse

Security data is among the most sensitive in the enterprise, yet many SIEM deployments lack consistent, fine-grained governance. Learn how Lakewatch leverages Unity Catalog to provide centralized governance across your entire security ecosystem. We will dive deep into the Lakewatch RBAC model, demonstrating how to manage Principals (Users, Groups, and Service Principals) across protected resources like Datasources, Detection Rules, and Notables. You’ll learn how to implement Lakewatch’s unique inherited permission model — where cases automatically inherit context from parent resources — to streamline investigations while maintaining strict "need-to-know" access. Takeaways include: implementing table, row, and column-level security for telemetry; defining Viewer, Creator, and Manager scopes at Workspace and Resource levels; practical patterns for SOC, cloud, and data teams to collaborate safely; sharing data with partners via Delta Sharing without compromising auditability or lineage.
Talk By: Alysson Souza, Product Specialist - Security, Databricks ; Arun Pamulapati, Principal Security Engineer, Databricks ;

Chapters

FAQs

What is Lakewatch?

Lakewatch is an open security lakehouse application built on top of the Databricks Data and AI platform that ingests all security data types—including logs, PCAP, and multimedia—and provides centralized alert management, risk scoring, and cross-domain threat correlation.

How does Lakewatch use Unity Catalog for security governance?

Lakewatch leverages Unity Catalog to enforce table-, row-, and column-level security on all telemetry data, ensuring only authorized principals can access sensitive information. Unity Catalog's audit and lineage capabilities track every data access, helping security teams meet strict compliance requirements.

What is the Lakewatch RBAC model?

Lakewatch defines permission scopes at both Workspace and Resource levels, with roles including Viewer, Creator, and Manager. Principals—Users, Groups, and Service Principals—are managed across protected resources such as datasources, detection rules, and notables, with cases automatically inheriting context from parent resources.

How does Lakewatch share data with external partners without compromising security?

Lakewatch uses Delta Sharing to allow controlled data sharing with external partners and downstream systems without duplicating or moving data. This approach preserves full auditability and lineage, ensuring that security data remains governed even when shared outside the core platform.

Full transcript

[00:08] Welcome everybody. My name is Arun Pamela. I'm going to let first Allison introduce himself and then we get through the session. Hi >> everyone. I'm Alison Sza. I'm a product specialist uh focused on Lakewatch. I'm based in Australia uh in Melbourne. So
[00:21] bit jet lag still getting accustomed to the time zones but yeah happy to have you here. >> Yeah I'm super happy to have Allison with me. He is our product specialist on lakewatch. He's been working with a bunch of our customers. As we get into
[00:33] the content, he will tell you how um some of our marquee customers are using lakewatch. My name is Arun Pomarati. I'm principal security engineer. I'm part of actually security organization. Before we get into the content, we want to do a
[00:47] quick poll. Sorry if I'm giving you some task here. Uh how many of you attended the keynote this morning? Excellent. Excellent. Thank you. How many of you are already using data
[00:59] bricks? Wow. So it's pretty good. All right. This is going to make our job easy. How many of you are from security organization?
[01:15] All right. Okay. Excellent. Um so we will be talking about lakewatch uh security of lakewatch and we will be connecting some of the important aspects that are for for you to understand when we when it comes to actually securing uh
[01:29] your databicks environment for lakewatch right so this morning uh Ali mentioned as part of the um keynote about lakewatch being one of the apps on top
[01:42] of data bricks Okay. So when I asked you how many of you are actually using datab bricks um lakewatch being app on top of data bricks uh your life is going to become so easy uh as part of like you know
[01:56] architecture security controls uh and what goes into uh actually the platform as part of lakewatch how you would be um you know governing that. Um so our goal
[02:08] here is to help you uh navigate through some of the primitives that are related to security of data bricks first and then we will dive into specifics to uh lakewatch like you know uh the
[02:21] governance parts of lakewatch like you know rolebased access controls some of the integrations that we have uh with um some of the content that you need to bring in to monitor your environment and then how we are going to interact with
[02:33] your rest of the operations related to uh you know security right so um we are going to first quickly introduce what um lakewatch is uh we are not going to spend too much time on um the value of
[02:48] lakewatch right there are going to be ton of sessions tomorrow um you know in in the deep dives where they will walk you through um how our customers are using lakewatch what type of analytics
[03:00] you can do and uh what kind of outcomes you can get out of lakewatch our goal here today is to spend a lot of time on data bricks security architecture and then how you can actually uh secure
[03:14] lakewatch content on top of the that that uh infrastructure. So you'll be walking away with clear understanding of what is database architecture, what are the sec security security controls that that are there, what are the governance controls that are there and then um
[03:27] Allison is going to deep dive into different types of uh different specific things that we did for lakewatch so that you can further enhance those and implement lakewatch in a right way. Okay. So uh of course if you missed it
[03:39] I'm going to do a quick introduction to uh lakewatch itself because um you know it it it is important that you understand what is that we are securing here. So lakewatch is open security um lakehouse. So this is very different
[03:52] from uh the traditional SIM that was that you are familiar with right. So those are those tend to store the data um or the security logs inside of preparatory uh storage formats and there
[04:05] is different cost model. When it comes to um data bricks, how we approach this is um the very well-known frame u standards that we that we built for your uh data, right? So we we deliver uh data
[04:19] bricks on data links. So with that you get the um opportunity to store all of your data in a very cost-effective way on uh cloud storage, right? So you get
[04:32] to from the get-go for lake um for lakewatch you get to take advantage of that and you get to store all of that data in open um open formats right you know the familiar park and then iceberg
[04:44] and uh you know of course delta lake uh as your storage mechanism the purpose the reason we built as mentioned by Ali this morning is that there is lot of surface area for enterprises uh
[04:57] traditional uh cyber tools that you have uh are you are getting becoming uh harder to manage in the world of agentic um you know AI
[05:09] with the advent of like you know all these um LLMs there is um you know quicker the the uh threat actors are finding uh zero day in lot of um a lot
[05:21] of tools that you that you use and then they are increasingly um putting those tools to um you know work and then finding ways into the enterprises. You have seen so many uh you know so much
[05:34] different so much news on that and for for an enterprise if you don't have equal capabilities to fight um those uh you know those threats that are coming in uh you would be falling behind and
[05:48] you know you would be putting yourselves for uh up for basically um risk out there right so with lakewatch what you get to do is you get to first of all take in all of these data sources that
[06:00] are out there unlike In the past, you you you could only pick and choose what data sources you can use for um you know doing your investigations uh and threat detections. Um because you are storing
[06:12] all of this data in cheap cloud storage on you know data links. In this case in data bricks it's going to be lakehouse you would be uh storing this data as as long as you want. So you don't have to
[06:25] you know push some data into cold storage and use certain data when you uh in the hot storage. It's not like that. You can you get to keep all of the data as much as long as you want and then um you would be able to put agents to work,
[06:38] right? Um that's the value of Lakewatch here that um with you would be able to augment your um sock teams and cyber teams with uh with agents as part of by using uh lake um like watch as as a tool
[06:52] with you for as a seam. Okay. So um the way it works is that you would be able to take in as I mentioned like you know um not your high value um you know
[07:04] traditional audit logs and some of the uh PCAP data and all of that you would be able to put in uh not just your tra traditional uh DNS logs or EDR but you also will be able to take in multimedia data into your um data lake and then
[07:21] here one of the things that that's going to happen with lakewatch is that you would normalizing the data uh using agent so that you do not need to hand develop data engineering for uh you know
[07:33] taking those logs and finding some schema and figuring out how to do that agents would be able to help. We have a lot of connectors um that we pre-built into that and you are going to store this in OCSF format right we we already
[07:47] developed the um you know that that u the framework that is needed for you to quickly normalize that data into OCSF format and then once you have the data that you need you'll be able to build um
[07:59] you know you'll be able to use agents to do the detections um because we uh we we we have a lot of content that is built we have um bunch of detectors enrichment processes that are out there
[08:11] for you to use and then you will be able to do investigations um as a uh as a sock person you would be able to triage uh some of the u some of the you know some of things that are found and you
[08:23] would be able to generate alerts and then integrate with rest of the uh infrastructure um that you have okay so that's quickly a um kind of what is wake summary do you want to add anything in
[08:36] general >> I think you covered well right the the Key message here is agents need context and by leveraging this architecture where all your data is open formats on the lake as Ali mentioned in the moment
[08:49] you can leverage that intelligence right because what's missing is access to all these data that today could be sitting in different silos and not accessible so we can find uh the needles in the middle of your uh mult multitude of security
[09:03] logs. >> Exactly. All right. So with that background first what I'm going to cover is um general primitives for data bricks as a platform and the security controls that are in place and then we will dive
[09:17] into lakew specific enhancements that we did that way everything comes together. So this is crucial for uh any data and AI platform right like you know I majority of you raised your hand when I asked like you know how many of you
[09:29] using data AI uh sorry data bricks as a as your platform um but it is specifically more important for security data because your security data has a lot of uh sensitive information in them
[09:42] like in your audit logs your emails you know everything that goes into your security infrastructure for um finding malicious intent or threat actors you know in your enterprise or some kind of
[09:56] insider who is trying to excfiltrate some data for all of that um there is a lot of sensory information that's going to be there we need to make sure when it flows into databris platform we need to make sure that it is um secured similar
[10:09] or even better than most of your you know most of the data that you're putting into data bricks platform so the piece that I'm going to talk is going to applicable to any of the datab bricks deployment that you have and then
[10:21] Allison is going to come and and and overlap that with lakewatch specific ones. So if you haven't seen databris architecture this is um um this is what I'm going to kind of um go through right now. So as you know as a databris
[10:36] customer we deliver data intelligence platform for you right. So it comes with in a in a kind of you know at a very high level there are three main components um to this platform. There is
[10:50] um storage on on the far you know based on how you see it but you know you can see that there is a storage um uh silo and there is compute and then there is management right that is control plane that we have. So the storage for for uh
[11:06] for most purposes like you know is going to be your cloud storage as you can see here um on the on the um on the blue lines is all your infrastructure right
[11:19] so if you're using datab bricks data um uh lakehouse right when you're using DBSQL for example your data data sets are actually in your um S3 bucket or
[11:31] ADLS or GCS right So we keep your data you know what you bring in uh as part of your analytics in your cloud storage and then um we have uh yield catalog that is
[11:44] going to be uh negotiating that data set between your compute. So whenever you run a business intelligence question on your data set you are going to go through this compute which is in the middle. The compute comes in mainly two
[11:57] parts. One is classic compute which is your infrastructure. So classic compute runs in your um in your account in your um you know cloud account and then serverless compute which is basically
[12:09] provided by us um you know which is u you know majority majority of the time uh if you're running a you know a quick query or something like you want to produce uh something uh a dashboard um
[12:23] that you that you would be looking at every once in a morning you would be using basically serverless compute uh if you're doing like you know a batch processor on uh you know 10 hours batch process that you need to do some ETL
[12:35] you'll be using classic compute uh and then you know it's a choice you don't have to use both of them uh you know we have customers uh both on classic compute as well as on serverless compute uh but
[12:48] the control plane is what you basically use to kind of manage all of these right so from the control plane perspective uh it's like dashboard for you in in your car basically it gives you the steering wheel which is basically notebooks and
[13:00] some some knobs like you know creating compute itself uh that you want to spin up and then you can basically um define various things that you want for example uh you can write a an ETL job inside of
[13:14] your notebook and then click run and then it would pick one of the computes that you actually specified at the time of uh you know running and then it would be going through um you know in case of AWS you would you know if it is classic
[13:27] compute you would have preconfigured um you So your IM um cross account IM role. So we'll be using that to spin up compute in in your environment and then you'll be using unity catalog to again
[13:39] look at uh your storage that is basically you would have created at the time of creating the catalog you would be having uh this again IM cross account IM role which you know which will be providing us temporary token to uh work
[13:51] with your data right so that's the flow in in general as things go between from the end user perspective uh it can be you know a dashboard that you build or it can be um some end user who is
[14:04] sitting in front of a uh notebook running trying to run uh code and run run things on there and or it can be basically uh some kind of APIs and ID tools that you would be using. Okay. So
[14:17] with with this understanding now let's look into different aspects of security that goes into this overall um you know uh platform right. So between all of these um you know
[14:30] different components of your platform there is network security controls right. So there is private link between um you know from from users perspective you can establish uh a parimeter control
[14:43] right you would be um you know it can be your VPN um you can define what the IP allow list or context based ingress is so that you know the users who are coming into your databicks uh platform
[14:55] are coming from known network and then you can uh obviously specify um you know things like a private link to go into your um uh for for your accessing accessing into your um your databicks uh
[15:09] workspaces, right? So then we have um once once you define where your users are coming, then you want to define how they are coming or what kind of um authentication they are going to do. Um
[15:23] the the uh default and the recommended way is basically single sign on or multifactor authentication and then you can actually bring in your users and groups into the platform and you will be
[15:35] basically uh provisioning those groups into the platform to get role-based access control right the role based access controls are defined by um your unity uh catalog or um unified governance through unity catalog and
[15:48] there for each one of these um data sets that you have that you are basically in data bricks you you are able to define who get to see what based on um you know
[16:01] based on you know schemas and tables and then how those are associated to a user or group right so we have um things like um column masking row filters um you
[16:13] know views and all of that that you would expect why is that important in the security world again um there is lot of sensitive data not every uh persona that comes into uh lakewatch uh need to
[16:26] see everything um it's it needs to have that role-based access control to the data set that you are bringing the security data set you are bringing right so the same primitives that we have for your traditional data applies to your
[16:38] security data right so once you have that um access to this particular um row or table whatever when the user specifically runs the a query it is going to spin up a compute that compute
[16:52] in shared responsibility model is owned by data bricks. So we harden that compute whether it is running in your classic deployment or serverless deployment the patching that goes into it um you know the things that are
[17:04] needed to um you know necessary for uh any type of um vulnerabilities that is basically going to be managed by data bricks even in the classic compute. So we harden it, we deliver it for you, you
[17:17] treat it as an appliance and then it is running in your compute. You are not responsible for uh basically doing any of the necessary any of the patching. It is basically uh we call immutable uh delivery of the compute. So when you um
[17:31] because everything is ephemeral when the compute comes on it comes with the patch latest image uh that we deliver as a runtime and then when it goes off it goes off. We don't need it's not a um you know uh it's not like old way of
[17:44] where you are running on a on a easy to install something and you are respon it's not like you are responsible for updating and upgrading anything on the uh data security side um you would expect it to be always encrypted all the
[17:57] all the data at rest transit is always encrypted but um one of the important things for most customers especially security sense to regulated customers is we offer customer managed key for storing the data uh with your keys all
[18:10] the time, right? You will be able to accomplish that um with your with your own key, right? In the serverless security if you're using which is going to lakeatches on serverless compute that we provide in serverless security. Uh
[18:23] obviously it is further more of data bricks in the shared responsibility model. The security of serverless comput is responsibility of datab bricks and then we basically provide you um uh
[18:35] necessary controls like egress controls that you that you get to deploy um uh you know as far as the uh networking is concerned on top of the serverless that we provide for you right and then of
[18:47] course the APIs uh programmatic access all the controls that I talked about network security controls they apply to your API services also um and we have um the control plane which is basically the
[18:59] central management piece for you where you get to basically do all these configurations that I mentioned earlier right you know if you want to configure um you know uh your compute like you know what size of compute you want and
[19:12] you know if you want to tag some unity catalog data sets that's where that's what you would be using right all right so um unity catalog is the central piece of all of these right as I mentioned
[19:25] earlier um as you deploy deploy data bricks you are going to configure um catalog to work with your storage uh through you know your IM cross account IM roles and then we would be using both
[19:39] for lakewatch as well as for your traditional database deployment so we would be using that to work with your data right but in addition to getting uh access to the data the security aspects
[19:51] of lakewatch are hinged on unity catalog right you would be able to set up your role-based access controls um you know all the all types of masking that you need on top of that and um as Ali
[20:03] mentioned this morning Unity catalog is not just for your traditional um you know tables but you would be able to work with your notebooks you would be able to secure those you'll be work able to work with some all kinds of files
[20:15] content when it comes to lakewatch in the security world uh and all all the stuff that goes around agents models they're all governed through your existing unity catalog controls that you have And the cost controls that go into
[20:29] both for the LLMs as well as traditional usage that you are consuming in part of as part of datab bricks as DBUS are also tracked inside of UD catalog. Your audit logs for your entire databicks platform
[20:41] are inside of UD catalog in a table in a schema called um system schema and um it is available um as audit table. Okay. So
[20:53] um you know you would I'm from databick security you would expect these things from us um you know as a as an enterpriseg grade platform in the shared responsibility model this is all our responsibility system hardening vulnerability scannings um and of course
[21:06] the encryption that goes into the uh that goes into your platform. Uh there's of course there's certain responsibility for the customers. Uh obviously for example when we were talking about these controls network security controls like
[21:19] kind of defining your user base where they should be coming from configuring your single sign on multifactor authentication egress controls those would be responsibility of customer because you we don't know your network
[21:31] and you you you you know your business and you know where uh your data sources are and you would be configuring those things. Okay. All right. So when it comes to compliance and standards uh again this
[21:43] is very important for um you know majority of you in in any enterprise but if you are especially in the world of um regulated industry you would also look for certain uh compliances. So these
[21:55] compliance standards that you see on the screen are also applicable to lakewatch from day one. Right? So that's important aspect that's why I'm going through this here in this talk is that of course on the on on the datab brick certification
[22:08] side you see our opera operational compliance you see our security compliance information systems compliance uh those are all our um your ISO standards and and you can see our
[22:20] compl with our compliance security profile you can actually accomplish uh your regulatory requirements u for for your um data sets. Okay, so with that I hand off to Allison to walk you through
[22:34] how Lakewatch takes advantage of the architecture that I just spoke about which are like an across the board available for all of your database workloads. As I mentioned Lakewatch is an app right built on trusted lakehouse
[22:47] um architecture but there are a few nuances that um Allison is very familiar with. He's going to talk you through that. >> Thanks Aron. So we're going to focus now a little bit more on the specific kind of components and resources that are
[23:00] applicable to lakewatch and then I'll really tie them into the architecture so you know where do they fit into that overall kind of security architecture of datab bricks. You may notice that this slide is very similar from the one in
[23:12] the beginning and and that's the whole point here if you have to take anything from my talk is that um lakewatch is built into the trusted lakehouse architecture that you're already using. It's not a separate product living alongside it. is a component within uh
[23:25] data brick data bricks workspace that you can use specifically for enabling cyber security kind of uh use cases. So uh and this is really important because particularly when I started working with some of the customers and you may have
[23:38] heard of one a few of our design partners like national Australia bank uh and it simplified the process a lot because the security teams were already familiar with the data bricks architecture and the security architecture that Arroom covered. So
[23:51] meaning that they already have blue blueprints internally. They already know how to harden and deploy this. So having all those same controls and mechanisms mechanisms in place simplifies the process of kind of establishing new
[24:04] environments for Lakewatch. And it's in my experience very common for security teams to want their environments to be completely separate. So they are not going to put their data in the same workspaces that typ typically the enterprise data is or in the same
[24:16] catalog. sometimes wants their own accounts completely separate from where the enterprise data lives. So these patterns and these blueprints that they already created for the enterprise data can be reused and uh deploy data bricks
[24:29] workspace in a compliant manner. So uh starting from from the right I'll talk about some of the key components of lake watch. So the first one probably the first thing you do once you get access and start using the environment is a
[24:42] data source. The name says it all. This maps a specific security log that you want to ingest into the platform. So typical examples here would be octa or entra ID logs. Uh the cloud providers
[24:54] are very common for cloud trail logs or edrs like crowd strike. So once you configure a data source that data source is attached to a preset. So a preset think about it as a recipe. It will take
[25:08] your data through the medallion uh for you without you needing to do any kind of data engineering work. So if we have a preset created for you of one of the data sources we support then that will do the job for you. Now of course we
[25:20] won't have all the possible log sources that exist out there uh as a created for you. So it also allows you to customize and use tools like Genie to kind of create those transformations. But essentially it will take it through the
[25:32] medallion and in the security side what we are doing is we have aligned with a a known framework called OCSF is an open source framework that kind of normalizes all the data into a known set of kind of schemas and tables and columns. Uh, and
[25:46] essentially when when you apply a preset to a log source, let's say for bringing cloud trail log, you're going to have a medallion with bronze being like the JSON just like AWS puts into S3 bucket.
[25:58] Then you're going to have a silver which is kind of that in a nice table that you can actually access and start uh understanding the cloud trail data you have in and then it will take to the gold which is complying to this OCSF framework where you can query a whole
[26:11] bunch of different log sources using a common set of of tables. So once you have your uh data source, that's the first thing you probably need to do. Now you have the the fun part where you actually going to start using
[26:23] that data to find uh things in your in your security logs that you want to be alerted on or want to follow up on. Uh typically you will build what we call detections and that's the second component of it. Think about a detection
[26:36] as a piece of logic and this again if you're familiar with data bricks you would not be using anything that you would not be familiar with. So we'll be using data brick SQL or notebooks to execute logic against a data source that
[26:49] you created. And on the security side, this data source is what will create kind of the the signals that the security teams need to investigate and look into. And again, this is what makes
[27:01] the the use of agents effective, right? Because each one of these sources, the more you have, it's going to generate more of these signals. And the agents are really good at reasoning around them and finding, hey, this was flagged as a
[27:13] low, but if I cross related with this other data source, that would increase this severity to a medium or a high. So creating detections is the second part. Again, the primitives are the same. It runs on the serverless compute. You
[27:25] execute using data brick SQL. Uh you can write notebooks. uh and basically that executes as a a workflow or a lakeflow uh job for you. Uh again tied to to Unity catalog. So uh
[27:39] the same way you would configure any data source access you would create connections to your storage provide credentials and then lakewatch can analyze and process that data. Uh now once a detection triggers within
[27:52] lakewatch we call that a notable. So a notable is an alert that someone needs to look into or an agent needs to investigate and triage for you. But an a notable will will be displayed in the late coach UI and we'll see a little bit
[28:04] more about how you can actually connect and export notables into an external case manage case management system as well which is pretty common. And the last part that we are going to be uh looking at is uh observables.
[28:18] So in in the security space, it's really good for you to understand the entity associated with an alert. So it could be that a single IP address, for example,
[28:31] is triggering multiple alerts. And this ties back to what I mentioned before, right? So you can risk score it and say, okay, this is atypical. This user is triggering now 10 alerts maybe from
[28:43] different severities. So I should definitely look into this. So we refer to that as observables. You see observables are entities extracted from detections or notables.
[28:59] Now uh from the authentication perspective, how do you actually authenticate and access lakewatch watch? Right? Since we're talking about like hardening and securing uh the lakewatch deployment in this talk, uh we're looking at it from two lenses. The one in the left, how do you authenticate to
[29:11] Lakewatch? It's very straightforward. If you already have configured which I believe most people would have done it uh configured SSO to your identity provider. So you could be using Entra uh you can use like Google any of the
[29:24] things that you have configured already on the workspace. Lakewatch inherits that same kind of configurations for you. So you can use the same SSO mechanism from the workspace to access lakewatch itself for user provisioning
[29:36] as well. uh very typical for us to hear that customers are using uh automatic identity management to provision users inside a particular workspace directly from uh the identity provider or leveraging something like schem to push
[29:49] those identities and keep those users and groups in sync. Um for machine to machine of course ooth off is the typical sim similar thing same data brick service principles that
[30:01] you're using you can give them access we're talk we're going to talk a little bit about arbback in the next slide but you can give it access just to perform the actions it needs inside uh lakewatch and on the right again I won't uh repeat
[30:14] myself here but data source authentication for both API integration or to connect to your storage and and read that data in or write the transformations back uh it's using unic catalog. So if you're familiar with how
[30:26] you would configure that, nothing changes. It's exactly the same mechanism. You create an external location, you create a credential, and the two allow the cluster to go and and retrieve that data and and perform the the processing for you.
[30:40] Now, uh we did introduce a few kind of roles that allows you to manage the access for these resources that we discussed. Uh when you install Lakewatch, we're actually going to be creating those automatically for you.
[30:52] And we're going to put a default uh um kind of role there that you can use. But of course, it's customizable. And it's customizable both at a workspace level. So you can have for example viewer
[31:04] groups that uh can do everything inside watch can view all resource. But it's also possible for you to configure it at a a particular resource level. Right? So you can have for example a more specific
[31:16] group. Let's say I have a group A that can view data sources. they this is the team that is going to be building the the data source ingestion. They don't need to be looking at notables for example or things like that. Uh
[31:29] similarly you can grant creator access. These are the users that are going to be creating those assets for you. And of course manager which takes everything else plus the ability of delegating and managing access from other users.
[31:41] Similarly to what you'll see in uni catalog and any kind of resource inside the workspace, those permissions, those roles can be applied directly to a user or a service principle. But more often than not and as a good recommendation is
[31:54] you would apply them to groups to make it easier to manage and and assign those permissions without needing to do individually.
[32:06] Now uh this is one of the parts that I get uh asked the most is okay so how do I bring these log sources into data bricks right we talked a little bit about the preset which is okay now I have my data in how do I transform and normalize it but there is a stage before
[32:19] that which is how do I get the data in in the first place there is a fourth which is if you already have the data in a table in data bricks you can just connect to it it's not like I said it's not separate it's in your catalog if you can access it you can ingest into the
[32:33] environment. But um and this is common as well for enrichments, right? Sometimes uh cyber security teams want to enrich cyber security data with enterprise data. Very common in
[32:45] financial services because there is correlation between for example financial crimes and cyber security right so something that you detect in the cyber security log could actually prevent some financial fraud from from
[32:58] being executed in the past. So there is a big movement there of enriching uh cyber security data with data that is kind of more on the enterprise or other areas of the business. But let's focus on these three mechanisms here. Uh the
[33:10] first one is getting directly from a cloud source. This is common uh some vendors like it's very common for cloud provider. So cloud trail or entry id logs can always export directly to
[33:22] storage and then you can consume it from there. uh some SAS providers offer this option but uh sometimes they require an add-on but essentially it will require it will depend a lot on the data source
[33:34] right so if the data source can export the data directly to storage that's great you can lend it there and then it becomes just an a simple pipeline with with data bricks similar to what you'll be doing to a lot of the sources today
[33:46] again the authentication is going to be using all the UC credentials and connection to cloud storage uh most of the times is private through the cloud cloud provider uh network. The only reason I say most of the times is
[33:58] because we do have scenarios where customers are connecting cross cloud to bring data that's landing for example you have a GCP bucket and your data is landing there and you want to ingest that in data bricks of course you need
[34:10] to manage that connectivity if you want to make it private it's possible but uh most of the times if you're in a single cloud provider all of that's managed transparently for you the second one is API pool and again uh
[34:22] if you're using lakeflow connect you'll be very familiar with this essentially you manage the authentication from uh the your compute to the external endpoint and you provide uh the the unit
[34:35] catalog connection with the credentials that is needed to access this. So uh examples here would be Slack is uh uses that octa uses that uh crowdstrike event stream exposes their audit logs like
[34:48] that. So very common pattern for you to ingest uh audit related data from external SAS systems that do not have the ability of uh exporting that data to storage. So these connectors that you
[35:00] see there, they're built for lake watch to ingest audit logs. So those are there today and they're going to be continue to grow as we have more and more uh customers requirement for for new sources. And the last one uh you heard as well I
[35:14] guess on the co on the um Ali's talk today we zero bus is now G. If you're not familiar to zero with zero bus it allows you to connect from an end from a
[35:26] an endpoint directly to data bricks and ingest that data without needing a bus or cfka for example in the middle. So you can push data directly from an endpoint. As you can imagine this would be a very common way for you to ingest
[35:38] things like sys logs. So I have a few servers and I want to bring sys logs in. Uh bringing a little bit on the application side a lot of the application logs can be exported in open telemetry. So if you're using open
[35:50] telemetry you can actually push push open telemetry data directly to zerobus and that data then becomes data that you can access from um uh from data bricks. Again, normal ones here will be SIS logs
[36:03] or or specific events inside the operating system of endpoints of laptops that you want to collect uh and make it available. Again, because the zero buzz endpoint is exposed through the workspace, any
[36:17] control that you apply at the workspace is inherited here. So, if your workspace is behind private link as mentioned, uh that's how the endpoints are going to send the data over your private network. If you have whitelisted or use
[36:29] contextbased uh access control all of that is enforced uh at that layer as well and one thing that I mentioned is common and this um on the projects that I'm
[36:43] working on I see that uh recurring is cyber security teams uh today have case manage case management tools could be service now Jira sometimes they have source which can uh provide like an automated response u and it's common for
[36:57] them to start using data bricks for a specific use case to to expand their capabilities maybe a new source that they are not investigating today because it's just too large too verbose can't be ingested on the scene uh but they would
[37:10] like to have that central kind of case management continued because that would impact a lot on how their team operates and find the tickets and gets assigned and tracked. Uh so lakewatch allows you
[37:22] to configure it provides you a UI to assess that but it also allows you to configure and export those uh those cases to an external system for example a Jiraa or we have customers using
[37:34] service now for example as well. Um again the controls you have created um for your workspace can be used here and in particularly one thing that is important is a lot of the times these
[37:46] case management systems could be running in an on-remise infrastructure or a private server in your network. So it is possible for you to connect to that uh over private link. All the secrets for authentication can be saved into data
[37:58] brick secrets which again are encrypted with uh customer managed key. So you can harden it this way and have a very secure connection to an external system be it private or be a kind of a a SAS type of provider.
[38:15] And now I think you want to >> yeah summarize >> to summarize um what you would be taking away from this talk is that lakewatch is built on your existing datab bricks primitives. Likewatch is the first datab
[38:28] bricks opinionated solution for cyber security. You saw this morning that we are delivering solutions as uh as apps on top of datab bricks which is lakewatch is the first one. You also
[38:40] heard about the customer lake which is basically CDP. So all those controls that you have already deployed in your databicks workspace when you enable lakewatch they apply automatically and then some of the things that you need to
[38:52] configure are some of the things that uh you need to do especially for lakewatch Allison walked you uh walked you through all of that. So uh generally you know um you get the governance through unity catalog and then encryption capabilities
[39:05] that you that you have uh out of the box with datab bricks you would be able to either use customer manager key I'll leave it uh as is which is basically you're going to get encryption out of the box from datab bricks and you are going to get audit logs that are
[39:17] necessary for um your security uh uh system right and then uh for compliance workloads you have complaint security profile you'll be able to use that same things apply And then for network
[39:29] security we uh we went through bunch of things for serverless there is multi multiple layers of isolation um for like you know when you talk about um this is a multi-tenant environment so you would
[39:41] be able to uh configure um uh the egress controls for your own workloads uh as as part of your setup and then all the controls that you have uh as far as um parimeter controls are concerned they're
[39:54] available through your traditional workspace configurations. Okay. So with that there is a lakewatch booth available um where you when you go to exhibition halls um there are like you
[40:08] know experts who've been doing this for last two years available to give you a demo if you are interested in looking at lakewatch. Uh there are going to be multiple talks on this topic um throughout this week. So if you want to see demo actually how to how lakewatch
[40:21] does actual security analytics if you are into that u feel free to go to those sessions. With that I conclude. If you have any questions, we'll be here uh for a bit. So, happy to answer those questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.