A Unified Approach to Data Exfiltration Protection and Egress Monitoring to Build Secure Agents

Summary

This video presents a unified framework for data exfiltration protection that categorizes security controls across three levels — data platform, compute, and storage — to defend against external attackers, poorly configured AI agents, insider mistakes, and malicious insiders.
Serverless egress monitoring using Databricks system tables provides visibility into outbound network traffic, and a dry-run mode lets teams build egress allow-lists without disrupting production workloads.
The framework is designed to protect both traditional user workloads and the new generation of AI agents, recognizing that data exfiltration is not a new problem but one that requires stronger governance primitives rather than entirely new solutions.

A Unified Approach to Data Exfiltration Protection and Egress Monitoring to Build Secure Agents

Data exfiltration is one of the most serious security risks organizations face. It can expose sensitive information, leading to reputational damage and regulatory penalties. The challenge is that exfiltration can happen in many ways, including external attackers, poorly configured AI agents, insider mistakes, or malicious insiders, and is often hard to detect until the damage is done. In this lightning talk, we'll break down where exfiltration risks come from and what they may look like in Databricks (spoiler: it's not always a malicious actor!). We'll then introduce a framework for categorizing protection controls across three levels: data platform, compute, and storage. Using this framework, we'll discuss what risks to address today, tomorrow, and in the future with readily available controls at each level. As we wrap-up our session, we'll give you a glimpse into implementing comprehensive egress monitoring: what's here now and what's coming. Walk away with a fresh perspective on data exfiltration risks and actions you can take today to secure and monitor your most sensitive workloads!
Talk By: JD Braun, Staff Security Field Engineer, Databricks ; Robert Zhang, Product Manager, Databricks ;

Chapters

FAQs

What is the unified data exfiltration protection framework for Databricks?

The framework categorizes protection controls across three levels: the data platform (trusted access and strong authentication), compute (blocking untrusted endpoints and exfiltration firewalls), and storage (network controls and credential protection). It is designed to apply consistently to both human users and AI agents, since exfiltration risks exist at all three layers regardless of who or what initiates the data access.

How does egress monitoring work in Databricks for AI agents?

Serverless egress monitoring uses Databricks system tables to capture outbound network traffic from workloads, giving security teams visibility into where data is being sent. A dry-run mode allows teams to observe egress patterns and build allow-lists without blocking any traffic, so production workloads are not disrupted during the rollout of controls.

Why is data exfiltration from AI agents a growing concern?

AI agents can exfiltrate data not through malicious intent but through misconfiguration — such as an agent that sends data to an untrusted external endpoint without proper guardrails. Because agents act autonomously and can make network calls programmatically, the same exfiltration risks that have always existed for human users are now amplified and harder to detect in agentic workloads.

What egress monitoring capabilities are coming to Databricks?

The video describes upcoming features including allow logs that record which egress destinations have been permitted, and outbound traffic flow logs that give a granular record of network activity from Databricks workloads. These additions complement context-based ingress controls and Governance Hub Security to provide a comprehensive network observability story.

Full transcript

[00:07] All right, we'll go ahead and get started. Thanks everybody. Hello. Another summit. Welcome on this Wednesday before the Chain Smokers. We are the first act before them. So, we're going to bring all the excitement with
[00:19] unified approach to data exfiltration protection and egress monitoring >> to build secure agents. >> Chain Smokers asked if they could use that as a song title. We declined because we wanted to keep it for ourselves because this is very valuable
[00:31] IP right here. I'm JD Braun. I'm a staff security field engineer here at data bicks. I'm here with Robert. Hi, I'm Robert. I'm a product manager building lots of features in networking security and observability at data bicks.
[00:46] >> Wonderful. So, we have a quick 20-minut talk for you all today where we have several agenda items to get through. The first is we're going to be talking about builder building stronger primitives not new solutions when it comes to data
[00:58] exfiltration protection. Then we'll talk about our unified approach to data exfiltration protection with one unified framework that works both users as well as our new era of agents. And then
[01:11] Robert will continue on with egress monitoring for agents and wrap up with one of the most exciting sections about all the things to be excited about for networking and logging and exfiltration protection within the platform. So
[01:24] before we get started, show of hands. How many folks are data bricks users in their everyday work? All right, wonderful. How many folks sit within the security organization or the platform administration for their data
[01:37] bricks environment? >> All right, wonderful. a lot of paranoids in the room. That's great. So, let's go ahead and get started on building stronger primitives and not new solutions. So, something before we get
[01:50] started with all the unified framework and the controls and what we can implement for agents, it's good to take a look back and think that data exfiltration was not a new problem that was invented by the rise of agents. It
[02:02] has always existed. The metaphor that I like to use is that when Henry Ford was building the one of the first cars meant for mass production, he didn't sit there and think, uh, if only we had this long,
[02:15] hard paved piece of land that we that went straight and sometimes curved that we could put these things on. No, roads have always been around. He didn't invent the road. He fixed the new piece
[02:27] of innovation to meet what was currently there. So we need to think about in the same way with agents is that our exfiltration protections are still there. We just need to harden them and
[02:39] make them suitable for the speed and scale at which agents can now operate. So let's talk about a unified approach to data excfiltration protection on the data bricks environment. So we break it
[02:52] up into three different pieces. We start with the first aspect of all user and client access is from trusted locations and strongly authenticated. The second layer is that there's no access to
[03:05] untrusted storage locations or private or public endpoints as well. And the final layer that all data access is from trusted workloads. So simplifying this Xfiltration framework into three core
[03:18] pieces. So let's dig in a bit deeper to that. So if we pull back the covers on each of these three areas, the first one is all around that whether agents or users who
[03:30] are accessing your environment need to come from trusted networks such as your corporate VPNs pushed through your cloud networks via private link etc. and they are authenticated through standard means
[03:43] whether that's a user through SSO and MFA or that's through an agent with OOTH with machine to machine authentication. The second one of no access to untrusted storage locations or public or private
[03:55] endpoints is that you are validating and verifying what your compute can actually talk out to. So you know that is only talking to the appropriate resources. We'll get into these in a little bit more detail. The last and the final one,
[04:07] the most important one is that any access to that underlying data store is coming from verified locations. Again, a big mitigation in the case of credential leakage. So now we'll go into each one
[04:20] of these and talk about the controls that we can set up in the age of agents as well as just general best practices that we like to do for users.
[04:34] All right. So let's start with all user and client is from trusted locations and strong authenticated. So again, every client and every agent that is coming and authenticating to your platform needs to be sure they're coming from that appropriate network and the
[04:47] controls that data bricks offers to ensure that are going to be controls such as IP alles on your account console. Robert will talk about some exciting developments in that area a little bit on next is going to be
[04:59] authentic access to the workspace either via IP ales or front-end private link. I'm making a big bold statement as well. There's also the upcoming contextbased inc ingress controls. It's now going to
[05:12] become even more important that we can discern what an agent can access, what a human can access, or even what different groups of agents or different humans can access via specific network locations.
[05:26] Robert will talk about more about contextbased angular controls now, but it's a big point that I'd like you to take away here walking away from this. The next is going to be the actual authentication mechanisms. So when you're working with agents on the data
[05:39] bicks platform, specifically within things like data bricks app, ensuring that you're using those on behalf of authentication for the user because we have to think about agents in the ways of both probabilistic and deterministic
[05:52] controls. If there's a probabistic control, if you're putting in guard rails on inbound and outbound requests to the model, those can be written over. Those can be manipulated by someone. But
[06:05] if the underlying deterministic layer is that authentication token, the agent can't do it even more. They'll just hallucinate and say, "Absolutely, I can grab all your data." But when they make the request, nothing is going to happen. Now, if we want to make a quick
[06:18] note on your users, making sure you have unified login, of course, that was a big announcement a couple years back, as well as auto automatic identity management to make sure you're doing that synchronization between your IDP
[06:31] and the data bricks environment. All right, so next one, no access to untrusted storage locations, public or private endpoints. This one is becoming exceedingly important in the era of
[06:43] agents because agents will do anything in their power to help you. They will try and use thirdparty packages. They will try and use random guides they found on the internet. They want to be your best friend and to do that they
[06:55] will go to any lengths. And we've already seen this as a threat vector where agents are picking up random instructions from guides on the internet and then exfiltrating data out. So making sure that your compute that the
[07:07] agents are executing on can only talk to those approved public or private endpoints. To enforce that within the data bricks environment, you're going to make sure for your classic compute plane that you have an exfiltration firewall
[07:20] that exists within your VPC or V-Net. Or better yet, think about just cutting off all outbound access. That's my preferred pattern. I prefer just an isolated environment. Makes me sleep better at night. The next is going to be VPC
[07:32] endpoint policies and network segregation. So making sure that you're not only considering what front door they can go out of, but if you're using AWS or Azure natural components like S3
[07:44] gateway endpoints, you harden that endpoint policy as well. It's a classic missed environment of making sure you're talking to those approved endpoints. And then last is going to be our serverless egress controls for serverless compute.
[07:56] Making sure that you've hardened that. And Robert is going to talk about the steps you can do to enable that so that you don't go away from this talk turn on egress controls and break all your production workloads and instead of at chain smokers you're on an all on call
[08:09] rotation. So we'll talk about that a little bit later. All right, last but not least and one of the most important underlying controls is making sure that your storage location has the appropriate network
[08:23] controls on top of that. So if there is a credential leakage, if your agents do have access to tokens and that gets leaked somehow within their memory, gets written somewhere else that you have that last layer of defense so that those
[08:36] credentials cannot be used from outside your network. So that stops that at the final gate. So as we look at this entire picture, we've monitored ingress, we've stopped egress at the compute layer, and
[08:49] then finally we're adding that layer of defensive depth within the network policy. To do so, on data bricks, you're going to be working with your storage account firewalls, whether you're bucket policies on AWS and the equivalent on Azure and GCP, as well as the workspace
[09:03] bindings on cataloges, making sure that your development workspace can only access development, production, access production. Again, we think about how agents are going to act. If a agent running within your development workspace has access to prod, they're
[09:17] going to access prod. It's going to be what they're going to do to help the user. They're not going to take that first step and respect any of those controls you think it may. Just because it says production, it's going to prioritize the user. And then third, we
[09:29] do offer the ability for private endpoints and network connectivity configurations. If you have private data sources that you don't want to open up those firewalls to set up those private link endpoints. So that was a lot about
[09:41] the controls, a nice unified framework that will work for both your users and your agents. With that, I'll pass it off to Robert to talk about the egress monitoring of the equation for agents. >> Awesome. Thank you so much, JD. All
[09:53] right, so JD just talked all about a lot of very important controls on both ingress and egress for both classic and serverless. So what I'm going to talk about right now is really about monitoring that observability layer on top of those controls so that you know
[10:06] that those controls are actually working properly. And today I'm going to specifically focus on serverless egress monitoring because there are a lot of really cool developments coming up and actually on the ground ready now for you to try out. Um so let's get into it.
[10:22] So first of all serverless egress. You've got potentially lots and lots of serverless workloads running in data bricks. How do you make sure that those workloads are reaching destinations that you want it to reach and are blocked
[10:34] from destinations that you don't want it to reach? Right? We all had to, you know, know about the light LLM attack. We don't want data exfiltrating to random domains that you don't trust. So, how do we monitor all of that? So, today
[10:48] when you set up a network policy with serverless egress controls, we log all outbound denials by that policy in this system table system.acess.outbound outbound network. And basically what this table does is it logs all the
[11:01] requests that were denied by your policies for the for every workspace across your account. And so you can basically query by hey when did this happen? Which destinations did it reach? Did it try to reach a storage destination or an internet destination?
[11:13] Which workspace did it come from? This is the place for you to kind of learn about all of this. And this table captures egress from all of your serverless workloads, which includes, you know, data bricks apps, your agents, and of course, your normal jobs,
[11:26] notebooks, and queries. So, let's kind of walk through what that step-by-step kind of journey looks like for using these logs to not only monitor egress, but also set up your policies.
[11:38] So, first of all, you can use this thing called dry run mode. um you turn your network policy to restrict to be fully restrictive. So it denies all egress but in dry run mode. And so what that basically allows you to do is it would
[11:51] dry run basically would log all the denials but it wouldn't actually actively enforce those denials. And what's useful about that is it's really really great for building up that allow list. Right? You've got lots of production workloads or or
[12:03] non-production workloads that are trying to reach your internet or various destinations and those are super valid destinations. You want them to reach those destinations. what destinations are those workloads reaching? Dry run mode is your way to do that. You
[12:15] configure your network policy, set it to dry run, and you'll immediately start seeing all those destinations that would have been denied. Um, and and from there, you can add allow rules to your network policy to the destinations that
[12:27] you want to allow. Um, once you create that allow list and you're comfortable that hey, like I turn on uh restrictive egress, my workloads aren't going to break, you can then switch that dryon mode to enforce mode. And from there
[12:40] your egress controls are active and being enforced across the workspaces you've attached a policy to. And then once you do that you can then continuously monitor all the outbound traffic that gets denied by that policy.
[12:53] So of course you can do that to one allow list new safe destinations. You say hey like I see this destination it looks safe. I know the workload using it is safe. Let me allow this this destination add it to my policy. And of
[13:05] course, you can also monitor denials to just catch any suspicious egress attempts from potentially malicious workloads. Awesome. So that's what we have today on the serverless egress control monitoring
[13:18] side. There's a lot of other observability tools we provide on the inbound side as well. But uh feel free to read our documentation there, but there is a lot to be excited about with respect to egress control monitoring in terms of features that are available now
[13:31] and coming very soon. So I want to focus on two enhancements to egress monitoring. So the first one is actually in private preview today is basically logging allow logs in this outbound network table. So if you think
[13:44] of outbound network, it's basically like a DNS request and storage access log. Currently we log denials from your policies. Now we will also log all the allowed traffic. So you actually know what actually eress your serverless data
[13:56] plane. And so with this of course you get your time stamp your workspace whether it was allowed or denied and which destination it tried to reach as well as the resolve destination IPs uh from DNS resolution. So you know uh so
[14:09] actually that's great for joining with the next table. So another table and so actually for outbound network if you want to enroll in this allow log uh preview it is in private preview just reach out to your account team and you can try it out.
[14:22] The other thing that's coming up in uh Q3, so in about three months, is this new outbound traffic flow log table. And so this is very similar to your standard cloud provider VPC or V-Net flow logs. And it's effectively an aggregated log
[14:35] that kind of dives deep into the allowed traffic that egress to your data plane. So uh you get your source workspace, you get your destination IP, which you can join back with outbound network. You've got your port, your protocol, the number of bytes that went out and the bytes
[14:48] that came in. uh connection metadata like did it go through private or public uh um endpoints. You also get your private endpoint ID if you're using private link and also data transfer metadata like which region did this
[15:00] traffic come from, which region did it go to or did it go just out to the public internet. So that's all about logging. There are actually a lot of other uh oh actually
[15:12] uh okay so big enhancement to the CUJ now your kind of customer journey is you can now use these allow logs to do a lot of new things. So first of all with allow logs even if you don't have a
[15:24] network policy set as restrictive you can use outbound network and outbound traffic to track which destinations are actively egressing your data plane at which volume at what frequency uh using what method across which regions. Um,
[15:37] and of course you can use dry run to then build your allow list. Switch to enforce mode. And now with these allow logs, you can do things like prune unused allow rules, right? You have an allow rule, but you haven't seen this destination egress in a while. You can
[15:49] get rid of it to enforce that least privilege outbound access. And of course, you can then ingest these traffic logs into your SIM, right, into Lakewatch or into Sentinel or into Splunk or whatever SIM you would like to use.
[16:02] So that's all about observability. There are even more controls coming soon. So uh as JD mentioned on the inbound side uh we've got two big enhancements coming to contextbased ingress. The first one
[16:15] is front-end private link to account level resources. So today we offer front-end private link into your workspaces. But this new feature which is coming into beta on Azure and AWS by the end of this month uh will basically
[16:27] allow you to configure front-end private link to things like the account console uh the account level genie uh governance hub if you've heard of that. that's also in preview right now. Uh so that you can basically have complete endto-end
[16:40] front-end connectivity uh front-end access into all of your data bricks assets. Uh we're also introducing contextbased IP ales so public IP ales uh to account level resources as well.
[16:53] That's happening in beta. And also um so this basically extends the existing contextbased ingress which is IP alles for workspaces. And the very last thing that I'm super excited about is we're
[17:05] basically building a native security posture monitoring tool uh in data bricks called governance hub security. And what it effectively does is it scans about 50 to 55 security best practices
[17:18] uh across all your workspaces in your data bricks account from networking controls ingress and egress um uh AM compute hardening compliance software supply chain uh data protection etc. And
[17:31] it would basically flag whenever any of your workspaces or account level resources are misconfigured, right? So maybe you have a network policy and you forgot to set a network policy for a new workspace, right? We'll flag that in this tool. Or maybe you don't have SSO
[17:43] configured. Of course, we'll flag that in the tool. Maybe you have PAT to uh paths that are about to expire. We'll flag that here as well. It is basically a single pane of glass for you as platform admins to manage your data bricks environment to make sure that it
[17:55] is locked down to your security bar. Um and that is coming uh in Q3. So yeah um that's about it. Um I guess call to action. If you haven't already
[18:08] tried out contextbased ingress, definitely try out contextbased ingress for uh for your ingress policies. It allows you to basically configure instead of all or nothing IP alackes and front-end private link. It allows you to basically use the identity and also the
[18:22] scoped destination as conditionals within your allow and deny rules. So you can say I want you know from this network source from this set of public IP sliders for these sets of principles
[18:34] I only wanted to access say these sets of API scopes in readonly mode right that's something we can't do with our traditional all or nothing IPLES contextbased ingress lets you do that for both uh public IP and also front-end
[18:47] private links so definitely give that a try um if you're interested in getting into the private preview of allow logs reach out to your account team and they'll get that set up for you and of First, keep your eyes peeled for all the
[18:59] new and fun security products we're shipping in the coming months. Thanks,

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.