Agent Ready Data Governance: Databricks on Databricks

Summary

Databricks' Data Platform team manages over 100 workspaces and 10,000 users across all supported clouds, and within six months of scaling AI agent deployment found that traditional ACL-based governance broke at agent speed and volume.
The team built a new governance model using Lakeflow Jobs data quality gates, automatic sensitivity tagging at ingestion time, SQL-level column redaction before data egress, and just-in-time access groups with scoped credentials that do not stack.
This video covers what broke when governance was designed for humans but applied to hundreds of thousands of agents, and the specific patterns implemented to enforce classify-before-read and redact-before-egress at machine scale.

Agent Ready Data Governance: Databricks on Databricks

When your data consumers shift from 10,000 humans to 100,000 high-speed agents, “governance” can no longer mean a wiki page or a PDF policy document. Agents don’t read wikis; they need semantic, machine-readable guardrails. In this talk, the Data Platform team discusses our pivot to “agent-ready governance.” We are moving beyond simple access control lists (ACLs) to a model where Unity Catalog provides deep semantic context, usage constraints and “rules of the road” that our 100,000 agents can natively understand. Join us to see how we ensure that our digital workforce operates safely, preventing the chaos of 100,000 bots making bad decisions at machine speed.
Talk By: Amit Pahwa, Sr. Staff Software Engineer , Databricks ; Prashant Gupta, Senior Software Engineer, Databricks ;

Chapters

FAQs

Why did ACL-based governance fail for AI agents at Databricks?

Traditional access control lists were designed for human users operating at human speed, but AI agents access data at machine speed across 500+ SaaS instances, making it impossible to enforce governance through manual policy management. Within six months of scaling agent deployment, the team found that agents were bypassing policies because they do not read policy documents.

What is classify-before-read governance for AI agents?

Classify-before-read is a governance pattern where data is automatically tagged with sensitivity classifications at the time it is ingested into the platform, before any agent can access it. This means agents encounter pre-classified data with appropriate access controls already in place, rather than relying on post-hoc policy enforcement.

What are just-in-time access groups in Databricks?

Just-in-time access groups are scoped credential sets created dynamically for a specific agent task and expired after it completes, rather than granting persistent broad access. This prevents agents from accumulating permissions over time and limits the blast radius of any compromised or misbehaving agent.

How does Databricks manage governance across 100+ workspaces and all clouds?

The Data Platform team centralizes security logs from all regions and clouds into a unified analytics layer to maintain cross-workspace visibility. Combined with automated tagging, SQL-level redaction before data egress, and Lakeflow Jobs data quality gates, this creates a governance model that scales with the agent workforce without requiring linear growth in the human admin team.

Full transcript

[00:08] Uh hello everyone. Uh as she said, my name is Emit. Uh before we get started, a little bit of housekeeping. Um you're going to get a survey in the app. Uh we really appreciate your feedback. So please, please do leave some.
[00:22] And then here's a little bit of legal ease. So the legal team really wants you to know, don't believe anything I say. uh uh any scenarios or stories that we're going to discuss uh are hypothetical uh and just don't get me in
[00:35] trouble. Uh okay. So at data bricks we believe data and AI can and should be infused across every function of your business. So we strive to do that ourselves.
[00:50] To support this we maintain the single largest and most complex data bricks implementation in the world. nearly 100 workspaces, actually over a 100 workspaces,
[01:02] over 10,000 users, and spanning all supported clouds in every region. We're using nearly every feature available in the product, including some we haven't released yet that you don't
[01:15] even know about. All of this so we can make the most out of our data. So every decision can be informed, so more actions can be automated. all of this so we can continue to make
[01:28] the product better and better. We want to infuse data into everything that we do. But that can be pretty difficult. You have to put in a lot of work to get
[01:40] the most out of your data and particularly governance over the data can be pretty difficult. You're all probably struggling a little bit with it here and there, right? Data governance is
[01:53] something that we on the data platform team care very deeply about. Like you, we're customers of data bricks using the same tools you use and we're entrusted by our company and
[02:05] in some cases you all to make sure that data is safe. This is what Pashant and I are here to talk to you about specifically about the governance of data as we transitioned from 10,000 employees humans to hundreds
[02:20] of thousands of agents for six months. We give or take we let them loose. Uh and today we'll tell you what broke, how it broke, and what we did about it.
[02:34] Raise your hand if you use AI in your day-to-day. Okay, every hand. That's good. Keep it raised if you connect to your internal SAS tools using the that AI.
[02:49] Yeah, almost everyone. And keep it raised if your co-workers do the same. At Data Bricks, we have more than 500 SAS instances.
[03:02] Every single one of them is somewhere where our data lives. Somewhere where our employees need to reach and somewhere our agents can reach. Your number might be 10 or 50 or maybe a
[03:15] thousand. The number doesn't really matter. The problem does. So what problem am I talking about? With 10,000 employees spawning dozens of agents with connections across 500 SAS
[03:28] instances, our governance story started to crack and fall apart. Inconsistent auditability. Can you see what everyone's doing? What all those agents are doing.
[03:41] Impossible to reproduce metrics being dri being used to drive decisions. And poor Jack over in IT pleading with the Salesforce rep to increase our rate
[03:53] limits again. Again, that's actually him. And you're all here at this conference because you know the value of data and that data is most valuable when it's
[04:05] combined, not isolated. To get the most out of your data, you have to bring it together. That's never been more true than today and it's going to be more true tomorrow. So again, how is this a problem? How do
[04:20] these agents make bringing data together a problem? It should be making it easier. The agentic way to bring data together is to point the agent straight at the
[04:32] tool. Use an MCP or something like that, which is what we did at first and likely what many of you did too or maybe still do. And it technically works. You can get
[04:45] the data. You can you can do it. But let's say you ask a question like get product usage for the accounts that ex uh expanded last quarter. What's it going to do?
[04:58] You're now joining your product data to your Salesforce data with no query layer. The agent pulls whole tables into context instead of a push down filter or a SQL query.
[05:16] Did you know 1 million tokens is less than five megabytes of data? How big is your Salesforce data? How big is your product data? And how many times is the agent going to duplicate that data as it tries to transform it to get
[05:29] you your answer? The agent will start to burn tokens like a furnace. It's great if you're token maxing, but it's you're running a business. you don't want to token or you
[05:42] shouldn't want to token max. Or worse, if you remember Ken's presentation from the Tuesday keynote, the agent realizes it's going to be slow
[05:54] or expensive and it just decides to make something up.
[06:06] It's pretty bad. So, what did people do? They started, we saw people just start shoving more and more data into these CRM into the Salesforce, right? Copy it in. And I hope I don't have to explain to all of you why that's backwards,
[06:18] right? We've spent decades trying to get rid of data silos. And here we are doing it all over again. Whether it's in your CRM or maybe on your laptop, depending on what your agent decides to do,
[06:32] it's backwards. The answer everyone here already knows is one lakehouse. That's the vision, right? Bring all the data in. Stop pushing it out. Ingestion is not a new
[06:44] thing. But things change with AI. AI has accelerated everything. We're adding more systems to our environment faster than we've ever done before. And the users like always want
[06:57] the data yesterday, right? So we have to evolve our ingestion tooling, make it faster, move at the speed of agents. And so the solution is actually two steps.
[07:10] It's ingest then govern. Now I'll pass the mic to Pashant who will explain how we made our ingestion agent first and the governance traps we
[07:24] ran into so you don't have to.
[07:38] All right. So AI changed how we think about governance. A 100,000 agents moving 10 times faster than any team of human ever could. So the blast radius of a single mistake grows exponentially.
[07:51] The governance model that served us for years was now suddenly obsolete. So we had to evolve our injection tooling. We had to evolve our platform and build something that keeps up with
[08:04] the speed agents actually move at. First let's talk about injection. So to keep up with the speeds agents were bringing data in, we built third
[08:17] party injection or 3PI. It's an internal tool that we built and it's effectively a no code governed injection platform. You give it a SAS system and it gives
[08:30] you back an end toend injection pipeline. It's built on top of data bricks tools that you're already using like lakeflow connect or unity catalog so that every employee and now every agent can bring
[08:46] data in safely and at scale. To be honest, we didn't really build most of this. Databicks already gives you tool to do this. It already gives
[08:58] you blocks to do this. 3PI effectively is just like instructions to snap these things together. At very top you have LakeFlow connect. That's your injection managed connectors
[09:13] schema discovery error handling and so on. In middle you have Unity catalog. That's your governance where your data lives and who is allowed to access it.
[09:25] And at the base you have lakeflow jobs. That's your orchestration layer. Think about generated pipelines that are running on schedule have retries and infrastructure guarantees already baked
[09:38] in. We didn't really build a catalog. We didn't build connectors. We just snapped these three pieces together and wrote one very thin layer on top. So can you
[09:52] starting today? So keep an eye on this stack as I walk you through what we built, what broke, how we fixed it. We'll keep on snapping more and more pieces on top of it until
[10:06] by the end it's a onehole wall. In 3PI, every injection goes through a sequence of stages and every one of them is an agent command. Let's take an
[10:20] example for a request and go through the journey. It starts with an intake. The agent works out SAS system on its own. It figures out by reading API
[10:32] documentation, authentication, rate limiting and then it runs an adaptive interview to capture the rest like schema, who would be the owner, how frequently do you want this to be synced.
[10:46] From there, the agent keeps on moving requesting credentials on users's behalf. It would run schema discovery. Then it hits one place a human still stands, review. That's where legal and
[10:59] security would sign off at the gate. After that, the rest runs itself, tested, deployed, and put on a schedule. So, you have a ticket in and a governed pipeline out. And this is what makes it
[11:13] scale. I wasn't bluffing when I said it's a no code platform. Everything the agent did on last slide creates these three config artifacts.
[11:25] And these configs are soul of 3PI. First you have connector. This is your source where data comes from, how often it syncs and like always when things
[11:37] breaks, who should be page. Next you have your table file. This is your destination. What data set are you bringing in? What columns it has? What
[11:49] all data quality checks you would want it to pass? and other things like how do you want this data to be partitioned. Finally, you have aback that is your policy. Over here we have a residency
[12:02] rule written as a SQL function and the platform enforces this on every single read. Platform would take these three source destination and policy and it would
[12:14] generate everything else your job schedule checks access control. Nobody actually handwrites a pipeline. So effectively there is no way for someone
[12:26] to go in a path that is ungoverned. Over here you're looking at an engineer from my team. Same engineer you'll see in same room in three different errors.
[12:39] On the left every injection was written through hand. The build alone took about a week and the backlog as you see on the back just kept on climbing.
[12:51] So we joined hands with an LLM. A week became about 2 days and we kind of felt unstoppable. But if you look at the pilot is still rising.
[13:03] So because requests were still coming in very fast, our requesters were agent as well. So that is the trap. If you make humans faster, you just lose slower. Now
[13:15] we had a choice. NXR team or should we rethink who does the building? Meet 3PA agent. It builds in SAS injection end to end with about just 20
[13:29] minutes of human effort and we stand exactly you where you saw us at the gate with a coffee in our hand. So we solved getting data in at scale.
[13:41] At this point of time, seven engineers maintain every SAS injection at data bricks across thousands of data sets. But the journey wasn't easy though. We ran into real pitfalls and that's what I
[13:55] want to share with you next. Let's talk about governance. For years, one of the core pillars of our governance was trust.
[14:07] We trusted people. We invested in training. We wrote policy documents. We had whole teams keeping those policy documents and training up to date. And we audited people on regular basis.
[14:21] That kind of worked because humans move at well human pace. With the agents, none of this really holds. things change too fast to catch after the fact and an
[14:34] agent would happily ignore a policy if ignoring policy gets the task done because what it actually wants to do is just appease the prompt. So over last 6 months we've watched our
[14:48] governance policy break over and over again. Some agents took restricted numbers and almost pasted on those on a Slack thread. Some tried to move data across
[15:00] compliance boundaries where it was not allowed to leave and one attempted to combine data which was not supposed to be joined. Here is why this keeps on happening. An
[15:13] agent would do whatever it takes to finish the task you handed it and whatever it takes most of the time means reaching for every permission you ever had.
[15:25] agent inherited your permission. It never inherited the judgment that tells you when to stop. And even when a human does slip, they slip at human pace. It gives you time to
[15:39] identify what broke, it gives you time to walk it back. With agent speed, our ordinary human based audit logs, after the fact reviews, or enforcement mechanism we've always relied on can't
[15:53] really keep up. By the time someone catches a mistake, it's already another mistake too late. So the answer can't really be a human
[16:05] looking over agent shoulder because at this speed no one catch and watch fast enough. So we had to build rules in the platform itself so that the wrong action simply isn't possible.
[16:18] Let me walk you through where ours broke, three different failures, and we'll go through three different stories. I'll also walk you through what all we put in place so that we can make sure it
[16:30] doesn't really happen again. All of these failures come down to a single question. How do we decide what an agent is allowed to touch?
[16:44] and you can't really set right permissions on the data you don't understand. So that's where story fun starts knowing what your data is. Let's go through the first story.
[16:58] For years classification used to be hygiene, a chore that you could always get to later. You'd scan your data now and then tag what was obvious and then
[17:10] leave something for rest of the day. that worked because well the person who was looking at your data was a human. Let's say a column ended up showing a
[17:23] live secret. A person would glance at it, catch it and move on. They'll add tag that column to their ever growing list of todos. Right? We used to tag
[17:35] data as part of our regular audits since the blast radius was still contained right up until the time the thing reading your data was no longer a human.
[17:47] In the new world, you have a bot reading your data. Ask it a very normal question and it quotes the secret right back in plain text either in Google Doc or Slack
[17:59] thread or even email. It just read what landed and at agent speed the scan you'll run next week already lost the race. You can no longer
[18:11] rely on humans classifying data when needed. You have to classify data at AI speed. So here is the fix. We scan data before
[18:23] anything is allowed to read it. Every injection the moment it lands gets classified before we open up the access. To do that at scale we rely on Unity
[18:35] catalog data classification. It's a datab bricks product that uses LLM to scan every table and tag sensitive columns for us like address or location or phone numbers.
[18:48] Now jog your memory back to the aback.ml file that we saw in 3P configs. This is where it pays off. The moment a column is tagged containing classified data, aback would mask it immediately on
[19:02] every single read. So when an agent asks the same question, it would get a masked value back and won't see the secret. All right, let's recap first. Remember
[19:14] the problem. You've got an agent reading data and ends up surfacing a live secret right into a Slack thread. The reason because nobody had classified that data yet. The scan was on someone's
[19:28] list for next week but agent got there first. The solution before you open up your data to read, classify it using Unity catalog data classification and tag every field using LLM so that Aback
[19:42] can hide sensitive data before any agent can touch it. So run it again. Same agent, same routine question. This time it gets a mass value back and your secret stays
[19:54] put. That's story one. Classify before read, not next week. But knowing what your data is only tells you who is allowed to read it, not where
[20:08] it is allowed to go. And that's a very different story and a very different problem. A problem of geography. Let's see and go through second story.
[20:21] When the data is produced in same workspace where you're reading it, it's a much simpler problem to solve. But as air told you earlier at datab bricks, we have one of the most complex implementation of data bricks product.
[20:34] We run over 100 plus workspaces in over 50 different regions and across every major cloud. So the people who are producing the data and the people who are reading it are
[20:46] generally not in the same region and often not even in the same compliance boundary. Which means we are moving data across planet scale and also across patches of governance frameworks.
[21:00] For years what kept kept it safe was human. When a person moved data from place A to place B, they knew that they are crossing a border. But the moment agents started moving it,
[21:13] that guardrail went straight out the window. Let's make this concrete with audit logs. Security monitoring or seam sits at the core of every company. You land
[21:25] your security logs. So you could do anomaly and fraud detection. And the data generally lives behind a very high compliance bar. But take the same data, scrub it,
[21:38] aggregate it. Now it is generally useful to other analysts who are sitting in a different region in a lower compliance boundary. So you ask an agent to make it available
[21:50] to the analytics team and it optimizes for exactly that. It takes the data, takes a shortcut and moves that raw data straight across to the lower compliance
[22:02] boundary. A careful person would have caught that. A careful person would have known that this destination is out of bounds. The agent didn't really see it was there.
[22:16] The agent didn't break a rule because it found a path that we left open. To be honest, the job is not just to move data. It's to move it while being compliant. And the fix isn't a sharper
[22:30] warning. The fix is to make sure you leave no path to find. So we made the governed path the only path. Here is how we evolved third party injection.
[22:42] Earlier it was a platform that allowed us to bring SAS data in. Now it also moves our data between workspaces. Here is an example. We generate security logs
[22:55] in every region and every cloud and we want to centralize them into our central analytical workspace. So what 3PI does is that it redacts the data into every regional workspace based
[23:09] on customer residency or federal rules which are visible through govern tags generated by data classification library. Then it delta shares the data into a
[23:22] central workspace and unions the share from every region so that you get a clean table. So the only thing that crossed the region was the safe aggregated view. The agent can no longer
[23:34] take a shortcut because the governed path is the only path. Now the user writes one declarative config per moment and 3PI does the rest. Let me walk you through what that config
[23:46] looks like. This is an example config that moves our security audit logs from the workspace where they land across every region into a central workspace.
[23:59] First you have source the regional workspaces where our logs live today and it tells you what we are pulling audit logs and effectively audit events table.
[24:12] Then comes the transformations and this is the heart of our compliance. This is the residency shield filter that keeps the raw logs from crossing the region. After that you have you're reducting
[24:24] some of the sensitive columns like emails or IP addresses. Then you have a SQL filter and finally you do aggregation so that only scrubbed and rolled up view ever leaves the region
[24:37] not your raw logs. Finally, you declare the destination, which is the central analytical workspace, and you're done. The point is that you can read all of
[24:50] this. A reviewer doesn't have to go through hundreds of workspaces, hundreds of jobs, and a lot of share configs. They can go through all of this in under 30 seconds and sign off to make sure it
[25:02] is safe. Let's recap. Second story. Remember the problem? Your security audit logs live behind a higher compliance boundary and same data is useful to an analyst in a
[25:15] lower compliance region but only when it is scrubbed. So you ask an agent, hey can you do this for me and it pushes the raw logs straight into the final place. The solution we use govern tags provided
[25:28] by Unity catalog and classified by data classification to power 3PI and redact data before we move it using delta share.
[25:42] So run it again. Same request, same agent. This time what it lands across the boundary is only the scrubbed view, not the raw logs. A human signs off on the move and nothing restricted ever leaves the region. That's story two.
[25:56] Redact before egress. Now both of these stories were about the data tagging it, making it compliant. The last story isn't about data at all.
[26:08] It's about the reach. Let's go through the final story. Everything we've built this far, clean, classified, and contained data, that was for exactly one reason, so that an agent
[26:22] could finally read it. Well, now it can. And it reads all of it. That's where the last story is. The moment an agent can read everything, your problem is no longer a single table. The problem is
[26:35] what an agent can do when it combines them together over the EOS. You just end up with access to a lot of things. Nobody really plans it. It just happens.
[26:49] Let's take a hypothetical scenario. Let's say you're on people's team and you've been asked to build a dashboard and employ year at a glance dashboard for yearly performance reviews. So you
[27:01] pull together everything that shows how someone is performing their GitHub activity, their Jira tickets, their Google Docs. Seems reasonable, right? And separately because you have been on
[27:14] people's team for so long, you also have access to workday, the demographics data, the age, religion, ethnicity, where someone lives. two very perfectly reasonable grants on
[27:29] their own. Now give all of that to an agent. It gets every key you have all at
[27:42] profile for every employee for last year for performance reviews. And to do that it does the thing that you would never do. It reaches into workday and pulls demograph demographic
[27:56] data straight into performance reviews. The agent thought it was being thorough. It thought it was being helpful. There is no line it in its head that says when to stop. And that's the scary
[28:10] part. You are giving it your permissions, but you're not giving it your judgment. So now you have demographics data sitting in performance reviews.
[28:23] This should not happen. And now you have an email from legal with a meeting already on our calendar for Friday. You don't want to be in that room. So if instructions won't hold our agent
[28:37] back, the only thing left to do is to make the join impossible. That's what we did. We made it impossible to even hold both keys at once. This is a new piece and it
[28:51] goes G soon are back exclusive groups. The idea is simple. You don't join a group. You assume a role. And as soon as you assume a role, it replaces your
[29:04] permission for that task. Your other keys drop. They don't stack. They don't join. They don't add up. So one role can never hold two keys
[29:16] together which were not supposed to meet. And the agent now literally can't make that join because the second set of keys would never in its hand. That's the key away takeaway. Replace don't union.
[29:31] And once the join is impossible to hold, there is only one question left. Who reached for it? Today we can answer that question. If you happen to catch today's keynote,
[29:44] Casey announced Unity AI gateway which gives you agent tracing with on behalf of we log the real person's identity through every call the agent makes. So
[29:56] the audit trail shows both you have run by which is the human who asked the question and run as the exclusive group the agent stepped into and the attribution is just the start. Out of
[30:10] the box, Unity AI gateway also analyzes and governs how LLM and agents are used to access your lakehouse. It gives you contextual policies, budgets, and smart
[30:23] routing. This is enterprisegrade control plane for agentic governance.
[30:35] Okay, final recap for for you. I I promise this is the final one. Remember the problem. You're on people's team. You're generating user profiles for review. You've got broad access to both activity data as well as demographics data. You ask an agent to pull profiles
[30:48] together. And to be very very thorough, it drops age, gender, and ethnicities straight into performance reviews and then you got a email from legal. The solution is areback with exclusive
[31:02] groups which makes the bad join impossible to hold. and then on behalf of and Unity AI gateway make every move accountable. You get both prevention and
[31:14] attribution. So run it again. Same agent, same request. This time it would build your profile from activity data but it won't have demographic columns to grab.
[31:26] You have three stories and now three doors closed. And with that I hand it back to Amit to take us home.
[31:38] Thanks, Pashant.
[31:50] data silos. We know the solution. You You bring all the data into the lakehouse. And if you were paying attention at the keynote with all of it in one place, Genia
[32:04] onology ontology will be that much better. That means there's more data. More data means more context, which means more insights to
[32:17] drive better drive decisions, right? So you can get the most out of your data, which our team definitely cares about. I assume yours does too. And Pashan explained how we evolved our
[32:30] ingestion platform to be agent first, dramatically reducing our iteration loop from weeks to minutes. A massive improvement.
[32:42] And he shared three key stories and takeaways to make sure your data is ready for agentic access. Classify it inest, redact before egress, and replace
[32:55] permissions. don't accumulate them right with arbback. And we built it all on top of the same tools you all have access to today.
[33:07] Well, our back is GA soon. Sorry. And I'll leave you with one last takeaway. We started with a very manual human in
[33:19] the loop approach to government governance. It wasn't the only thing we did, but it was a a fairly major pillar. A human was doing a lot in this. Lots of governance focused around training,
[33:32] auditing, and trusting people with access. It was a trust but verify model. But that no longer works. Agents are happy to ignore policies in
[33:44] favor of completing whatever you just asked them to do. And the blast radius has been dramatically increased. simply by how fast they can move.
[33:59] You have to frontload your governance and apply enforcements and guard rails. All right. So, finally, if you would like to hear more about data bricks at
[34:12] data bicks, uh I'd encourage you to attend some of the other talks our team is doing tomorrow. We've already done five or six of them today and yesterday. I hope you were able to attend some of those. Um, and if you have any
[34:25] questions, feel free to to come up and we'll stick around for a little bit. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.