AI Agent Governance: Identity, Authorization, Observability, and Integrity

Summary

David Nassi from Databricks walks through four governance categories for AI agents — attribution, authorization, observability, and integrity — using a concrete example of a developer who built a Claude-powered CI agent that automates pull request workflows on behalf of engineers.
Identity-based attribution preserves user context as agents operate on behalf of engineers, while context-and-intent-based authorization applies least-privilege principles in non-deterministic systems where exact execution paths cannot be predetermined.
Databricks Unity AI Gateway governs LLM and MCP usage with secure credential storage and per-user OAuth, while MLflow distributed tracing and LLM judges provide end-to-end observability and safety validation.

AI Agent Governance: Identity, Authorization, Observability, and Integrity

Governing AI agents at scale requires more than good models. It requires systems that preserve user identity, enforce fine-grained permissions, maintain observability, and validate integrity. David Nassi from Databricks walks through four governance layers using a real-world example: how a developer named Alex built a Claude-powered CI agent that runs on behalf of engineers with access to GitHub, Jira, CI logs, and documentation.
Learn how to apply identity-based attribution, context-and intent-based authorization, end-to-end distributed tracing with MLflow, and LLM judges for safety checks. Databricks Unity AI Gateway helps govern LLMs and MCPs with secure credential storage and per-user OAuth, enabling agents to run safely on behalf of users while preserving audit trails for compliance and debugging.
🤝

Chapters

FAQs

What are the four governance categories for AI agents?

The four categories are attribution (identifying which user an agent is acting on behalf of), authorization (applying least-privilege permissions in non-deterministic systems), observability (end-to-end distributed tracing to understand what agents did and why), and integrity (using LLM judges to validate safety and correctness). Together these provide a framework for governing agents at enterprise scale.

How does identity preservation work in agent governance?

When an agent runs in a shared service account, it loses the identity of the individual user who triggered it, making it impossible to audit who is actually responsible for each action. Databricks Unity AI Gateway supports per-user OAuth so agents carry the identity of the triggering user, enabling accurate attribution across all agent-driven operations.

How does MLflow support AI agent observability?

MLflow distributed tracing captures end-to-end traces across agent actions, tool calls, and model invocations, creating an audit trail for debugging and compliance. This allows teams to reconstruct exactly what an agent did, in what order, and why, even across complex multi-step workflows.

What is context-and-intent-based authorization for AI agents?

Context-and-intent-based authorization applies the principle of least privilege dynamically based on what an agent is trying to do rather than granting broad static permissions upfront. Since agent execution is non-deterministic, this approach allows systems to grant or restrict tool access based on the intent inferred from context at runtime.

Full transcript

[00:03] Hey everyone, and my name is David Nassi. I'm a director in our AI product group. Prior to joining Databricks, I worked for 10 years at AWS also overseeing some of the serverless app platforms there with AWS Lambda. Every organization is trying to build agents,
[00:19] automate away a lot of the tedious work, and and and just increase productivity with AI. And so we're seeing customers do incredibly creative things, but they're running into a wall where you know, how can you then govern these things at scale?
[00:36] And today what I want to do with you is to just take an illustrative example of some of the challenges organizations are facing with governance and some kind of emerging approaches in the industry and how we at Databricks and what we see
[00:51] customers do that we're seeing as being kind of successful over there. And we'll talk about Alex. So Alex is a developer in this you know, mid-size company. They have about 80 developers. They adopt cloud code. At first they do it interactively and see like huge yields
[01:09] in productivity. You know, the team is really excited. So then they're like, "Okay, let's let's try to give it more more autonomy and start to automate like entire processes, right?" So they build a CI system. It's an entire like back end
[01:24] which you know, provides cloud code a harness. It has you know, sessions and state and execution environment. So that's kind of the brain and then it has the hands which are the tools with GitHub access and Jira and CI logs
[01:40] and docs. And it runs in a service account. And so the flow is like the engineer can in their slack and just do agent, you know, fix this on a PR and then cloud like kind of picks it up and does all the work. And it's pretty amazing. So the team sees like a huge optic in
[01:58] productivity and there are many more PRs being merged. So, it's all awesome, right? Well, it is, but it's it then the team starts running out running into all all these kind of governance issues that it's scale already hard to figure
[02:13] out. And so, I'll lay it out in these like four main kind of challenge categories. One is just attribution. These agents, they operate on users' behalf on humans' behalf. And so, we need to capture that and attribute it to specific users
[02:28] so we can figure out what was the agent doing and on the behalf of whom it was doing it. Then there is the challenge of authorization. So, in these non-deterministic systems, how do you effectively apply the principle of of least privilege, right? When an agent
[02:45] can decide to have, you know, very great many of permutations of like call sequences and what exactly it's trying to access. And then when things do go wrong or you're trying to improve the system, how do you have the observability to understand what drove the agent's behavior? And finally, can
[03:02] you attest to the integrity of the requests and responses to their safety and and their compliance with your organization, right? So, let's just kind of walk through some techniques and some approaches that can help kind of rein in on on the
[03:19] um on these challenges. So, at the most basic level, identities really e to understanding what the agent is doing and on behalf of of whom it's doing it. So, even in autonomous systems, what we often see is
[03:36] that the autonomous system operates on requests submitted by users. And what we see is that almost in all cases just giving an agent like this, you know, a service principal with this like swath of like broad permissions just doesn't work in that mode
[03:52] Uh, because you lose that attribution as the agent operates. The most basic basic thing here is to make sure that the agent agent's identity carries two elements. One is the actual agent identity and also on behalf of whom it operates. Then you can close
[04:08] the loop on who triggered the action, right? So, just to give you one example, we recently launched our Unity I Gateway product, which allows you to govern, um, both LLMs and MCPs. But, in this example, it just shows you how as an administrator, you can set up, um,
[04:25] your MCP to be governed. And when you do so, you provide your credentials, uh, to your system like GitHub. And then we will securely manage the the credentials and like manage all the OAuth infrastructure, including like the token refresh per user and so forth. And
[04:41] so, then when you go and you build an agent, uh, with, uh, with that MCP, it's very easy to have it just run, uh, on behalf of the user. And that user identity is preserved. So, Alex implements it and it gets her way, uh, farther than just running with this
[04:58] broad service principle. But, then identity separate to authorization. So, I think the main challenge in agentic systems is authorization. The traditional authorization systems are they've been designed for these very, um,
[05:13] well-defined control flows. But, an agent can decide to do many things. And so, how do we, uh, you know, scope down its permissions? So, one emerging approach that we're seeing is to actually apply authorization based on context and
[05:29] intent. So, this is kind of like an emergent, uh, design pattern that we're seeing. And we're thinking deeply about how to actually, you know, build it for customers so you can have safer systems. So, let's just take a very simple, like very simple example,
[05:45] uh, of this context-based intense intend-based authorization. So, here the the context is really which tools that the agent tries to access have potential harmful destructive behavior or can mutate things and which ones are kind of
[06:02] read-only, uh, are not mutating and which ones you want to have a user approval on. Here, when you have an agent who accesses MCPs, one authorization pattern that you can apply is to kind of categorize the tools that
[06:17] are defined in the MCP to, you know, what is allowed, what requires consent, and what is denied. And that is a huge leap from how most agentic systems built today where you just give it access to the MCP and then it just goes wild and even if it
[06:33] use, you know, you has OBO permissions, then user's permissions are, you know, pretty broad. Like I can delete repositories, but I don't want my agent to delete repositories, right? So, that's like a very very simple example. But, let's kind of see how it can look if we
[06:49] take it to the next level, right? So, in this design, there are kind of three properties. So, the first property is that the agent starts with like predefined permissions that are pretty narrow and are fine-grained. So, fine-grained in the
[07:05] sense that if it accesses like Google Drive or or GitHub, it it doesn't have access to the entire system. It has access to specific folders or just a specific set of repositories. And then it acquires additional permissions based on in that particular session with
[07:22] additional approvals from the user. And then once the session is done, it, uh, relinquishes all the extra permissions. And so, any new session starts fresh. So, that's kind of the design principle, uh, that we think can be durable for these kind of systems. So, how does does
[07:38] would it work in practice? At design time, um, a developer would give define a very, uh, fine-grained set of permissions to the agent. And then there will be overriding, uh, overriding, uh, further, uh, down
[07:53] scoping those permissions from, uh, done by admin policies. And then during run time to acquire any new permission, the agent would require additional consent from the user in on on whose behalf it is acting. And these permissions can be parameterized, so if
[08:09] Alex tries to fix a PR for, uh, a specific branch, then the the branch is parameterized and the agent has permission to it, but the next session will not have that permission. So, we talked about authorization, but, um, let's talk about observability. So, what
[08:24] we see here is an additional kind of class of challenges because of the non-deterministic pattern and how the control flow of agents is undefined, uh, at at request time. So, that's kind of one challenge. The next challenge is, uh, unlike in
[08:40] traditional systems where the state is well-defined and it lives in databases and variables and queues, here the context window comprises like a huge chunk of what the state is and often it's completely opaque to us. The adversarial, um, actions are very new
[08:55] here as well, so things like prompt injections and things like that like that can happen which do not happen in traditional systems. So, we need to kind of rethink the observability layer as well. And I think what's interesting here is that we actually in this part of the kind of governance domain, we do
[09:12] have a good pattern and we have a pattern that worked for us in the past very well, which is tracing of distributed systems. So, tracing is not not a new concept to us in in any kind of way, but it lends itself particularly well to agentic systems,
[09:29] uh, because it kind of covers a lot of the jobs to be done that we wanted, um, to achieve in observability. So, every LLM, uh, call can be modeled as a span in in our tracing system, and the context essentially captures the
[09:45] the payload, which includes the metadata and the context window. Uh and including the prompt. As the agent operates, we have the full end-to-end view of everything that it did, including the state that it was in and why it took certain decisions. So, tracing is key, and what we see is that
[10:03] customers who apply tracing, that's kind of the first fundamental kind of path to success in energetic governance. Um also has its uses for like evals and and improvements, but we're talking about governance. And we're the stewards of um
[10:19] MLflow, which is kind of the leading tool for uh tracing and ML ops uh ML ops and, you know, improvements and evals. And it's actually damn easy to um instrument agents with uh tools like
[10:36] MLflow. So, one line, you can instrument a tool like bot code. It takes one line, and all of a sudden you have, you know, full observability of what's happening uh through your execution of of of Claude. So, I encourage you to kind of
[10:53] give it a spin and see how it works. It's really easy, and uh the value of it is incredible. So, we talked about the first three layers. The fourth layer is just the trust. And in terms of trust, what I mean is like, can you actually assert that the agent
[11:09] the request the agent received was legitimate and what it returned was compliant? And here, too, like it's a broad class like a big class of kind of challenges that we're facing and we need to figure out what are the primitives. But what I wanted to share with you is one pattern that works incredibly well
[11:26] for most of our customers, and it is the pattern of an LLM judge. So, essentially, you get to instrument your agent in a way where each request and response to the agent goes through a separate LLM judge. It's
[11:44] a separate model and a separate call, and there can be multiples of them, which are then have the, you know, very narrow task of identifying certain issues. So, they could look for certain PII patterns and identify PII. They can identify safety
[12:02] issues. They can, um, identify prompt injection, patterns, and and so forth. So, we think this is a pretty scalable, durable pattern that we can apply uh, for this integrity problem. This is something that our team is working, um,
[12:18] on on making available in some of our, uh, products, including AI Gateway. We recently, um, announced, uh, that capability. Alex implemented these four patterns, and if previously she was paged in in the middle of the night and asked, you know, your agent put
[12:34] something in the production, what happened? She couldn't actually answer that. Now, she has the full observability with a trace ID. The context windows through the operations were captured over there. She has an audit log of all the actions and all the tools that were called and
[12:50] why certain policies were triggered, um, and allowed or denied certain operations. So, it's now pretty trivial for her to answer this question. And so, governance becomes much more much more manageable. Obviously, governance doesn't stop here, and this is not a
[13:07] complete set of challenges that we're seeing with agentic systems in terms of what we need to solve. What we didn't touch, just like as a, um, set of examples is how to do proper credential vending, how to implement kill switches when when things go wrong
[13:24] in autonomous agents. Um, actually, when you have hundreds of these agents running, how do you kind of streamline consent and how do you manage that and scale? And model governance in terms of as you deploy new model versions, how do you manage that and how
[13:40] do you ensure there are no regressions in terms of safety and security. So, yeah, it's definitely a big topic and it's exciting, you know, being this green field area with all of you. So, thank you so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.