Safe Agent Deployment: Risk Frameworks and Platform Design

Summary

Ben Lehrburger from Anthropic's applied AI team shares the internal risk framework that guides agent deployments, covering four dimensions — mode, delegation, autonomy, and interaction — and mapping six interaction models from human-directed tools like Claude Code through fully autonomous services.
Three critical failure modes Anthropic encountered in production — token leaks, sub-agent permission bypasses, and credential management gaps — led to platform architecture changes including hooks in the Claude Agent SDK for deterministic control and Managed Agents to keep credentials separate from model logic.
Anthropic's internal CI pipeline demonstrates the real-world scale of these practices, with Claude reviewing approximately 70% of pull requests, making robust governance and debugging capabilities essential for day-to-day operations.

Safe Agent Deployment: Risk Frameworks and Platform Design

Deploying AI agents in production reveals challenges that lab testing misses: token leaks, sub-agent permission bypasses, and credential management gaps. Ben Lehrburger from Anthropic shares the internal risk framework that guides agent deployments across four dimensions: mode, delegation, autonomy, and interaction. The framework maps six interaction models from human-directed (Claude Code) through fully autonomous services.
Learn about three critical failure modes Anthropic encountered and how platform architecture mitigates them. See how Claude Agent SDK uses hooks for deterministic control, how Managed Agents separate the brain from the hands to keep credentials secure, and why durable session logs enable debugging and recovery across infrastructure changes. Includes insights on Anthropic's internal CI pipeline, where Claude reviews 70% of pull requests.
🤝

Chapters

FAQs

What is Anthropic's risk framework for agent deployment?

Anthropic's risk framework evaluates agent deployments across four dimensions: mode, delegation, autonomy, and interaction. It maps six interaction models ranging from human-directed tools like Claude Code to fully autonomous background services, helping teams understand how much oversight is appropriate for each deployment pattern.

What are the three critical failure modes Anthropic encountered in agent deployments?

The three failure modes are token leaks, sub-agent permission bypasses, and credential management gaps. These were discovered as Anthropic moved agents from lab environments to production and directly informed how the Claude Agent SDK and Managed Agents architecture are designed.

How do hooks in the Claude Agent SDK improve agent safety?

Hooks provide deterministic control points in the otherwise non-deterministic agent execution loop, allowing platform-level policies to be enforced regardless of what the model decides. This makes it possible to add consistent safety checks, logging, or policy enforcement without relying solely on the model to behave correctly.

How does Claude participate in Anthropic's own engineering workflows?

Claude reviews approximately 70% of pull requests in Anthropic's internal CI pipeline, demonstrating the real operational scale at which Anthropic runs its own agents. This internal deployment drives ongoing improvements to governance tooling as edge cases discovered at scale inform how Anthropic designs its external-facing agent products.

Full transcript

[00:05] Uh, my name is Ben Lerberger. I'm here with Anthropic. And specifically I sit on the applied AI team. So, that's like in between product research and folks building on Claude, helping you guys translate all the cool stuff that they're doing internally, which sometimes I don't even understand either, and helping you guys like
[00:20] squeeze the most use out of Claude. And then taking feedback and using that to influence what product and research do. Notably, I am not like a security and observability researcher. I am just in charge of translating all of the cool things that they find out for you guys.
[00:35] So, that's going to be my goal today is to make a lot of our internal learnings and also from the field intuitive for you guys to start building with. Quick agenda. Going to walk through our risk framework and how we think about agent deployments.
[00:51] What this actually means in in practice and like failure modes that we've run into with our agents. What we're doing to improve our architecture with our external-facing products, things that you guys can build on. And then anchor this a little bit in the current
[01:07] in sort of like the current point in time that we're at. Many folks may be familiar with the recent Midas release. Like that makes I think we scheduled this before that happened, but that makes this all the more pertinent and I'm glad that I'm here. And then I want to keep everything honest. Like we have not solved
[01:23] a lot of this and so you guys are probably going to going to share practices. David is going to as well that that like we can learn from too. So, this is purely collaborative. If anyone read the abstract of the talk, you saw this is part of the framing, which is like all we want to do is have
[01:40] agents running ambiently in the background and doing work on our behalf at Anthropic. This is like my core goal is to automate myself so I can like go hang out somewhere on the beach or whatever. But this is unfortunately not the reality at the moment. So, you can
[01:56] imagine getting getting like an incident in the middle of the night, and we have Claude take the first pass at all of these. It's It's triaging before an on-call engineer gets pinged. It's an open question about how do you give agents right access safely? Like, if you if you have a if
[02:13] you like deploy and you don't have enough pods, how do you let the agent sort of update that in in a with within known good values? Um otherwise, you end up in situations like this where it's nuclear entire code base. If anyone's like doing local development with with coding
[02:29] agents, you feel this pain have probably felt this pain. Um and so, we're going to talk about how to try and avoid that. The framing around all of this is that most of the systems that we're working with today were either made for humans or they were made for services. Uh and humans are
[02:44] pretty easy because they like each have their own individual credentials, and so they can be held accountable for things. Uh they are non-deterministic. Like, you don't know what they're going to do with those credentials, but you can at least trace it back to them and be like, "Yeah, like, why why'd you do that?" Um and then services are sort of the flip
[03:01] side where they're very deterministic. Everything is governed by code. You know, like, clicking this button is only going to do this. Uh and so, they have service accounts. They have their own credentials, and that's safe because you know exactly what behavior is going to follow from them. But, we're at this weird point in time now where we
[03:17] have agents which don't fall into either of these buckets. They are non-deterministic. You don't know what they're going to do, and they like to use service accounts. They don't really like authenticate on their own. And this means that we have to design for this new paradigm. Open question about how do we fit the existing tooling to uh sort
[03:35] of comply with with our with our new counterparts here. And this has led to our internal risk framework at Anthropic. Uh I don't think this is published anywhere, but this is like from the internal zeitgeist. We really think about risk in terms of like how far is an agent from a human?
[03:54] And so like mode is is a human watching. Delegation is how is the agent authenticating? Does it have its own credentials? Is it using a service account? Autonomy is how much can it decide to do by itself before it has to like check in for help or have a human
[04:09] decide what to do? And then interaction is actually how hands-on keyboard is an agent. Do they have an opportunity to intervene or not? And this leads to is all these six different agent interaction models. And moving from top to bottom here,
[04:24] these are in order of like the most human in the loop and what we are most comfortable with today all the way down to what makes us super nervous. And the first one is human directed. So like if you just install Claude Code out of the box, this is what you're going to
[04:40] get. Claude doesn't want to like take any action without you explicitly approving permissions. And so you'll have to intervene for every single one. Then there's human super supervised. Say you turn on auto accept edits mode with Claude. Now you are overseeing it, but you've given it permissions to take like
[04:56] a very narrow slice of action in your workspace. Human in the loop. This is where the agent has the ability to escalate to an individual. Like if you run {{}slash} install GitHub app in Claude Code, you have Claude running your CI pipeline and
[05:12] it will just review your code and then relay the findings back to you. So it is autonomous, but you are definitely in the loop. We're starting to get to the dangerous zone here quite literally with like dangerously skip permissions. This is human out of the loop. You have now fully delegated permissions to an agent
[05:27] and it took us quite a while to figure out how to do this right. We just launched auto mode in Claude Code which Yeah, it's it's it's a lot of fun. But it was like there were some learnings that I'll actually share in a little bit that you know, the road to get there was
[05:43] not was not totally straight. Okay, and then this is like this is the real danger then this is where we're not really having any activity today. We're just this is more of of of what we're thinking about or where we want to get to. So, managed services where you have an agent acting not on
[05:59] behalf of individuals but on behalf of a team. So, say that you are like deploying an agent as your like your on-call triaging agent. It's not responding to any one person. It is escalating to a team. Permissions are expanded and then lastly are autonomous services. This is where I
[06:16] want to get to so I can go on the beach. This is like agents that do not have to escalate to humans ever. We work with customers and people internally who are experimenting with like just purely agentic businesses and startups like having agents run every aspect of it. This is totally hands-off. They will
[06:32] never escalate. They will just run. Our experimentation down this ladder has, you know, we've seen some trials and tribulations. One of them having to do with auto mode and so I want to share some of those with you guys just so we can all like learn from mistakes that we have made and then also know how that informs some of our
[06:48] product decisions. Number one is and these are all like from from the last month or so. We were reviewing one of one of our agent directories and saw that it essentially like STD aired
[07:06] a token into into the output stream. And we never gave Claude a token. It just pulled this from the CI environment. This is like like CI agent. And what we realized is that we had no guard on a subprocess
[07:22] processes ability to pull tokens from the CI environment. So, it was just a matter of time before one of those subprocesses misused one of those tokens and as a result we had to implement this new environment variable like scrubbing your CI tokens from from
[07:38] subprocesses. Right? Like this is a very narrow example but I do I it anchors on Murphy's Law. Anything that can go wrong will go wrong, and so we very much want to defend against that. Another one uh is um uh and this was like probably a couple months ago.
[07:55] Uh we wanted to productize dangerously skip permissions as auto mode. And when we were testing this, we realized that this whole time uh an a a like parent Claude you know can spawn sub agents, um there was no guard
[08:11] on the permissions that those sub agents could have. In other words, we were expecting that they would inherit the parent's permissions. When in fact, they could uh like the the orchestrator could just say, you know, bypass all for the sub agent. That is incredibly dangerous and not by design. Obviously, we made it
[08:28] so sub agents uh just inherit the parent's permissions, but like it took a lot of experimentation and actually having this thing live in prod for a while to even find this out. Um and so, we're trying to be proactive about designing against this. The last
[08:43] one uh which happened very very recently is we were testing a new model, and Claude was flagging or the model was returning in the transcripts uh that there were like a bunch of prompt injection attacks. And we were looking at the logs, and there were no prompt injection attacks. And it turns out that
[09:00] there was like another uh sort of security review process that was injecting an extra prompt after inference time. So, it didn't make it into the logs, but the model was seeing it in its uh in in the response that it was getting. And the point here is that
[09:16] like the logs that that we can review as as as humans are sort of the source of truth, um and that is our only lever to debug. And so, in our product, which I'll talk about now, um it we are like really anchoring on those for for for security. What these all have in common
[09:32] is that none of them were the the the fault of the model. Um models are not inherently dangerous. They are only dangerous if like the human is dangerous or if the platform that they're running within is poorly designed. And so, the point of leverage for us now is how do
[09:47] we build secure platforms not only for us to develop with internally, but for folks like yourselves to build with externally. Cool. So, we have two agent frameworks these days. Um, number one is the Claude Agent SDK. This is just the Claude code harness exposed for you to build on top
[10:04] of and like deploy wherever you want. And then we recently managed agents, which is essentially the same thing except we hosted on our own infrastructure for you. So, you can just go like zero to one with an agent with a system prompt and a few tools. Uh, both of these, by virtue of them
[10:19] having different deployment environments, we had to design uh, a little bit uniquely for. Our first like line of defense and my favorite because it's very easily accessible from Claude code and the Agent SDK is hooks. So, the core idea behind this is the agent loop is non-deterministic, but you can inject at
[10:36] certain breakpoints in the agent loop with deterministic code. So, hooks are you can like they they pretty much anchor on like before a tool is called, after a tool is called, before you return a message to the user. Um, and they say like, "If that happens, uh, then take this action." So, they are
[10:52] deterministic. The agent cannot get around them. And the result is that you can sort of filter your agent trajectories or control them a little bit deterministically. And the other primitives that we'll see are in this vein. It's about how do you bring determinism into a non-deterministic agent loop? The other one the other ones
[11:09] have to do with managed agents. Uh, this was a very hard problem. Uh, and the the crux behind our architecture with it is like with the Agent SDK, the agent harness, uh, what we would essentially call the brains, like whatever's hitting the Claude API and then the harness around it, um, is
[11:25] living right next to your your like .env file or your environment variables. It also is responsible for the session logs. So, the point is if you're to deploy an agent on the agent SDK and then some part of it breaks, whether it is like the the session log's not working or the harness going down or
[11:42] even like the agent SDK version changing, you cannot recover any part of that. You can't spin it back up again. Sometimes your session logs are gone. So this is a nightmare for observability and the key is to break out dependencies. Uh what we call like separating the
[11:57] brain from the hands. This is a horrible example, but like if you cut off my hands, it would be terrible if my brain stopped working too. But the point is uh if we keep those separate, then you can have one brain managing uh governing many hands or many hands uh talking to, you know, one brain, vice versa. So
[12:14] uh what this allows us to do is like remove any environment variables or credentials from the agent's workspace. The agent is never like, you know, going to pull down those environment variables from CI. It's going to reach out to an
[12:30] entirely separate service like a secrets vault, which by the way, this would have been the ideal infrastructure to begin with. It just took us a while to get there. Uh but it'll ask, it'll be like, "Hey, like uh given this user and this agent, do we have permission to uh to take this action in GitHub?" Pull down that result
[12:47] and then send it back to the sandbox environment, which is where the agent is like writing its code and running its results. In that way like your credentials are never in a place that the agent could write code to do something with them. So we're separating out all of these attack vectors.
[13:02] The other one as I was mentioning is the session log. Uh so this now lives with managed agents in durable storage. No longer is it like sitting locally. Um the session log is the source of truth. So say that I start up a managed agent session, I like do some work and then I go away for 24 hours. We can now spin
[13:19] back up a container and rehydrate it using the existing session log. So our infrastructure is no longer the bottleneck. It no longer determines the success or state of our agent. All of this exists in separate pieces, so you can just take that session log and pick back up from where you left off. It also
[13:35] allows you to change agent framework versions. You can upgrade the SDK because that's happening in a sandbox where only the harness is living, right? So, we're separating all these things out. And the question is why are we investing so heavily in in in these platforms now?
[13:51] Um I do think this is an evergreen problem. Like I I mean, there's a reason that like I'm here and our security researchers are like hands-on keyboard back in back in San Francisco. Uh but there we are also at a special point in time. Uh specifically like our eyes have been have been opened to this
[14:08] in large part due to due to Mythos, which is capable of a lot more offense than we thought models were going to be. Um again, I want to remind you like the point of this is that the model does not determine what you can do in terms of like agentic deployments. That's the
[14:23] platform you build around it. But on the offensive side, models are getting very good and very dangerous. Uh and so, we're working on ways to like distill Mythos's intelligence down into something that is accessible for everyone to use. Um and we will not like release it
[14:40] publicly until we get to that point. But when that happens, you're still going to have to have defensible systems that are designed against some of these like inherent attack vectors that that I was talking about. Uh and I have to acknowledge that like we don't have this figured out. You You
[14:56] might have seen a few weeks ago like we literally shipped our source map in a Claude code release. So, like things still fall through the cracks. This is not solved. Um and we are very much learning from the community as well. Uh but
[15:11] I think agents in particular are important because you want agents to be acting on your behalf, but you know, maybe a few years ago you didn't have systems that were taking action in the world that also had access to your internal data stores, to your off credentials. And so, at the same time
[15:28] that our models are getting better, we are also exposing more and more of our vulnerabilities to the world and allowing autonomous systems to do stuff with them, right? So, we kind of have to like meet this equilibrium where it is and try and actually get ahead of it. Um, what does this mean for how we're
[15:44] doing things internally at Anthropic? Uh, we have some agents that are like sitting further down that ladder, uh, not totally just like off on their own. They all have escalation paths. They can all reach back out to a human if they need to. Some of the things that we've had pretty good success with is like uh,
[16:01] our CI pipeline, which you heard me mention. Um, we have an agent built on the agent SDK that runs most of our CI to the point that like 70% of our PRs are only reviewed by Claude. The idea behind this is like one, the agent can always escalate up to a human.
[16:17] And then two, we have fenced off 30% of mission-critical part of the code base where like there we have we still have code owners on it. So, this is evergreen. We are constantly trying to delegate more responsibility to Claude, but it takes quite a bit of work. Same thing with like internal uh, internal
[16:33] research. Uh, we try to like expose our knowledge base for for agents to work with and that has proven very effective. Uh, but obviously also like a pretty big attack vector, too. Where we want to get is having all of our verticalized like specialty like I
[16:49] am on the applied AI team, AI researchers really want this. Having agents that can do pretty much everything that they can without having to escalate to the human. Um, and we are really far from that, I think. Um, but we're constantly working to move
[17:04] more and more uh, of of like our day-to-day tasks to autonomous systems.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.