How do I prevent a custom AI agent from taking unsafe actions?

Summary

• Prevent unsafe agent actions with defense in depth on Databricks: service policies that allow, deny, or require human approval for tool calls; AI guardrails for prompt injection and unsafe content; least-privilege tool permissions; on-behalf-of identity scoping; and human-in-the-loop approval.
• Service policies are runtime controls that evaluate every interaction between an agent and a model or tool, both before the call and after the response, returning allow, deny, or require-approval.
• Built-in guardrails can block unsafe or harmful content, block jailbreak attempts, detect and redact sensitive data such as credit-card and Social Security numbers, and block hallucinated responses — attached to model or tool services without custom code.
• Least-privilege permissions in Unity Catalog scope each agent to the specific tools it needs, and policies can restrict a specific tool or condition when it can be called — limiting the blast radius of any single agent.
• For high-stakes actions, a policy can return ASK to pause a tool call until a person approves it, so an agent cannot push code, write to sensitive locations, or send regulated data without a human in the loop.

How to prevent a custom AI agent from taking unsafe actions

A custom AI agent can call tools, run code, and reach enterprise systems on its own, so keeping it safe means controlling what it is allowed to do at runtime — not just what it was designed to do. Databricks applies defense in depth: service policies decide whether each action proceeds, AI guardrails screen the content flowing through the agent, least-privilege permissions limit which tools it can touch, on-behalf-of execution scopes it to a user's own access, and human-in-the-loop approval gates the highest-stakes actions.

Why Databricks prevents unsafe AI agent actions

• Service policies evaluate every action. Service policies are runtime controls that evaluate every interaction between an agent and a model or tool. They enforce decisions at two points — before the call and after the response — and can allow, deny, or require approval for actions such as modifying files, pushing code, reaching an enterprise system, or handling sensitive information.
• Built-in guardrails, no custom code. You can attach built-in guardrails to model and tool services that block unsafe or harmful content, block jailbreak attempts that try to circumvent a model's safety instructions, detect and redact sensitive data such as credit-card and Social Security numbers, and block responses that contain hallucinated content. These policies are deterministic (pattern-based) or LLM-based and require no custom code.
• Least-privilege tool permissions. In Unity Catalog, you set who can call which tools and MCP (Model Context Protocol) servers, and layer service policies to restrict access to a specific tool or define conditions on when a tool can be called — for example, allowing a destructive operation only for administrators. Scoping an agent to the tools its task actually needs limits the blast radius, so an over-permissioned agent does not become a single point of compromise.
• On-behalf-of scoping. With on-behalf-of (OBO) execution, an agent accesses data with the permissions of the end user it is acting for, not a broad account tied to the agent — which inherently limits what it can do and helps prevent privilege escalation. Service policies can also scope decisions by caller context, for example requiring approval when an agent, rather than a person, attempts a write action.
• Human-in-the-loop approval. For high-stakes actions, a policy can return an ASK outcome that pauses the tool call until a person approves it, and agents can request user confirmation before running code — with options such as approve once or always allow within the current session. Because approval fatigue is itself a risk, the design encourages reserving approvals for the actions that matter.
• Prompt-injection defense. Guardrails inspect inputs for prompt-injection and jailbreak attempts in real time and run inline on every request, as part of a broader set of controls for securing agents and autonomous systems.

Getting started

• Read how Unity Catalog secures your agent actions and controls for securing AI agents and autonomous systems.
• Create and attach a service policy to allow, deny, or require approval on your agent's model and tool calls.
• Follow the tutorial to add service policy guardrails to a model service and govern the content of model service interactions.
• Review mitigating prompt-injection risk for AI agents to scope least-privilege permissions and on-behalf-of access.

FAQs

How can I stop an AI agent from performing a dangerous tool call?

Attach a service policy to the tool or model service. Service policies evaluate every agent interaction before the call and after the response, and can allow, deny, or require human approval — so a dangerous call can be blocked or paused for review before it runs.

How does Databricks handle prompt injection?

Built-in guardrails inspect inputs for prompt-injection and jailbreak attempts and check outputs for unsafe or sensitive content, running inline on every request. They can be attached to model and tool services without custom code.

How do I require a human to approve high-stakes agent actions?

Configure a service policy that returns an ASK outcome, which pauses the tool call until a person approves it. Agents can also request user confirmation before running code, with options such as approve once or always allow for the session.

How do I limit what tools an agent can use?

Use Unity Catalog to grant least-privilege permissions on the specific tools and MCP servers an agent needs, and layer service policies to restrict individual tools or condition when they can be called. On-behalf-of execution further limits the agent to the end user's own access.