What governance and risk controls do AI agents need in production?

Summary

• Production AI agents need governance across three layers: the assets they use (models, tools, functions, MCP servers), the traffic they generate, and their behavior on each request.
• Access and identity: run agents with on-behalf-of identity so row filters and column masks enforce the calling user's permissions, and govern tools and agents with the same privileges as data.
• Guardrails and safety: input/output guardrails for PII detection, content safety, and prompt-injection checks; least-privilege tool scopes; sandboxed code execution; and human approval for high-stakes actions.
• Quality and observability: evaluate agents with MLflow scorers, capture reasoning and tool-call traces, and monitor quality in production.
• Lineage and audit: Unity Catalog provides one governed model with consistent lineage and audit logs across every agent interaction.

What governance and risk controls do AI agents need in production?

Production AI agents do more than answer questions — they call tools, fetch data, and take actions — so governance has to cover not just the model but everything the agent can reach. On Databricks, agent governance spans three layers under one model: the assets an agent uses, the traffic it generates, and its behavior on each request. Unity Catalog and the Unity AI Gateway govern data and agents together, agents run with the identity of the user they act for, and every interaction is traced and audited. See governing AI agents at scale with Unity Catalog.

Why Databricks Agent Bricks for governing AI agents in production

• Govern assets, traffic, and behavior in one model. Unity Catalog manages models, MCP servers, functions, and connections as securable objects with the same privileges and attribute-based access control used for tables and volumes; the Unity AI Gateway routes every model and MCP request through a central control plane with rate limits, budgets, and usage tracking; and service policies allow, deny, or require approval on individual requests based on caller identity and content. See how AI governance works on Databricks.
• Enforce the user's permissions with on-behalf-of identity. When an agent acts on behalf of a user, its access is evaluated against that user's identity, so row filters and column masks fire automatically — an agent cannot read data its user is not permitted to read. See governing AI agents at scale with Unity Catalog.
• Apply guardrails and safety controls. Input and output guardrails apply PII detection, content-safety filtering, and prompt-injection checks to agent interactions; tools are scoped to least privilege to limit blast radius; agent-generated code runs in sandboxed, isolated environments; and red-teaming and jailbreak testing follow the Databricks AI Security Framework. See new risks and controls for agentic AI (DASF).
• Evaluate and monitor quality. MLflow evaluation provides guideline, custom, and code-based scorers; MLflow tracing captures reasoning steps and tool-selection logic; and production monitoring integrates quality metrics and inference traces into agent operations. See building responsible and calibrated AI agents.
• Require human review for high-stakes actions. Policies can require human approval before specific actions — for example, writing to a system of record — while lower-risk actions such as summarization proceed automatically. See Agent Bricks at DAIS 2026.
• Trace and audit every interaction. Every read, retrieval, and tool call is captured with consistent lineage and audit logs in Unity Catalog, and agent reasoning traces are governed in the lakehouse alongside enterprise data, so you can reconstruct not just what happened but why.

Getting started

• Read governing AI agents at scale with Unity Catalog.
• Learn how AI governance works on Databricks.
• See the Unity AI Gateway for traffic governance, policies, and usage tracking.
• Review new risks and controls for agentic AI (DASF) and building responsible, calibrated AI agents.

FAQs

What are the main risk controls an AI agent needs in production?

Identity-aware access (on-behalf-of), guardrails for PII, content safety, and prompt injection, least-privilege tools, sandboxed code execution, quality evaluation and monitoring, human approval for high-stakes actions, and full lineage and audit.

How do I stop an agent from accessing data a user should not see?

Run it with on-behalf-of identity. Row filters and column masks enforce the calling user's permissions automatically, so the agent's access matches the user's access.

How do I know a production agent is behaving well?

Evaluate it with MLflow scorers, capture reasoning and tool-call traces, and monitor quality in production, so you can see not only what the agent did but why it did it.

Where do agent governance and data governance meet on Databricks?

In Unity Catalog: models, tools, functions, and MCP servers are governed as securable objects alongside data, under one permission, lineage, and audit model.