How should I use an AI gateway to govern agentic AI?

Summary

• Agentic AI systems require governance at three layers: which agents and models teams can access, what external tools and data they can reach, and what policies constrain their actions.
• Unity Gateway extends Unity Catalog governance to AI assets, treating agents, models, MCP servers, and skills as governed resources with permissions and audit trails.
• Enforce on-behalf-of execution so agents inherit the requesting user's exact data access levels, preserving row-level security across tool invocations.
• Apply runtime policies (allow/deny/require approval) and LLM-based guardrails to block harmful outputs, redact PII, and enforce business rules without stopping legitimate work.
• Monitor end-to-end agent execution: see which agents accessed which tools, called which models, and triggered which policies.

How should I use an AI gateway to govern agentic AI?
Agentic AI (systems where agents autonomously invoke tools and access data on behalf of users) introduces governance challenges beyond single-model systems. An AI gateway for agentic workloads extends governance from models alone to the full agent ecosystem: agents, tools, models, and user identity/permissions.

What is agentic AI governance?

Agentic AI governance ensures agents operate within security, compliance, and business boundaries:

• Asset control: Who can create, modify, and deploy agents?
• Tool access: Which external systems can agents reach?
• Execution context: Do agents inherit user permissions or operate with static credentials?
• Runtime enforcement: Can policies block outputs that violate rules?
• Observability: Can we audit all tool calls and policy violations?

Key governance capabilities to evaluate

Capability	Why it matters for agents	What to test
Agent and asset registry	Agents should be discoverable, versionable, and access-controlled like data assets.	Register an agent. Grant one team access, deny another. Verify control works.
On-behalf-of execution	Agents must inherit user data access levels, not operate with static credentials.	Configure an agent with row filters. Have two users invoke it. Verify each sees only their rows.
MCP and tool governance	MCP servers and tool APIs that agents call should be registered, audited, and access-controlled.	Create a tool registry entry. Restrict access to one agent. Verify other agents are blocked.
Runtime policy enforcement	Policies define what agents can and cannot do, applied at request time without requiring code changes.	Define a policy: "no agent can delete from this table". Have an agent attempt deletion. Verify it's blocked.
LLM-based guardrails	Guardrails powered by an LLM can understand context and enforce nuanced rules (e.g., "block PII in outputs") without rigid string matching.	Enable PII redaction guardrail. Have an agent generate output containing sensitive data. Verify redaction works.
Cost attribution and control	Track which agent, team, or project consumed which models, and enforce spend caps.	Set a spending limit for one agent. Have it invoke expensive model calls until budget exhausted. Verify requests are denied.
End-to-end observability	Trace every agent invocation: which model was called, which tools were accessed, which policies were checked, which user was executing.	Run an agent workflow. Query the audit log. Verify all steps are recorded.

How Databricks approaches agentic AI governance

Unity Gateway as Control Plane
Unity Gateway, built on Unity Catalog, serves as the front door for agentic workloads. Register agents, models, and MCP servers as UC securables. Teams request access; admins grant it with role-based policies, mirroring data governance.
On-Behalf-Of Execution
Agents must preserve user data access levels. If Alice has row-level access and invokes an agent, it queries with Alice's filters, not a static account. Agents call Genie Agents or MCP servers directly (not via the gateway) to preserve the user's token and context.
Service Policies for Agents
Define agent-specific rules: allow policies (permit specific agent+tool combos), deny policies (block access), approval policies (require review). Policies key on user, agent, model, MCP service, request/response contents.
LLM-Based Guardrails for Agents
LLM-based guardrails evaluate outputs contextually: PII redaction (scan and mask), business logic (check risk policies), tone and safety (ensure on-brand). Define by specifying a model and prompt. The gateway evaluates every response in real time.
Cost Controls and Observability
Unity Gateway provides unified cost tracking across all models and agents. You see spend by:

• Agent (which agent called which models).
• Team or project (cost attribution).
• Model and provider.

You can set hard spend caps: requests are denied if the budget is exceeded. For observability, all agent interactions are logged to Unity Catalog inference tables, giving you complete audit trails.
Multi-Model and Multi-Provider Support
Agents can route to Databricks-hosted models (Claude, GPT, Gemini, Llama), external providers (OpenAI, Anthropic, Bedrock), or a mix. Unity Gateway handles routing transparently, so agents don't need to know which provider is being used. You can swap models, add fallbacks, or load-balance without changing agent code.

Critical success factors

Start with one use case. Pick a workflow where governance adds value: customer support agents, financial agents, IT agents.
Test on-behalf-of execution. Verify agents inherit user data access levels. This determines governance success.
Define policies collaboratively. Work with security and compliance. Avoid over-restrictive rules.
Monitor and iterate. Start with logging, then tighten policies.

FAQs

How does on-behalf-of execution work technically?

When a user invokes an agent in Databricks Apps or a notebook, the request carries the user's authentication token. The agent inherits this token when calling Genie Agents or MCP tools directly. This preserves row-level and column-level access controls.

Can we use different models for different agents?

Yes. Each agent can specify which models it prefers. Unity Gateway routes to the right model (Databricks-hosted or external) transparently. You can also configure fallback models if the primary is unavailable.

What happens if a policy is violated?

It depends on the policy type. Deny policies block the request and return an error. Approval policies queue the request for review. Guardrails can modify the output (e.g., redact PII) or block it. All violations are logged for audit.

How do we know if an agent is misbehaving?

Unity Gateway logs all agent activity to inference tables: which agent, which user, which tool, which model, which policy checks occurred. Query these logs to investigate suspicious behavior or audit compliance.

Can agents access data they shouldn't?

Only if you grant them permission. Agents inherit user permissions, so they respect row-level security. If an agent needs broader access than the user (rare), you can configure improved permissions with explicit policy approval. This is audited.

What's the difference between service policies and guardrails?

Service policies are rule-based (user, agent, tool name) and deterministic. Guardrails use an LLM to evaluate context. Guardrails are more flexible but add latency.