How do I manage a fleet of coding agents and know they're actually shipping good software?

Summary

• Managing a fleet of coding agents takes two layers: control over what agents can reach (models, tools, spend) and evidence of what they did. Whether the code is good is judged by your existing engineering gates, such as tests and code review.
• The Unity Gateway CLI (ug) routes coding agents such as Claude Code and Codex through Databricks model services, so developers use Databricks credentials with no provider API key on their machines.
• Unity Catalog privileges, rate limits and budgets control who can use which models and how much they can consume. Tool access is governed through MCP services.
• The usage system table (Public Preview) records the coding agent, version and session ID when the client provides them, and the usage dashboard's Coding Agents tab reports activity such as sessions, commits and lines of code. These are activity measures, not quality scores.
• Tracing, service policies for tool guardrails, and central agent configuration are in Beta.

How do I manage a fleet of coding agents and know they're actually shipping good software?

Treat it as two separate questions. The first is control: which models, tools and budgets each developer and agent can use. The second is evidence: what each agent did. Unity Gateway helps with both for model and tool traffic, because coding-agent requests pass through one control point. It does not replace the checks that decide whether code is good, such as tests, static analysis and human code review in your own delivery pipeline.

What does fleet management need?

Need	What to look for
One path for model traffic	Agents connect through a central endpoint, and developers do not each hold a provider key
Access control	Who may use which models and tools
Limits	Request, token and spend caps per user or group
Activity records	Which agent, version, session, model and tool produced each request
Tool guardrails	Blocking or approval for risky tool calls, such as writes to a repository
Central configuration	One setup for models, tool servers and skills across developers
Quality evidence	Test results, review outcomes and defect data from your delivery pipeline

How does Databricks approach this?

One path for agents. The Unity Gateway CLI (ug) connects coding agents such as Claude Code, Codex and OpenCode to models, MCP tools and shared skills. The Databricks tutorial states that every request routes through a Unity Gateway model service using your Databricks credentials, with no provider API key on the developer's machine.
Access and limits. Unity Catalog controls who can use which models: grant EXECUTE on a model service to a group. Rate limits cap requests or tokens per minute per service, user or group, and budgets set monthly thresholds that alert or block usage.
Activity records. Unity Gateway logs usage to the system.ai_gateway.usage system table (Public Preview). Its session_metadata includes the coding agent, agent version, client surface, reasoning effort and session IDs, and MCP rows record the tool invoked in mcp_metadata. The built-in usage dashboard has a Coding Agents tab with active days, coding sessions, commits, and lines of code added or removed.
Tracing (Beta). The unified trace table stores gateway request traces in OpenTelemetry format. The Unity Gateway CLI can also send client-side traces from Claude Code and Codex to the same table, so you can see the tools and skills used alongside model requests. Delivery is best-effort, so the docs say the table complements audit logs rather than replacing them.
Tool guardrails. You choose which tools an MCP service exposes. Service policies (Beta) can allow, deny or ask: a built-in GitHub policy blocks write tools while allowing read-only ones, and an ASK decision pauses a tool call until the user approves it.
Central configuration (Beta). A workspace admin can publish one coding agent configuration covering enabled agents, models, MCP servers, skills, and features such as tracing.

How do you judge whether the software is good?

The documented coding-agent metrics measure activity, not quality. Judge quality with signals from your delivery pipeline, such as test and static-analysis results, pull request review outcomes and defects found later. Use the gateway records to see which agent, version and model were involved in a session you want to investigate.

FAQs

Can I require approval before an agent does something risky?

On MCP services, yes, with service policies, which are in Beta. A policy returns ALLOW, DENY or ASK. ASK pauses the request for user approval before the tool runs, and external agents receive the prompt through MCP URL-mode elicitation. On model services ASK is not available, so Databricks blocks the request or response instead.

What happens when an agent hits a limit?

A rate limit returns HTTP 429, and clients should retry with exponential backoff. A budget set to block usage stops further requests through Unity Gateway for that user, who sees a message that the budget is exhausted until it resets or an admin raises the threshold. Budget enforcement is approximate, so it is not an absolute spend cap.

Can different teams' agents have different permissions?

Yes. Model each team as a Unity Catalog group, grant EXECUTE on different model services and MCP services to different groups, and set custom rate limits per group. Callers also need USE CATALOG and USE SCHEMA on the parents.

Where can I see which tools an agent called?

Query system.ai_gateway.usage with service_type = 'MCP_SERVICE' to count calls by mcp_metadata.tool_name. MCP changes and invocations (mcpCall) also appear in system.access.audit.