Control and Governing Your Coding Agents with LLMOps

Summary

Coding agents like Claude Code, Codex, and Gemini CLI require LLMOps tooling to address three organizational challenges: observability into what agents are doing, evaluation of output quality, and governance to control costs and prevent unsafe actions.
MLflow serves as an all-in-one LLMOps platform providing one-command tracing setup for coding agents, LLM judges for scoring output quality on metrics like user frustration, and AI Gateway for budget caps, guardrails, and PII reduction controls.
A live demo shows Claude analyzing its own MLflow traces through an MCP integration and proposing structural improvements to its workflow, closing a trace-to-evaluation-to-structural-improvement feedback loop.

Control and Governing Your Coding Agents with LLMOps

Coding agents like Claude Code, Codex and Gemini CLI are transforming how software gets built, but without observability they remain a black box. This talk shows how to take control of your coding agents using MLflow. We cover tracing agent sessions to understand token usage and tool calls, evaluating output quality with LLM judges and custom scorers, routing LLM traffic through AI Gateway for cost control, and automating workflows with MCP and skills. Walk away with practical techniques to monitor, measure and maximize your coding agent investment.
Talk By: Daniel Lok, Software Engineer, Databricks ;

Chapters

FAQs

What is LLMOps and why does it matter for coding agents?

LLMOps refers to the tooling and practices for working more effectively with large language models in production environments. For coding agents, it matters because agents like Claude Code, Codex, and Gemini CLI operate with significant autonomy—understanding what they are doing, measuring whether their outputs are correct, and controlling costs requires specialized infrastructure beyond standard software monitoring.

How does MLflow trace coding agent sessions?

MLflow provides a one-command tracing setup that captures token usage, tool calls, and session structure for coding agents. The resulting traces can be viewed in the MLflow UI or queried programmatically, enabling teams to understand exactly what an agent did during a task and identify failure patterns across historical sessions.

What is AI Gateway and what governance controls does it provide?

AI Gateway is an MLflow component that acts as a proxy layer for LLM traffic, enabling teams to set budget caps per user or team, apply guardrails to prevent unsafe actions, and reduce PII in requests before they reach the model. It provides a centralized control point for managing coding agent costs and compliance across an organization.

How can Claude analyze and improve its own coding agent behavior using MLflow?

This video demonstrates Claude connecting to MLflow through an MCP integration, reading its own traces, identifying failure patterns such as insufficient context or repeated tool call cycles, and proposing structural improvements to its workflow configuration. This creates an automated feedback loop from trace analysis to actionable workflow improvements.

Full transcript

[00:07] Hi everyone and welcome. My name is Daniel. I'm a software engineer working on mainly AI open source projects, in particular MLflow and the recently announced Omnigen project. Um so, you know, today we'll be talking about how to control and govern your coding agents with LLMOps. And
[00:20] coding agents have been a very hotly discussed topic over the last, you know, couple of days. I think everyone is talking about them and the sun has become more and more central um to all developer workflows, even non-developer workflows. Um so, as we begin to trust
[00:33] them with more autonomy, it becomes more and more important to have tools in order to sort of uh number one, understand what they're doing. Number two, you know, make sure that their blast radius is limited. Right now they
[00:45] have lots of autonomy. We need to make sure that they're not doing the wrong things. Um making sure costs don't, you know, sort of, you know, overrun the budget, so on and so forth. And probably most importantly, making sure that they're effective um in
[00:57] working on the individual task that we give them, right? Um you know, all the models are the same, right? Claude is Claude, but the context in which they operate is all very individual, right? Um you've got your code base and each of them have sort of built-in biases, you
[01:11] know, sort of uh coding practices, so on and so forth that that, you know, all these models need to be aware of. Um so, LLMOps as a general category are sort of tools that help you, yeah, essentially, yeah, work with LLMs better
[01:23] and sort of solve some of the frictions around them. Right? Uh so, just a brief overview of the talk. Uh we'll first go through what some of the problems with coding agents are, um both from an individual perspective of a developer sort of
[01:35] working um yeah, working with coding agents every day, doing doing sort of coding tasks, uh as well as from an organizational standpoint. So, like managing teams of people working with uh coding agents, so on and so forth. Uh we'll then go
[01:47] through sort of what LLMOps is, just to make sure we're on the same page talking about it, cuz it's a very broad category. We won't have time to sort of uh you know, go into like each individual component. Um but
[01:59] we'll we'll speak of like a few particular things. Uh and then afterwards we'll have sort of a practical live demo of how you can use MLFlow um to kind of integrate into your coding agent workflow and ideally make it better um suited for sort of whatever
[02:12] you're doing. All right. Uh so my hope at the end of the talk is that all of you walk away with um you know, some way to to use MLFlow uh in order to kind of improve coding agents uh working on your code base.
[02:25] All right,
[02:37] come a long way uh in the last year even the last few months. I'll say even in the last um at this point last year I was using Cloud Code to write code um but the way of working was very different, right? I would sort of be
[02:49] hand-holding it through a session. It'd be very single-threaded, you know, I kind of work work with it on one task and I sort of make sure the tool calls are correct, you know, make sure I'm auditing every edit that it makes, you know, in a very single-threaded way. But now the models are so good that um
[03:03] essentially I'm running a few Cloud Codes in parallel, right? I think this is very common among my teammates as well. Uh where we sort of work on many different tasks and trust that, you know, they're sort of doing the right thing in general and our involvement as humans is
[03:15] basically just at the input stage, so constructing the plan and the output stage, sort of reviewing what the model has done. Right? But you know, what goes on in the middle is kind of becoming more and more black box, right? So that's that's the first problem, right?
[03:27] We have real no no real visibility out of the box uh into behavioral behavioral patterns of these coding agents. Um you know, we sort of trust that they kind of do what they, you know, do the right thing in general. Um but when they
[03:39] go off the rails, you know, there's no sort of audit trail, there's no way we can sort of um go back and check, you know, what what they've done. I mean yes, you can scroll back in your Cloud Code session, but be realistic, you know, who's going to do that, right? Um so that's the first problem and because
[03:51] of that it becomes very difficult to know how to improve their performance when you're working with them, right? So, I mean, I've run into this situation many times where, you know, I'm I'm running a few clock codes in parallel, but I'll notice that one of them is sort of spinning for a long time, you know,
[04:03] it's been half an hour, it's been 1 hour, there's been no output. Um I look at the session and it looks to be calling tools that, you know, are eating up a bunch of tokens, but don't seem to be contributing to progress. You know, it's reading a bunch of unrelated files.
[04:15] And often I have to stop it to ask, you know, what it's doing and sort of course correct, right? And I think over the course of doing that over many sort of days, weeks, months, we develop sort of intuitions on what tasks coding agents uh do well in
[04:28] our code bases and which tasks they struggle with. And based on those intuitions, we, you know, create skills, we create, you know, cloud MD guidance, you know, agents.md files, you know, these sorts of things. But it's all very vibes-based, right? Um
[04:40] it it's kind of like, yeah, we feel that there's a problem. We kind of make a solution that we think will work and then we just hope that it works, right? So, it's it's very difficult to have like sort of a concrete uh way to improve them and make sure
[04:53] that they have improved, right? It's all very vibes-based. So, that's that's the second problem. Um but so so that's as an individual kind of working day-to-day with uh coding agents. But as an organization, you know, you have larger uh concerns,
[05:05] you know, making sure that your developers don't sort of burst the budgets, uh making sure that um you know, governance is kind of supported. So, models don't do things that they're not supposed to do. There's PII reduction, you know, these sorts of things.
[05:18] Um so, this is where LLM ops comes in. As like a general category, I don't think there's an sort of accepted one definition, but in general, it's just sort of tools that help you manage, you know, these problems that I've I've gone through to make it easier to um work
[05:32] with LLMs and to solve some of the frictions around them. Uh we won't have time, as I mentioned, to go through like, you know, all of the problems, but I wanted to talk about three in particular, uh observability, evaluation, and governance. Um
[05:44] So, observability is kind of concerned with, you know, making sure that audit trail is there as I mentioned. You know, making sure that the coding agent's execution chain is not a black box, right? So, having some sort of record of what the coding agent done, so that, you
[05:56] know, if you want to come back or query it programmatically or something, there is some physical record of the, you know, the sort of execution chain there. Um, so that's observability. For evaluation, uh, this is more concerned with how the
[06:08] coding agent is doing on a specific task. As I mentioned earlier, you know, right now we kind of do vibes-based evaluation. We feel that the coding agent is good at some things and not at others, so on and so forth. Um, but it would be great to have tools,
[06:20] right, to create more structured evaluations or at at least some sort of more, um, codified, uh, metrics, right? We we want to actually specify what performance means and and and be able to
[06:32] like measure it over time to make sure that our skills, our guidance that we provide the coding agent actually improves, um, its performance. Right? And lastly, you know, governance, this is kind of kind of the standard thing that I mentioned earlier. Having guardrails, having budget caps, um,
[06:45] making sure that the the model doesn't access data that it's not supposed to or, you know, kind of leak external data externally, so on and so forth. Um, and actually solutions for all of these things exist kind of, you know, it's it's a bit fragmented in the
[06:57] market, right? For observability, you have tools like, um, LangFuse, um, or LangSmith. Uh, for evaluation, you have things like BrainTrust, you know, there's this, um, many things. And for guard, uh, for governance, there's guardrails.ai, you
[07:09] know, there's there's a bunch of tools for each of these. But as I mentioned earlier, you know, I work on MLFlow and MLFlow is really, um, great because it's kind of an all-in-one platform where each of the solutions to all these problems are well integrated with each
[07:21] other and it's very easy to kind of, um, yeah, basically solve your problems individually, but have all the solutions kind of link up, um, to make it easier to kind of extract value and make sure that you get to sort
[07:33] of production faster. Right? So, I mean, yeah, I just I'll bring you through, um, you know, MLFlow has MLFlow tracing for observability. Um, we have judges, you know, LM judges, code-based judges, um, for structured
[07:45] evals. Um, and there's MLflow AI gateway. So, I I'll mainly be talking about the open source version uh, of MLflow in this talk, but you know, on on Databricks, there's basically an equivalent for every single one of these things.
[07:58] Um, yeah, there's, you know, Unity AI AI gateway that's been announced, you know, in in the keynotes and such. So, yeah, don't be worried if, you know, you see this is all kind of MLflow branded. I'm just talking about open source, but Databricks has equivalents. Right?
[08:10] So, the next thing I want to do is bring you through sort of a live demo um, on how you can use MLflow to instrument cloud code or any coding agent really, you know, um, I think we support Codex, Quen Code. There's like there's quite a few that we support now to make sure
[08:23] that each of the interactions with your coding agent so, you can think of this as, you know, uh, a cloud code session. You know, let's say I I run a cloud code session to perform a task, making sure that uh, each interaction in the session
[08:35] is actually logged and saved to like some physical location um, that can be kind of queried in a structured programmatic way down the line. Uh, I'll also bring you through like how you can uh, perform an eval. Uh, just a just a lightweight one, you know, I I think
[08:47] when when we say the word eval, people kind of get scared, you know, like, "Oh, we're not there yet." But, you know, it can be very simple, just like sort of natural language metric. Um, you basically say like, "Hey, you know, like I want to did the agent perform well in
[09:00] this sort of thing?" You can describe that in natural language and have kind of like a basically a a structured output uh, and a consistent way to kind of measure performance uh, of let's say any individual cloud code session. And then,
[09:12] of course, tying it all back together, how do we use this data in the end to create, you know, changes in our code base, like enduring changes that will actually like measurably im- improve our coding agent's performance. Right, so without further ado, let me jump in.
[09:25] So, as I mentioned earlier, um, MLflow has a bunch of tracing integrations with uh, you know, we support cloud code, Codex, Gemini, like and and we're adding more and more over time. Um, so, you know, if you have any requests, please raise it on the MLflow
[09:38] open source GitHub. Um but I'll be going through how to trace cloud code. It's super simple. All you have to do is basically add the uh MLflow cloud plugin. Um and and and that's it, you know, like you just install a cloud
[09:51] plugin. Uh and I've already done this ahead of time to uh you know, sort of save some time, but I will be going through like what the setup looks like. Again, it's super simple, right? I just run MLflow tracing uh the setup command.
[10:04] And what this does is it's sort of um basically it asks, you know, where's your MLflow server, where should I store the traces, so on and so forth. Um so, it'll be asking us the questions. Um uh so, please uh use I think there's
[10:19] actually supposed to be a structured uh you know, kind of question and answer, but it's all right. Uh I have a just a local MLflow open source server running here. Uh this server and the dice
[10:32] double experiments. All right. Okay. Uh yes, project only. So, you can you can basically set this up at the user level,
[10:44] so this will apply to all your cloud code sessions on your machine over all sessions, but for you know, to keep this demo scope, I'm just going to um yeah, basically uh set it up just in this project, which is the Omnigen repo. Uh this you know, this is not a um the
[10:56] topic of the talk, but you know, just as a side as a side thing. Um All right. So, now that that's been set up, I should actually start seeing the traces come in. Oh, yes. So, I have a cloud code
[11:08] session. Um and now if I spin up a new cloud code and okay, I I'm just using Omnigen as the front end for cloud code. I I'm uh during the keynote yesterday, they went through a bunch of features of
[11:20] Omnigen, um but you know, you can also just use it as a front end for cloud code basically. Uh it's a great way to sort of manage this parallel working um mode that you know, I think more and more more of us are sort of going towards. we have like many cloud core
[11:33] sessions open. I'm going to get a great just front end for that. So I'll say like tell me a joke. And I think so it's getting ready but basically what should happen is that I should see
[11:46] that interaction show up here. So that's done. All right, so you can see okay. Tell me a joke. Okay, let me start to add that joke to the read me file.
[11:58] And I also okay I for this demo I'm just going to like fan some user frustration. So I'll say that joke was so terrible. Uh
[12:10] just so you know we have some way to follow eval to sort of um say that Yeah, the user was frustrated basically. Okay. So we have the trace
[12:23] um and all all of this is sort of logged in into like chat sessions, right? So any given session is um you know sort of logged as as a separate entry here.
[12:35] Uh okay. So yeah, the terminal view is is available too. Um I am frustrated. Let me just let me just make sure that Claude knows I'm frustrated.
[12:49] All right. Okay, so we should see all of these things come in. Right? And as I mentioned earlier uh in order to sort of quantify uh quantify our frustration, right? We want
[13:02] to actually add some sort of assessment here. Which is to say like on this given basically what you can think of this as on this chat session, right? In this cloud core session I want to have a metric for like how
[13:14] frustrated I was. So I'm going to actually have a few built-in LLM judges. So you can actually specify like a natural language prompt. We have a built-in one called user frustration which sort of measures Um, yeah, how
[13:26] frustrated a user in this given session. There's three outputs, right? There's none, which is like the user was not frustrated at all. There's resolved, which means that the user had some frustration within the course of this chat, but it was eventually sort of like, um, you know,
[13:39] resolved. And there's unresolved, which means the session ended with the user still being frustrated. So, okay, um, I'll I'll kick this off. Uh, I'm pointing it to my, uh, OpenAI uh, GPT-5 endpoint to run the judge since it is a
[13:51] natural language like LLM-based judge. And while it runs, I'll just take you through the AI, uh, gateway, um, thing. >>
[14:05] And what this is is is basically MLflow AI gateway, right? So, I can sort of specify a secret here. I can make sure that, um, the budgets on this endpoint are set. Um, and I can add guardrails as well. Like,
[14:17] the budgets tab is here. I I won't have time to demo all of the AI gateway, um, uh, features here, but just know that you can set budgets on each individual endpoint. Uh, you can also add guardrails like natural language ones.
[14:29] Um, you know, we have a few built-in, uh, judge like built-in guardrails for common, um, functions such as detecting harmful responses, PII reduction, so on and so forth. You can actually make them
[14:41] run pre- and post- LLM, which means like, uh, should this request be blocked before it even reaches the LLM, or should something from the LLM be scrubbed, right? Um, so yeah, I I won't be demoing the the guardrails, uh, thing, but
[14:54] yeah, essentially like all requests that go through, uh, this endpoint, uh, should be logged. So, now let's go back to the sessions. Uh, earlier I ran the user frustration, uh, metric.
[15:07] And OpenAI tells me that indeed there was some unresolved frustration here with some, uh, rationale, right? This user specifically stated I'm frustrated. Uh, and of course this is kind of a contrived example, but you can imagine it working on a longer Cloud Code
[15:19] session where you you were actually frustrated. I think uh, it's funny there was a Cloud Code source the source code leak of Cloud Code a while ago and I think they actually count the number of times users say WTF uh in in in like a
[15:31] given session as a measure of frustration. So, this is kind of similar. Um it just runs in a structured way through MLflow. Right? Um so, as you sort of build up more and more sessions, uh I'll I'll show you an example of I'm actually
[15:44] using Cloud Code to develop uh Omnigen. So, this is sort of all the traces that that I've accrued last couple of days days and you see like I've ran the user frustration um sort of metric on all of my sessions so
[15:57] far. I was using it to develop quite a few features, you know, over the past couple of days. And it kind of the results kind of align with my subjective experience, right? Most of the sessions go on fine, you know, Cloud kind of one shots them.
[16:09] Um but there were a couple of sessions where I felt like, "Okay, you know, it's spinning for a long time. The um basically the uh the result wasn't what I wanted and I had to kind of cross-correct a couple of times." Right? So, that's kind of reflected in these
[16:21] evals here. Um so, I I demoed earlier kind of being able to uh you know, run an ad hoc um an ad hoc eval through this UI, but you
[16:33] can actually set it up such that uh this user frustration metric runs on all future traces. So, you don't have You just set this up once and then for all future times that you're interacting with Claude and generating these sessions, um this metric will sort of
[16:45] run automatically. So, as you use Cloud Code more in your code base over the course of, you know, days, weeks, you'll end up with something that looks like this. You know, you have a bunch of data over here. Um but the question now is what do you do with that data, right?
[16:58] Like, "Okay, I have all these I can see that I was frustrated in some of my sessions." How do I use that to actually reduce my frustrations in the future and make some structural changes to my code base um to, you know, actually make things better? Well, one way is obviously, you know, you can spot-check.
[17:11] You can see like, "Okay, uh do some pattern matching by yourself." Um but the better way, of course, is to ask Claude to do it. Um the I'm a flow has an MCP server, right? That you can kind of hook Claude up to. Uh it's it's pretty simple. All
[17:24] you have to do is add it as an MCP server to Claude um via some CLI command. Uh and I've already set this up, so Claude knows. And I think I did this yesterday. Obviously, you know, it takes a long
[17:37] time for Claude to fetch all of this and do the analysis and all of that. So, I'm not going to do this live, but I'll show you the um the session in which I did. Right?
[17:49] right? I didn't even specify where my server was. It kind of just figured it out. Um And you know, this is sort of the experiment that I that I was mentioning. I have all these like eval zone, right? And I asked it to uh look at all the
[18:01] sessions in which I ran into frustrations and propose some structural changes. So, Claude went through and sort of, you know, did some analysis, fetched all the fetched all the things, and it came up with some diagnoses, right? Number one,
[18:14] uh you know, it's often when Claude is making a PR, uh it fails CI, right? So, it has uh you know, identified one problem there. Uh another problem, I'm often telling Claude to run some CLI commands from scratch. Maybe that should be a skill.
[18:27] Um you know, so on and so forth. They generated quite a few recommendations. Um and it turns out that this code base doesn't even have a Claude.md file. So, it proposed, you know, something that I can, you know, check into my code base basically to automatically make it
[18:40] better for the future. Uh it also proposed some skills that I could create. Oh, yes, also like specifically a front end Claude.md. So, there's a project level and a front end specific one cuz it seems I ran into a lot more problems in the front end.
[18:52] Uh it proposed a couple of skills that I can I could, you know, write to to make it sort of uh reduce the friction over there. Um And that's pretty much it. Uh it it it basically asked I mean I I could ask it
[19:05] to, you know, implement these itself. And what I should see in the future is after I implement these things, eventually the incidents of these over the last like 7 days for example, you can group by time. Ideally, we should see these numbers tending more towards
[19:17] green, right? Um hopefully like reducing frustration. So, I hope yeah, this is a very practical example of how you can use um MLflow, MLflow tracing, and eval to sort of improve um the performance of your
[19:29] coding agent moving forward. And this is super easy to set up. Um the documentation is on mlflow.org. So, please go ahead and check it out. Um yeah, I hope all of you will start tracing cloud code. Um and ideally using
[19:41] it to make some structural changes. All right, I think that's all the time I have. Um so, if you walk away with one thing, just, yeah, install MLflow and set up cloud code tracing. Thank you, everyone. Yeah.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.