Governing and Improving Production Agents with Omnigent and Codex

Summary

The 'humans steer, agents execute' philosophy structures agent development so that human operators focus on defining domain expertise, opinions, and permissions—curating the context that matters—rather than performing manual execution tasks themselves.
Multi-agent architectures combining OpenAI models and SDKs, Databricks enterprise context via Unity Catalog, and MLflow observability enable continuous improvement loops where both explicit and implicit feedback from agent usage feeds back into the system.
Evaluation frameworks must be embedded inside the core agent harness to keep pace with rapid foundational model updates, and deliberate human gates can be progressively removed as system reliability grows, with OpenAI's open-source SDKs enabling deployment across direct APIs or governed internal cloud gateways.

Governing and Improving Production Agents with Omnigent and Codex

The concept of "humans steer, agents execute" serves as a foundational approach to developing production-grade AI systems. This presentation covers how integrating advanced reasoning models, robust data infrastructure, and explicit human-in-the-loop control planes allows developers to build safe, context-aware, and highly collaborative multi-agent ecosystems. By focusing on curating high-quality enterprise context and structuring iteration loops, engineering teams can transition from manual verification to scalable, autonomous workflows.
𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀
• High-leverage AI development requires human operators to focus on defining domain expertise, opinions, and permissions rather than manual execution.
• Multi-agent systems can combine distinct sub-agents, such as supervisors and challengers, to evaluate code and optimize task success.
• Single-agent architectures are increasingly handling sub-agent execution internally to minimize context loss between separate models.
• Integrating both explicit and implicit expert feedback directly from agent usage forms the foundation for continuous improvement loops.
• Evaluation frameworks must be treated as part of the core agent harness to keep pace with rapid foundational model updates.
• Deliberate human gates can be strategically positioned across agent steps and gradually removed as system reliability and confidence grow.
• Open-source agent SDKs offer deployment flexibility, enabling identical reasoning models to run across direct APIs or governed internal cloud gateways.
00:02 — Introduction and the "Humans Steer, Agents Execute" Philosophy
00:51 — The Architecture: Combining OpenAI, Databricks, and Omni
02:50 — Defining What Context Matters in Development
04:35 — Supervisor and Executor Agent Paradigms
05:43 — Designing Feedback and Continuous Improvement Loops
07:46 — Embedding Evaluation Inside the Agent Harness
09:05 — Managing Human Gates and Transitioning to Autonomy
10:23 — SDK Versatility and Governance Options
11:25 — Multi-Agent Incident Response and Code Generation Demo
Check out future Agentic + AI Observability events: https://luma.com/AgenticAIObservability

Chapters

FAQs

What does 'humans steer, agents execute' mean in practice?

This principle means that human operators focus on defining domain expertise, opinions, and the permissions that govern what agents can do—rather than manually executing tasks. Agents carry out the work within those defined boundaries, with humans setting context and reviewing key decisions while agents handle execution.

How does MLflow enable agent improvement in this architecture?

MLflow serves as the observability engine, capturing traces of agent decisions and actions so that both explicit feedback (direct human ratings on agent responses) and implicit feedback (patterns observed from real usage) can be integrated back into the system. This creates a foundation for continuous improvement loops where the agent gets better over time based on real operational evidence.

What is the role of Databricks in the OpenAI-Databricks-Omnigent architecture?

Databricks contributes the infrastructure and enterprise context—Unity Catalog tables, volumes, and jobs that represent the organization's data and business logic. This governed enterprise context allows OpenAI's reasoning and orchestration models to act meaningfully on behalf of the business, with MLflow on the Databricks Data and AI platform providing the observability layer.

How do evaluation frameworks keep pace with rapid model updates in agentic systems?

This video argues that evaluation must be embedded inside the agent harness rather than treated as an external step, so that every foundational model update is automatically assessed for behavioral changes before reaching production. Deliberate human gates positioned at key agent steps can be gradually removed as the system demonstrates consistent reliability.

Full transcript

[00:05] I am Wesley. I'm a applied AI engineer at Open AI and today I'm going to talk about sort of bringing Open AI, Databricks, and Omnigen, whatever you want to call it, sort of bringing them all together and sort of bringing the best out of all of them. And the way I'm orienting this talk is around this
[00:18] concept of human steer and agents execute. This came from an Open AI blog post from I want to say early this year. Truthfully, I was like really mad I didn't think about it because this little slogan cuz this is how I've been thinking about developing agents for a
[00:30] while but this is like a way better concise description that I had come up with. But this I think really really aligns well with a lot of what Corey just went through with what Omnigen's less Omniagent offers and then also what
[00:42] Open AI and kind of Databricks offer together. I think this is a really complimentary story across all these three different providers. I know Omnigen is and Databricks are sort of one entity but really that they're kind of three and what I'll go through. So specifically, what I'm talking about
[00:54] here is for the Open AI side, as you might imagine, the models. So the general intelligence, the reasoning layer. But then also on top of that, the SDKs that we offer. So we have the Open AI agent SDK as well as the Codex SDK which are open source ways that you can
[01:07] either use Open AI models or whatever your chosen model is with those harnesses. So Open AI side of the equation is bringing the reasoning, bringing the the intelligence, and then also bringing like a harness around that to make sure that you can configure an
[01:19] agent in the way that makes the most sense. But then on top of that, you've also got Codex which is the Open AI coding agent as as a sort of separate entity. Databricks brings the infrastructure, the context especially in the form of enterprise context. So I'm sure there's plenty of Databricks
[01:31] customers here. So you might have Unity Catalog tables, you might have Unity Catalog volumes, a ton of data, a ton of jobs that contain your enterprise context that we can bring to these models, to the general intelligence to
[01:43] then act in on your behalf in in ways that make sense to you. But then on top of that, observability. So as what Corey and I worked on together is MLflow as a real engine behind that. And MLflow is going to be the means in the demo that I'll show later of how I'm both getting
[01:57] observability in the form of traces, but then also integrating direct human feedback. And then the last thing that I'm using on the Omniglot I'm going to think about that every time I say it. Omniglot is for the human approval process. So it hits on a lot of what Corey talked about
[02:09] earlier, but I'll go through an example of having explicit steps of where I'm approving certain steps and and not approving it. So Omniglot kind of becomes my control layer on top of all that. So if you combine all that together from OpenAI, you've got both
[02:21] the models and the surrounding architecture around it from Databricks I about the infrastructure, the enterprise context, and then Omniglot is sort of my control plane where I'm interjecting that human review. So this is like going back to what I talked about from the beginning, human steer, agents execute.
[02:35] This is like I believe like a very human-centric way of developing in the future where we now our jobs kind of become how do we curate the right context to provide to the agents so that they can execute. And then this sort of this combination of of services is is I
[02:48] think a great way of of trying to enable that. So in terms of like what is that context that matters? I think this truthfully is like what's becoming the biggest developer task at this point is determining what matters. So this can be in the form of domain expertise, it can
[03:00] be in the form of very strong opinions about how things should be done. It can be in the form of how do I pull things out of people's heads across the organization into actual data so that I can deliver that to the agents in the right way. And I think actually Omniglot
[03:12] is a really interesting way of doing that. Like the approval process is part of it. So often that's done maybe in in a meeting room or in a conversation. It's not codified anywhere. I think it's going to be interesting to monitor now that we're sort of codifying it through
[03:25] those formal approval processes, you can now use that as inputs to agents. Say we didn't like the model routing. Well, maybe if you're providing that feedback in an explicit manner, you can then inform the next iteration of your agent to then in a smarter way to your
[03:37] context, to your specifications, route to a different model because that fits your specific use case. Then evidence perspective. So a lot of this with the quality of models and the expansion of the context window, you can't just dump everything and tell it to figure it out.
[03:49] That's obviously going to be a high latency, higher cost equation. So then again, going back to what I talked about before, I think a huge aspect of our role to then how do we distill what really matters to avoid that sort of brute force technique. Expert feedback,
[04:01] I'm going to table that cuz I'm going to talk about that in much more detail in a minute. And then the permissions of what the actual agent can do. So this is like a a really comprehensive way to think about what context matters. But I think really like this, so at my role at
[04:13] OpenAI, I work directly with customers and I had a very similar role at Databricks. And what I've been advocating for is like as developers, this is where we should be spending most of our time. Thinking through, this is where you get the most leverage out of scaling systems is thinking through how
[04:26] do my opinions matter? How do I curate the right context so that I can create optimization loops that then let agents actually figure out the execution portion. So this is another a little bit tangential concept, but I thought it might be interesting to bring into this
[04:38] and it is part of the demo. but something I've been experimenting with a little bit more lately and I think it's becoming more more popular is sort of this supervisor executor concept. What I'll have in the example later is a of a
[04:50] supervisor agent that then kicks off to a challenger agent to then sort of evaluate the quality of what the supervisor did and see if it comes to a differing opinion. I'm going to be doing this in a multi-agent system in the demo I show later. What I'm starting to see
[05:02] more and more of is this actually occurring within a single agent architecture where instead of spinning off dedicated multi-agents, you now have a single agent where that single agent is spinning off sub-agents with its own
[05:14] context. I think that for from an OpenAI perspective, something we really focused on with the 5 6 series of models and of that is you have you minimize the one-hopping of context where now you need to provide the right context from one agent to another where things can
[05:26] get lost. Instead, you're relying on the agent itself, the model to figure out the right context within its own context parameters given that sub-agent. So I think this is just an interesting concept that I wanted to at least bring up and I think especially that single
[05:38] versus multi-agent, I think it's going to be increasingly tilting more towards single agent. But again, in the demo I I do have a multi-agent example. This is really what I spend the most of my time on. I've kind of been a little bit obsessed with this idea for, I don't know, like 12-18 months or so, but the
[05:52] whole concept, in my opinion, is that domain expertise, strong opinions, is really the scarce resource at this point. That our job is to really codify the way that we think, get the right information, and then pass that to the agent. The combination of OpenAI,
[06:04] Databricks, and I'm going to do can do is exactly that. So, in the example I'll go through and just in general, the observe phase, I'm using MLflow here to get tracing and to to make sure that I'm capturing everything that's happening within my agent. Domain experts then use
[06:17] the agent. This So, there's sort of two versions of this. One would be like explicit feedback that your agents give you. I'll keep using that example of we have the model router. It used Tycoon, it should use Fable, it used GPT-54-Nano, it should have used 56-Soul. That's a
[06:30] version of like explicit feedback, where I say, "This is bad, I shouldn't have done this." The other version of that is implicit feedback, and I think this is the area that is increasingly becoming more capable of actioning on, where just by the mere fact of the experts using
[06:43] the agent itself, that feedback alone, seeing what happens in that, can then inform the next version of your agent. So, sometimes it can be explicit feedback. Other times it can be just implicit of repeated failure modes, of asking the same thing over and over
[06:55] again as a as a as own input, where you have a separate agent that reviews that and identifies what those failure modes might be, and then turning that into explicit feedback. Because increasingly what we want to do is to move closer and closer to where agent usage itself
[07:08] becomes what actually informs the next iteration of your agent. You're not as dependent, because obviously people's time is valuable. You can't expect them to continue to review over and over and over. That becomes the scarce resource.
[07:20] So, how do we leverage them actually using the agent in as minimal time on their perspective as possible in order to drive our own versions of improvements. So, on the improve side, this is where I personally have been using Codex really extensively to then
[07:33] take interpretations of whatever feedback I have, both implicit and explicit feedbacks, and then recommend the next update of your agent improvement loop. So, now here's a new version of my agent that's going to be better based on explicit signals that I got from my trusted sources. And then
[07:47] finally, it's a little bit of a jargony statement, but what what I always tell customers is like you need to think about evals as part of the harness. This is I think a very common failure mode I see especially lately with the pace model release increases. So like just to
[07:59] put it into context, I think the gap between GPT-2 and GPT-3 was like 15 months and then 5.5 to 5.6 was like 6 weeks. So we're releasing the pace is just getting out of control and the capabilities are getting out of control.
[08:12] If you don't have robust evals to be able to tell you how good your agent is, it's going to be really really hard to assess whether the new next version of the models actually going to work for you or not. On top of that, given the increase in capability, if you're not
[08:24] evaling your evals in the same capacity, if you think of these as like two very distinct workflows, it becomes extremely common, something I see very frequently with the customers I work with, that the evals themselves aren't actually reflective of the quality of the agent.
[08:36] It kind of gets disconnected where the evals themselves are measuring your prior version of the agent, if you will, and they're holding back what the actual agent performance is. So thinking about evals as part of your harness is a great
[08:48] way to kind of get around that. And that's it's all related here by if we can incorporate feedback into the process, we can then use that to drive new improvements. But again, I think the one thing to take away that the concept is that like taste, I kind of hate that term, but it's kind of become common,
[09:00] taste, expert feedback, and that's the scarcest resource. How do we then take that and make that extend that as far as we can? So this is sort of where for the Omnigent step comes in, again a very human-centric portion of it. And the way I've been thinking about it is starting
[09:12] with human gates at many many places that we think are relevant. And then as you get more confidence in the system, as you get more just successes in general, how do we think about removing those gates such that we're driving
[09:24] towards more autonomy? And so as just an example of what I just talked through, let's say we have a hypothetical agent out there where the agent's generating traces, we have a developer that's curating the the right traces to then pass to a a agent to determine what
[09:36] improvements to drive. Then we have markdown file that gets generated that actually informs the next version of the agent. And then we have a actual release cycle where we take that new agent and push it to production. You could have human gates at each of those steps. A
[09:48] human person that's involved or human that's involved with reviewing the curated traces to see if they're the right ones. You could have a person involved with reviewing the markdown file that gets generated. But perhaps over time you get to the point where you
[10:00] get confidence in the way that the agents are doing that themselves where you remove those gates to where you're now leveraging your time in the place that makes the most sense because you built a system that's better. So that's I think like what Omnigen offers is the ability to apply these gates in very
[10:12] deliberate places. But then the challenge I would say in in a similar framing as what I said about evals being part of the harness, think about ways that you can pull those apart so you can spend time in higher leverage situations as you get more confidence in the way the system's working. Obviously OpenAI,
[10:24] I'm assuming most folks are familiar with the with the models at least, but the the SDK is a flexible open source way of leveraging the models in different places. So in the demo I'm going through, again, I'm assuming many Databricks customers in here.
[10:37] You can use OpenAI models both directly through OpenAI APIs. You can also use them through Databricks foundation model APIs through the AI gateway. I have a like as I mentioned, part of the reason I built a multi-agent system was to show this concept is that you can I have one
[10:50] aspect of my multi-agent system is calling OpenAI model directly through the OpenAI APIs. Another one is using the Databricks foundation model APIs hosted on a Databricks app, but both are using the OpenAI agent SDK. So you have flexibility of how you want to deploy
[11:03] your models while still So you can basically take the exact same model and say I want to deploy this on Databricks for this or whether it's governance, contractual, whatever the reasons are, you can just take that model and then deploy it in Databricks in the same capacity as you would directly from the
[11:15] OpenAI APIs. It's something I I often see like is not super intuitive, but it's powerful concept that you can just take this one model that I developed, the flexible open source SDK, and then deliver it wherever you'd like. And then in this demo, yeah, like I mentioned,
[11:27] Omnigen is sort of that control plane layer. So this is the actual architecture of what I'll what I'll show in a second. So to what I actually I'll show a video. I've used computer use for the video more kind of just for fun. So the human is a bit misleading here, but
[11:40] you can pretend you'll definitely be able to tell from the language used. It's like the most I call it code XE language of all time. It's like very very machiney. But you have a human pass into the omnigen what I'm calling the omnigen commander. It's the UI that
[11:52] Corey was showing earlier. You've got three different sub agents. So I have an incident agent. So this hypothetical scenario is basically like something broke in my application. I have an incident agent that goes in to try to figure out what went wrong. I have a a challenger agent that then reviews what
[12:04] the incident agent said and and gives its assessment as to whether it thinks it identified the right problem. And then mostly for demo purposes, I will reject that and I will pass this off to a native code XE session to then spin up
[12:16] the right answer to the solution. So Databricks is providing data as part of this problem. These agents have access to internal data within Databricks to make its determinations. As I mentioned, I'm using the foundation model APIs for the challenger agent specifically. I'm
[12:28] using the direct OpenAI API for the incident agent. And then MLflow, you'll see how I integrate human feedback into this process as part of that code XE rejection. So this is my computer use equivalent of a human put this message
[12:40] in there. The checkout agent's running and it's going to kick off the challenger agent. So the checkout agent's given a diagnosis. It's passed to the challenger agent. I like this in omnigen that has the different sub agents working. This is a custom agent that omnigen is pointing to. So it's not
[12:53] pointing directly to a Claude code a code XE session. It's pointing to a specific custom agent that I built that I've given it access to. So you can see in this bottom it says checkout agent. That means that it's referencing that specific custom agent rather than one of
[13:05] like the direct harnesses themselves. So I'll pass this along to the challenger agent gives its assessment and it figured out there's a bunch of these problems going on. And then it's registered this in my MLflow with an explicit trace ID which I'll show why in
[13:17] a second why that's so valuable. So then the next thing it is identifies is given these problems, it's time to ask for a a human review. Now I need somebody to come in and actually assess. Is this the right fix? Should we then deploy this
[13:30] right away based on these two custom agents? For dramatic effect and for demo narrative, I will reject this part of it. I I don't think that's the right answer. Instead of just spinning up a Codex agent, you see that MLflow URL,
[13:42] the MLflow assessment. I'm going to go into my trace and apply direct feedback on the trace of what I think should have happened. So now, after I do this on this you see this right side thing, again, very Codexy language, but Codex
[13:54] goes in or my my computer vision agent goes in and says this is what should have happened. This is what I think should happen. So then it comes back to my Omnigen session, which takes that feedback in and assesses what the next step should be, which it determines
[14:06] should be spinning up a a separate Codex session that will take in all of this evidence, what the agents thought, what the challenger thought, what my human feedback was, and then propose its own solution. So I'm getting approval here. Is it okay for me to spin up a new Codex
[14:19] session? That gets spun up, so you can see that in the on the right side here. New new sub agent is now part of my Omnigen ecosystem. So Codex goes off, it works for a little bit. I'll fast forward that part a little bit. I do like this in Omnigen how it's like chatting with each other. I think that's
[14:31] a fun little concept. So then it comes back. Codex has built a new solution. It comes back and tells me what that is. It gives me the context and the MLflow trace ID. And then I get one final approval request of can I push this new one to production based on what it was.
[14:44] And now I push that one to production. So this is I think a kind of a like I I tried to bring it all together where we've got the OpenAI models. So these are just to be clear, these are three different OpenAI models essentially. The commander agent, which is directly
[14:57] through the OpenAI APIs with the OpenAI SDK that's kind of has tooling to go into Databricks and pull a bunch of data. I have the challenger agent, which is assessing the original commander agent's input. And then I have my final one is a Codex agent using the Codex
[15:10] harness that then builds the final solution because I wasn't satisfied with those two initial ones. And throughout the process, I'm leveraging Databricks data and I'm using Omnigen as my control plane to then make those approvals. So I I it's an interesting way of showing an instructive way of showing how they all
[15:23] kind of three work together. This to me, I felt it's the been the future for I know again 12 18 months and it kind of feels like the present right now. So now it becomes a challenge for us to figure out what is the right context, what are the right systems to put together so
[15:35] that I still retain control as the person in charge of the system, but I'm providing the system with the right context so it's making the choices that I want, not the general choices that a generic model might make. So yeah,
[15:47] that's all I wanted to discuss today. Thanks for the time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.