Sponsored by: Open AI | Turn Data Signals Into Intelligent Action With OpenAI and Databricks

Summary

Enterprise value from AI requires more than intelligence—agents must work from governed business context, respect permissions, and take action, which is why Databricks provides trusted data and OpenAI provides the reasoning and orchestration layer.
This video demonstrates an MLB Statcast agent built on the Databricks Data and AI platform using the OpenAI SDK, GPT-5.5, Unity Catalog functions, and Genie to turn governed data signals into reviewable, actionable outputs.
Codex, OpenAI's agent for knowledge work and coding, can connect to Databricks Genie spaces to access governed enterprise data and orchestrate a versioned improvement loop driven by evidence from real user traces, with human-in-the-loop gates controlling which agent versions reach production.

Chapters

FAQs

How do OpenAI and Databricks work together for enterprise AI agents?

Databricks provides the governed business context—trusted data, definitions, permissions, and source-of-truth grounding through Unity Catalog—while OpenAI provides the reasoning, orchestration, and action layer through its models and SDKs. Together they enable agents that can interpret business signals, coordinate tools, and prepare next steps within governed boundaries.

What is Codex and how does it connect to Databricks?

Codex is OpenAI's agent for knowledge work and coding, acting as a teammate that can reason, act, and complete complex work across tools and workflows. Teams can add the Databricks Genie plugin to Codex so it can connect to governed Databricks data and carry that context into the rest of its work, whether starting from ChatGPT/Codex or from inside Databricks.

What is the versioned agent improvement loop described in this video?

The improvement loop involves daily candidate generation where an agent's instructions, tools, and control flow are automatically evaluated and recommended for improvement based on evidence from real user traces. Human-in-the-loop gates control which agent versions get promoted to production, ensuring changes are validated before deployment.

Why is intelligence alone not enough for enterprise AI to create value?

As explained in this video, intelligence by itself does not create enterprise value—agents need to understand business signals, work from approved and governed data, and take action within defined boundaries. The pace of model deployment has accelerated from years to weeks, but value only appears when intelligence can reason across context and help teams reach outcomes faster.

Full transcript

[00:09] I'm Wesley. I'm a uh AI deployment engineer at OpenAI. And today we're going to explore what changes when Frontier Intelligence uh can work with your data uh systems and context your business already trusts using OpenAI and
[00:21] data bricks. So for enterprises, value shows up when intelligence can reason across context, tools, respect permissions, and help teams get to outcomes faster.
[00:33] Over the past several years, we've seen a rapid increase in what models can do. They can reason, they can analyze, they can write code, and they can work across increasingly complex tasks. We've also seen an significant acceleration in the
[00:47] pace of model deployment from years down to weeks and that's a pace that's likely to continue. Um and in short intelligence really is not the constraint for an enterprise model intelligence by itself does not create
[01:00] value. It needs to understand the business signal work from approved data and take action within the governed boundaries. And that's where data bricks comes in. So once an agent can work from focused governed context, it can participate in
[01:13] the workflow itself. The access model has to work for teams, employees and admins. Data bricks provides the governed business context, trusted data, definitions, permissions,
[01:26] and source of truth grounding. OpenAI provides the reasoning, orchestration, and action layer. interpreting the signal, coordinating tools, and preparing useful next steps in whatever surface you're in, whether it's your
[01:38] browser, desktop, mobile, etc. So, if you're not familiar, Codeex is open's agent for knowledge work and coding. Think of Codex as a teammate that can reason, act, and complete
[01:51] complex work across your tools and workflows. Codex is the center of the demos today and how to extract the most of your data when working with both OpenAI and data bricks. So let's talk about a specific example
[02:04] of this in practice. Uh I'm not sure if folks use data bricks genie or have used the data bricks mcp connection uh from codecs but here a team can add the data bicks genie plugin and connect their
[02:16] genie spaces in codeex. Codex can then begin from authorized data bricks context and carry that context into the rest of the work and the rest of the context that codeex has access to
[02:28] whichever direction you want to build. And by direction I I mean sort of into data bricks or into open AAI first. Uh codecs can orchestrate that work. If your team start and chat GBT or Codex it can connect to govern data bricks data
[02:41] and bring that context into the workflow. If you build inside data bricks with agent bricks genie or model serving codeex can work across the code tools tests and deployment process there
[02:53] as well. You don't have to move your your data or pick one or the other. uh codeex brings open AI intelligence directly to that data. So this gives builders a consistent way to turn enterprise context into useful agent
[03:06] behavior whether the user enters from open AAI or data bricks codeex is the orchestrator. So I'm going to go through a demo in a second and this is sort of the high level architecture but this is a a kind
[03:20] of a concrete way of showing how open AI and data bricks can work together. So the of the demo that you'll see I'm it's it's being served in a data bricks app and sorry if I'm yelling I feel like I'm talking over the music in the background but it's being served over a data brick
[03:34] app and using a model serving endpoint with with LLMs. Um I'm using a OpenAI agent SDK runtime to actually create the agent and then I'm also using the MLflow responses agent for the orchestration or for the observability layer. uh uh GVT
[03:48] 5.5 via the data bricks foundation model API is the actual intelligence and I'm using the govern managed MCP tools that data bricks offers to directly connect to Unity Catalog. So let's just get into
[04:01] the demo here and I'll kind of walk through what all that means. So this is a Major League Baseball I don't know if there's baseball fans out here, but this is basically a a agent that sits on top of Major League Baseball Statcast data
[04:15] and can answer specific questions. So uh what I'm going to show is sort of asking a question and then walk backwards through what the agent actually looks like. So in this case I'm asking uh does a specific pitcher what does that specific pitcher do in two two counts
[04:28] and then what is the spin rate on the knuckle curve? So when I kick this request off, you can see in this top right corner, it starts calling Unity catalog tools. And let let's just walk through what the architecture actually looks like. So at the start, like I
[04:41] mentioned, this agent is an agents SDK, OpenAI agent SDK, uh run agent where I'm using GBT 5.5 um as the intelligence layer. It's all registered within Unity
[04:53] catalog, which gives me a lot of benefits. Um, specifically, um, I have this hosted within, uh, a model serving endpoint. Um, it's versioned. It could see version 15. We'll get into a little bit of that later. And then I have all
[05:07] of the, uh, the signature and the parameters associated with the model as well. All of this is registered in Unity Catalog while I'm leveraging the opening SDK and, uh, GPT5.5. In addition, I've given this agent
[05:19] access to a set of Unity catalog functions. Uh the benefit of this is uh what I like to think of it is instead of taking chances with probabilism, I'm falling back to determinism when I can
[05:31] uh hardcode a function that the agent could have access to. Um so in this case, all uh 5.5 has to do is recognize the semantic intent of the user and identify it needs to call this
[05:43] registered function. So now my uh agent has access to this function but just giving it access to to um only hard-coded functions themselves limits the capabilities of the agent. So in
[05:55] addition to that I've given it access to a genie space which is sitting on top of the raw data itself. So this genie space is sitting on the raw uh statcast data. It has much more flexibility in what it can answer but there's more risk associated with that because it is a
[06:08] probabilistic query inherently. So going back to the the original query, you can see in this top right corner the first part of the question, what does the pitcher do in 22 counts? That's
[06:20] resolved by my Unity catalog functions because I have an explicit deterministic function ready to answer that question. The spin rate is not available via via Unity catalog functions. So it falls back to the genie space to answer that
[06:32] part of the question. This is a great way to balance that determinism and probabilism and making the most out of both OpenAI and uh data bricks as open AI as 5.5 is responsible for putting together this full output of uh not just
[06:46] the raw data but also how the batter should approach the the situation and a recommendation on how to um actually attack the the pitcher as well as actually citing where the data was retrieved from. So this is an example of
[07:00] using both data bricks and openAI to their core competencies. Use the governed data from data bicks registered and version models in Unity catalog and then use OpenAI for its intelligence layer.
[07:16] Okay. So going back to the presentation for a little bit here. The next thing I want to talk about so that's sort of just like the baseline. This is how we get started. The next thing is like how do I create a continuously optimizing loop leveraging
[07:29] codeex with this agent that's hosted on data bricks. So in general we're moving into a different development cycle for agents. Uh model capability is changing very very quickly and the harness around the
[07:41] model has to keep up. And by harness I mean the instruction tools data access runtime deployment rules and something I'm very opinionated about the eval themselves are all part of the harness.
[07:53] uh a rule that helped an older model might actually restrict or limit a newer one. U an evaluation that one cost the important failure mode may stop measuring the behavior that users actually care about. But at the same time, the most improvement uh signals
[08:06] are already being captured um in the form of actual usage from your end users. So the feedback, the actual usage on the application, that's the stuff that really matters and we need to to to leverage that as much as we possibly can
[08:20] um to improve our agent. Sorry, one sec. There we go. Uh so to execute this improvement loop with Codeex, um everything starts with
[08:33] feedback traces and evaluations uh derived from actual real usage. So in the demo I'm about to go through, Codeex reads evidence alongside the current agent and decides which part of the harness should change and tests that
[08:45] next version. Data bicks provides the governed data, MLflow evidence and deployment controls. We as the developers are responsible for defining what good looks like and approving important changes. Codeex is responsible
[08:58] for reasoning through the improvement loop and orchestrating the process. And data bricks is where all the data lives. So I'm going to show you a version of this loop running in uh in a in another demo here. So let me switch screens.
[09:13] So the way I'm showing this, so going back to the beginning, we looked at that model versioning. I had uh uh version 15. Um so I've been running this improvement loop to where every single day I get a new candidate agent automatically created for me. So what I
[09:26] want to show here is what the V1 of the agent looked like and then what the V15 of the agent looks like and how it made that transition. So I'll show I'll start in sort of the same way. I'll kick off an example of the query and we'll come back to this at the end and show the
[09:39] direct comparison. So this is my codeex automation that I run. This kicks off a daily optimization um job where I essentially recreate my candidate architecture for the agent
[09:51] itself. I personally have written um some policies, some contracts that the agent has to or has to navigate across, but it's actually operating on a direct markdown file it receives with with
[10:05] tactical recommendations on how to improve the agent. I'm creating that via this process called Halo, which is called hierarchical agent loop optimization. And the objective of Halo is to spit out a markdown file with tactical
[10:18] recommendations. And when these recommendations are not just restricted because I think historically when we think of optimizations we often talk about prompt optimizations but given how complex harnesses are getting that's increasingly not sufficient. So this is
[10:31] a truly a harness optimization where we look at all aspects of a of the harness and ways that we can improve them. So this is the actual markdown file that gets generated by this codeex orchestrated halo um halo job. So you
[10:45] can see I have three very very specific recommendations. One being to change the way that the runtime control flow processes different tools. Another being to update genie and uni catalog tools in a specific way. Um and a third to
[10:59] actually create a completely separate genie space to test out SQL queries. So these are very very far outside the typical promptbased optimizations. These are pretty fundamental changes to my system and by passing these to codeex it can actually execute these changes.
[11:14] So in addition to those top three recommendations, I also get uh a recommended list and you can see this is across a lot of different categories. That second column there, some of them are eval, some of them are genie space changes, some of them are observability
[11:27] changes. So every single day my codeex um automation gets this file presented to it with tactical recommendations on how to improve the the next version of the agent.
[11:39] So then when that job executes um it kicks off uh Codeex is now responsible for creating the next version of my agent using that Halo markdown file as tactical recommendations and then it runs a uh evaluation job on a held out
[11:53] data set to see whether the actual candidate agent is is is better than the original U baseline agent. So this is what I see every day. Every day I get um uh around 10 in the morning I get a a report out that shows me what my new
[12:06] candidate uh how it performed on a set of judges that I've curated whether it's better than the original baseline model and then if it's better uh I'm going you know full send here and deploying immediately to production but you can
[12:19] also add in human gates to review u but I give the agent autonomy to determine whether this new candidate version is better based on the criteria that I've defined. So, going back to the original
[12:31] comparison um of the V1 agent versus the V15 agent, uh you could uh as I go through this, you'll see that the it's pretty small, but the um the new agents about 40 seconds faster. Uh the answer
[12:44] itself is significantly more specific, better data, way less tool calls, and uh it cites the actual data it uses. So, it's it's it's a significantly better response in essentially every single
[12:57] category. And all of this was from me not really doing anything. It's taking in traces, observing failure modes, and automatically generating a new version of the agent.
[13:10] So I think like this, if I can have you all take one thing away from this, it's this loop engineering concept that's orchestrated by codeex. So we the most valuable information that we get, the scarce resource at this point is not intelligence. It's actually like
[13:22] expertise and domain usage feedback. That is the part that is really really challenging to get. Let's actually take that feedback and extend it as far as we can where we take the feedback that we get, we take the failures that we get v
[13:34] uh via observed traces and we use that alone to optimize our system in an automated fashion. At this point, the optimization step, it's not something that we need to be as manually involved in. We can outsource that to an agent to
[13:47] execute as long as we can define the right criteria. That's really where we fit into the step. So in this case each each of the optimizations that my agent creates on each new version is based on actual evidence it observes as part of
[13:59] this process. So this is the full output of this process over time. So this is the first 15. Um when I put this presentation together I'm on like V35 now. Um so the
[14:12] original agent the baseline agent was like uh 2.5 on my my quality score essentially and it's up to 4.2. An interesting part of this is I keep saturating the eval. I had to redo the eval set because the scores got too high
[14:24] and I had to start over because uh it was the the questions were too simple for where the agent got to capability wise. And on the other side, the latency is significantly better. So like long story short, the new version of the agent is way way better and it's not from anything uh I'm really manually
[14:37] doing. Um it's just purely from just continuing to optimize that process and and providing feedback uh and letting codeex handle the automation. So if I go back to the presentation.
[14:50] All right. Uh I I did want to talk about like what I kind of learned going through this process. Um because when I first started you you inherently want to have more control over the process. So I created a bunch of more deterministic fixed rules
[15:03] of what I wanted uh codeex to do in deploying a new agent. And I kept ending up with really annoying failure modes where it would make some sort of uh what I deemed to be illogical decision of what to do next. Um, and then when I
[15:16] tried to figure out why, it would say, "Well, you created this rule that told me this is what I should do." I was like, "Oh, yeah, that was my bad. Sorry about that." Um, but that kept happening over and over and over again. So, what I kind of realized over time is that by confining it to those really, really
[15:28] strict deterministic rules and determining what to do next, it's limiting the capability of the agent in like a very meta way because it's kind of what I started uh saying about my agent at the beginning. Um, in this case, what what really accelerated this
[15:40] process was treating myself as the responsible for providing all of the evidence, providing the verification criteria, providing like guidance, providing the right evidence and then letting the agent
[15:52] reason itself and then make the right choices from that. Giving it too many fixed rules just creates strange failure modes that um really limits the agent capabilities. So my responsibility is to define what good looks like. the agent's
[16:05] responsibility is to actually execute and make reasoned decisions based on the evidence that I present and my criteria for what good is. So this was like a really interesting journey for me um to to go across to kind of see to the point
[16:17] where now I'm using codeex to say like why did this happen? This this seems like a a unnecessary failure mode and then we kind of work it out together of how to how to make the that uh failure not happen the the next time around.
[16:36] So if there's one thing you can remember from this, I would say it's this loop engineering concept orchestrated by codeex. Um and so to reiterate what we covered here, every day I just wake up with a new candidate agent and I get all these automated improvements for free. Um where the only thing I require is
[16:50] people actually using my agent. Um and then defining what good actually looks like. Uh and this is all again orchestrated within data bricks. So codeex is the orchestrator and all of the agent construction new agent
[17:03] versions that's all happening in data bricks. So we need a repeatable system that turns usage feedback uh traces and evals into tested changes across the whole system. Humans remain responsible for the direction uh but we and we
[17:16] curate the feedback um and that really dictates how the changes happen but we actually leverage codeex for the automation side. So what I you know went through today is just one example of how OpenAI and data
[17:29] bricks uh can work together um and how to take advantage of both of the best parts of each platform. The govern data the uh versioning the unity catalog components of data bricks combined with
[17:41] the intelligence of open AI models and codecs can drive this type of automation uh and really accelerate your agentic adoption both internally within an organization and then building things out for your customers.
[17:57] So, uh, thanks so much for the time. Um, I'd love to chat afterwards here if you have any questions. Uh, and we have a booth kind of down by the middle if you haven't seen it already. We're giving away $100 of codeex credits. So, um, you
[18:09] definitely definitely take advantage of that. Uh, it's right near the data bricks middle thing. So, um, right out there. Uh, and yeah, thanks so much for the time today. Appreciate it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.