Agent Ops on Databricks, Powered by Omnigent

Summary

This video presents a practical AgentOps playbook for taking AI agents from working demos to safe, reliable production deployments on the Databricks Data and AI platform, addressing the gap between the 97% of organizations deploying agents and the 12% achieving production scale.
Core AgentOps principles include distributed tracing for observability of silent production failures, evaluation flywheels that validate every agent code change, and declarative YAML specs that treat prompts, tools, and routing logic as versionable code.
The Omnigent multi-agent orchestration framework is demonstrated for managing sub-agents and switching models, with CI/CD pipelines running automated test gates across dev, staging, and production environments.

Agent Ops on Databricks, Powered by Omnigent

The journey from traditional machine learning models to AI systems demands a new, specialized set of operational practices. This talk traces the evolution of core DevOps principles from the decade of MLOps to the Agents era, where we must plan for deploying and maintaining autonomous, multi-step AI agents that can reason, plan, and take actions in the real world using external tools.
The talk will show how classic DevOps tenets like continuous deployment must be radically re-engineered to tackle core AgentOps challenges.
Authors of The Databricks Big Book of Agent Ops will cover:
Ensuring Maturity of Operational Controls: How critical operational controls must evolve to support multi-agent architectures.
Distributed Tracing and Observability: Transforming single-step I/O logs into traces essential for debugging and explaining agent behavior.
Governed Deployment and Rollback: Establishing AgentOps CI/CD pipelines that treat configurations (prompts, tools, routing logic) as code
Talk By: Jeanne Choo, Staff AI Forward-deployed Engineer, Databricks ; Kyra Wulffert, AI Solutions Architect, Databricks ;

Chapters

FAQs

What is AgentOps and how does it differ from MLOps?

AgentOps extends the DevOps and MLOps control loop to autonomous multi-step AI agents that reason, plan, and take actions using external tools. Unlike MLOps, which focused on model versioning and serving, AgentOps must also govern distributed tracing across multi-agent calls, manage configuration changes to prompts and routing logic, and run evaluations that validate non-deterministic agent behavior.

Why are only 12% of organizations running AI agents at production scale?

Industry surveys cited in this video show that while 97% of executives report deploying AI agents, only 12% reach a scalable production stage, and only 52% are running evaluations. The gap is framed as an operations problem rather than a model quality problem, which is the central thesis of the AgentOps framework.

What is the evaluation flywheel and why is it important for agents?

The evaluation flywheel is a CI/CD practice where every agent code change—including changes to prompts, tools, and routing logic—triggers an automated evaluation run before deployment. This gate ensures that configuration changes, which are as impactful as code changes for agents, are validated before reaching production.

How does Omnigent support multi-agent development on Databricks?

Omnigent provides declarative YAML specifications for defining agent architecture, sub-agent codification, and orchestration logic. It enables teams to switch models and tools by changing a config rather than rewriting code, and supports production auditing by sampling live traces for compliance review.

Full transcript

[00:08] So, thank you for joining the session. And today we are going to talk about Agent Ops on Databricks, and we will give you a practical playbook for taking AI agents from like working demos to a solution that you can deploy in production in a safe,
[00:21] reliable, and scalable manner. And we will be illustrating a each Agent Ops principle by a live demo. So, who we are?
[00:33] >> Yeah. Um my name is Jean Chew. I work on our AI forward deployed engineering team, and I help customers deploy Databricks
[00:45] in their environments according to our best practices. >> And I'm Kida Bulfer, and I'm a specialist solution architect working in data science, ML, and AI agents
[00:57] nowadays. And well, let's get started. So, some statistics. So, we have a based on industry surveys, 97% of executives say
[01:10] that they are deploying AI agents, yet only 12% are getting into that production scalable stage. And on the production operation side, we have a around 89%
[01:24] saying that they are like covering observability, but just 52% are running evaluations. So, the gap between teams building agents and trusting them is quite significant. And the thesis is that it's
[01:38] not because of the model quality, because models are getting really good nowadays, but because this is an operations issue. So, um the idea is that Agent Ops helps us
[01:51] address these issues and reduce that operational gap. So, in this session we will be covering those core challenges that teams are now experiencing when operationalizing
[02:04] agents. And also, we will then like match set of challenges with principles and then see that agent ops principles in in life without demo.
[02:21] So, let's start with a how do we get to today? So, historically, we have a DevOps classical software engineering uh where we have a this set of practices that minimizes the the lead time from
[02:34] writing code to productionizing it and reducing also the time to recover a system in case something breaks. And it is based like in versioning, CI/CD, um
[02:46] automatic test gates, and also continuous monitoring. And this control loop we apply to machine learning is part of MLOps and applied to agents is part of agent ops.
[03:01] So, most of us have lived the MLOps life cycle where we have like this intersection between three principles, data ops for like that data quality and reproducibility,
[03:13] model ops for the model versioning, tracking, and serving, and then DevOps for CI/CD and automation. And we have this classical cycle where you have data engineer preparing the data, the
[03:27] data scientist training and evaluating the models, and then the ML engineer taking that, packaging it, and and serving it. And then, there is a continuous monitoring for drift, and then retrain in the loop
[03:42] back again. And in this model, the the model is the central artifact. So, we are training a model, but we are evaluating a we are monitoring a model. And agent agentic systems, they stretch
[03:56] this because um in in agentic systems, we have a although we have the same loop of build, test, deploy, and monitor, we have
[04:09] different components. It's not just a model, and we have also a new persona, which is the software engineer. And um this is because um
[04:21] agents are not just models, they are systems, they require orchestration, they require tool calling, memory, retriever, reasoning, and that line between model engineering and software
[04:33] engineering is now blurred. So, um we can see also like that data preparation is now like context retrieval, evaluation is now not just like metrics, classic metrics as we had
[04:45] before, but we have now AI judges, and we have also uh subject matter experts in the loop to evaluate the system. And also uh the agents, they are making like real actions, so we need to trace and be
[05:00] um able to observe each of the steps that are involved in that decision-making, and also to ensure that there are guardrails in order to define what can and cannot do the agent.
[05:12] So, we went from having a model to building, testing, deploying, and monitoring an automat autonomous system that is built around one model. So, it's
[05:25] quite complex now, and this complexity creates a lot of challenges. And this is the the next part of the our session. So, what are these core challenges that we are experiencing and are uh the main
[05:37] cause of this 12% of just like having agents in production from all the ones that we are deploying and creating at the moment. So, one of the main challenges is a
[05:49] control fail safe and governance. And here is that a traditional gateway is not enough to control what an agent is doing. Maybe you can control access to the model, but not once the agent is running each and
[06:02] every step of it. Um so, what we need here is a per action control plane on top of the tool calling so that we can deny, allow, or ask for permission
[06:15] before making an action. And also here the keyword is to prevent, not to mitigate, because for instance, if you have a prompt injection uh trying to convince your agent to exfiltrate
[06:28] data, you want to block it before it happens, not just to see in a dashboard that it happened the day after. Um and so, this is like a this is a runtime policy. Then, a next challenge we observe is a
[06:41] reliability and evaluation gaps. So, when we have like a single uh model call, um we can like see an error it was or not,
[06:53] but in the agentic systems, we have a multi-steps change uh where you are calling several tools, and this can uh and in compound errors. And these compound errors can have also
[07:05] uh be uh silent errors. So, that means you get an answer from the agent that looks really confident with a good format, but is completely wrong. And this is very difficult to identify by a
[07:18] vibe check. So, vibe checks, they are good like to have that feeling, uh might look productive, but they are not a good way to um test agents that will go to production.
[07:36] And and that missing of the measurements in your in your project is one of the reasons why many uh of these AI projects stall. Another um ch- challenge we observe is observability and feedback. So, as we
[07:48] were mentioning before, uh with an agent we need to ensure that we can trace every tool call, every handoff to the sub agents. And besides tracing that, we need also to track the
[08:01] feedback. So, if an user an user thumbs down an answer, or if a SME expert marks like, "Okay, this is I'm flagging this flag this answer because it's incorrect." Or if an AI judge is just a
[08:13] scoring low in a certain metric, we need to track that into the trace so that we have a single object where we have what the object what the agent is doing, and also how others are seeing that that
[08:26] action if it is good or bad, and why. So, so that we can close the loop. And maintenance is a challenge that can be quite expensive. And why? Because um
[08:41] production agents, they have usually very complex requirements, like a complex UI, lots of safety requirements, durability, you need to have the ability to access enterprise tools, to be
[08:53] flexible to change models. And um what usually happens is that teams are building from a scratch that pipeline to support the agent per use case. So, there is no
[09:05] reusability, and there is a lot of redundant effort, uh which makes that every single project is very expensive in time and also in resources.
[09:19] And another very important challenge is that the agent, your agent, has to be business context aware. So, that means that the agent uh knows about your business processes, uh knows your data catalog, is aware where
[09:32] the good numbers are, speaks your um the the language of your of your brand. And this is like a huge differentiator because this is very unique of your company. And this is what makes
[09:46] the agent really resonates with the end users. Otherwise, you can have an off-the-shelf agent that is technically correct, but it's not providing the answers you need for your company.
[09:59] And all of these were technical challenges, but there is a human challenge which is like very tough to handle, which is like a stakeholder alignment. Why? Because production agents are super interesting. Everyone wants to take a
[10:12] part of it. So, when you have a a room full of a um stakeholders with different concerns, so we have like the CEO concerned about return of investment, the data team
[10:24] about concerned about access, a compliance team about risk, okay, this is risk or safe. Uh the product manager pushing for new features, the AI engineer looking for how can we iterate
[10:37] faster to product product and ship uh faster. Uh the app developer like very interested in having a stable user interface, and the end user having a
[10:49] good agent that works. So, lots of people with different priorities, and this is is very important that it is actively managed since the beginning, otherwise we will have here uh a stalled project
[11:02] due to human um and not due to technical issues. So, if we take these challenges and we don't address them, they can compound into
[11:16] technical debt. So, if we don't have governance, then um you will have a a system with lots of tools, tools sprawl, agent sprawl, many things doing exactly the same, nobody
[11:28] knows what is in there. Uh Uh, if we don't have a observability, then you will have uh, obscure pipelines. You won't be able to debug. And if you don't have evaluations, then you will just have like a subjective,
[11:40] "Oh, I think it is working." which can be very problematic, uh, between the stakeholders. So, uh, another important point here is because we have these very, um,
[11:52] complex systems, uh, non-deterministic behavior. Um, this, um, having this kind of technical debt doesn't have a linear cost. So, the cost grows even more if you have them.
[12:06] And so, these were the challenges. So, what does good look like? So, the, um, the idea is to have like a safety controls and also to have a genuine team
[12:18] alignment in your solution. And for here, for this, we saw of a, a very nice model where we have like the Formula 1 team that is working at three tempos.
[12:30] So, a per lap tempo, which is like very fast iterations, very tactical. A per race tempo, where you have, uh, the strategical, uh, needs for, for a run.
[12:42] Um, it could be like evaluation uh, from production. And a per season, uh, loop where you have a long-term vision of your, of your project. And, uh, so, how do we translate this into a concrete
[12:55] processes that engineers can follow? And here is Jim. >> Yes. So, now we move on to a solutioning phase. And for every phase here, from laps to
[13:09] races to seasons, has specific technical requirements that we need to be addressed by any framework we use. Such a framework needs to be composable and
[13:21] use many pre-built components so that teams can ship across the organization without needing to reinvent the wheel. So, labs you can think of as a development loop where we open a PR with
[13:36] a new feature that we want to roll out into production. This is on a shorter time scale, maybe now especially with agents on the day of 1 to 2 days. Races
[13:48] can be thought of as a production deployment which we might do every day if you're really good, but realistically maybe also a release where that happens on a per week basis. And this is where
[14:01] we need a strong evaluation flywheel to measure that we are not breaking anything with our feature and also that we do not break anything by everybody contributing
[14:14] tens of features to the same main branch, which is the reality that we now face. Seasons can refer to longer time scales like a sprint planning cadence or
[14:26] monthly stakeholder check-ins where we re-evaluate our production priorities and our strategic direction. So, let's look at each stage one by one.
[14:41] On a per race basis, we want fast responsive changes where we can rapidly roll out a new feature with strong governance controls and then test that that feature actually delivers the ROI
[14:56] that we want. So, let's illustrate this with a case study. And the case study here we're going to use as a Telco company who has a customer facing support agent.
[15:08] And traditionally we normally think of one agentic workflow per use case. And maybe even a few months ago we would have been talking about how do we
[15:20] productionize a multi-agent reference architecture for this customer service app? But, this is no longer the status quo anymore. As
[15:32] Manthey mentioned in his keynote, now agents touch every aspect of the org. So, we not only have our customer-facing multi-agent
[15:44] support agent, we also in the background have the data scientists on our team who is using a custom model with custom traces to analyze our production logs
[15:56] and identify failures that we haven't seen before. We also have the software engineer on our team who may be using a custom coding agent with specific PR
[16:08] guidelines that the organization needs to follow. And so, now we're talking about multiple agentic systems within one single use case. And crucially, all
[16:22] these different agentic systems need to work well together in order for us to have an actually continuously improving flywheel. And this is where Omni agent comes in.
[16:35] Omni agent brings together these multiple agentic solutions using common standards. So, you could have your custom agent and the custom agent is
[16:47] then specified using a declarative spec, which we call your agent YAML file. And this can be both internal and external facing. So, you might have an internal coding agent or trace analyzer
[17:02] agent that you customize with this config. You also have our customer-facing support app that is also managed with this declarative spec. You can also combine this with built-in
[17:15] agents that we offer such as Debbie and Polly. Everything goes into the single Omni agent server, where the server is where you manage your conversation
[17:27] sessions, your policy stores, and any artifacts like the slides that people upload or their Google Docs. And everything is within this Omni agent Omni agent server.
[17:40] Finally, we have the end-user apps. In this case, this is our customer-facing support app, and also the out-of-the-box Omni agent web UI that we use to serve
[17:54] both our customers and our internal team. So, by putting everything together into this one platform, this allows us to address the key challenges that Kira
[18:06] mentioned before. For maintenance, we have easy composability because now you can customize your support agent by switching in different tools, different MCPs just by changing
[18:19] your agent YAML spec. You can also change your models and harnesses, and switch things in and then out easily. We also have our runtime policies for governance, and for team
[18:32] alignment, we can easily collaborate by sharing conversations between team members. This is not only useful for a PR review, but also useful when, for
[18:44] example, as an engineering team, you need to hand over to the team that writes your technical documentation for a specific feature. You can rapidly then transfer a
[18:56] conversation over to the technical docs team. They can start that conversation, but then start asking, "How do I best craft technical documents for this new feature?" And it is that
[19:10] handover cross-team process that we find the most useful. So, let's look a little deeper into what this declarative spec actually is. To use our customer support agent as an
[19:24] example, this is quite a complex agent. We have sub-agents that request specific queries for billing, for account queries, for tech support. We also have
[19:36] function tools, MCP tools, and policies that we want to specify for this agent. So, this is an example of a YAML, and you can see here we have functions.
[19:49] Sub-agents are actually also codified in the same way. So, you don't have to deal with a complicated declarative spec. Everything is nested like this.
[20:01] What's even more convenient is that all your agent YAML specs can be translated by Databricks asset bundles to be deployed into your development, staging,
[20:14] and production workspaces. These resources then appear in Databricks Unity Catalog as governed objects for you to set access permissions on.
[20:26] This is another example of how we have security policies defined, and I'll go a little bit more now into what this looks like from a customer-facing perspective.
[20:54] Okay. So, let's start off with a simple query. Someone asked, "What plan am I on?" And you can see here that we've routed this query to the account agent, and the account sub-agent has used one tool to
[21:06] gather information for the customer to tell the customer that he or she is on an unlimited data plan. We can also ask about account balances and outages. Again, we've routed to a
[21:21] billing sub agent who then calls two tools. But what if I have an internet outage and I want to raise an escalation? This is where we can set a guardrail on
[21:34] the agent who wants to create a Jira ticket. We can ask the user to verify that they really want to escalate this use case before we actually approve this tool.
[21:52] Subsequently, we also have a refund request. And you can see that we also go through the same approval gate because this refund request actually touches an important service.
[22:07] Finally, we also want to guard against things like PII detection. And instead of every single team needing to implement a PII detector, which I've seen before, everybody has a different custom judge or rule-based system or
[22:21] they use a third-party service to do PII detection and then you you have different solutions, we have one PII solution and this is shared across
[22:34] all the agents across all use cases. And in this case, when a customer asks about a credit card policy and our agent actually tries to query some sensitive data, this is automatically flagged, so
[22:49] we mitigate this data exfiltration and the customer is asked to go back to the secure account portal instead. So this is how we We actually use one
[23:02] declarative spec even on a complicated multi-agent architecture to then switch in different tools, different MCPs, and different policies. So, we can mix and match and
[23:15] try different things rapidly. So, now when we move on to the per-race phase, what really matters are strong evaluation capabilities because we want to make sure that we
[23:28] don't break anything in production. And also, we need to make sure that we are successfully uncovering unseen failure modes, which is particularly
[23:41] hard to automatically detect. And great evals are not only about mitigating failures in production. They actually address many other concerns that
[23:53] enterprises are facing now with their agent systems, namely around governance, context awareness, and alignment. It's only with the proper LLM judges and
[24:06] coding function tests that you can see that you are able to trust your agent outputs. One thing that I've seen, for example, is that you don't want your LLM generating somebody's account policy.
[24:20] You need to check that the LLM actually calls the account policy tool in order to deliver a trusted result. The other context awareness is that your
[24:32] audience can be very specific. And so then we need to ensure that user preferences are properly respected. And finally, we need to make sure that
[24:44] your business goals are properly reflected in the agent's outputs. For example, you may need to make sure that you are actually delivering a delightful customer experience that goes above and
[24:58] beyond just giving a technically correct answer. So, MLflow has been focusing on this a lot. It does have many layers of
[25:10] feedback loops from programmatic checks to LLM judges, but the key question here is not only how do we instrument our app. The key question is how do we
[25:23] instrument our app and apply the same checks across development, CI/CD, and production. And this is where we actually, again, are able to use a
[25:37] declarative spec to centralize all our emails that we want to run across environments.
[25:51] So, that what we have eventually is that we can deploy our tests in development, but then after that, we can hook up this same staging
[26:04] test whenever someone pushes a PR, and you can see here we run our eval suite, which is then transferred to a Databricks workspace,
[26:16] and then we can see that we are running certain checks here. All of them seem to pass, but eventually maybe I want to double-check something.
[26:29] So, then if I want to double-check something, I can see that these checks have actually been saved to an MLflow experiment. And in this MLflow experiment, I have a specific question that I'm interested
[26:42] in. I need to be able to understand trade-ins versus private sales. And yeah, it seems to actually have passed. I have a brand compliance judge, a data privacy judge, and a response
[26:56] clarity judge. So, I can see from the overall pipeline that the tests are passing, and I can actually audit the judge outputs just for like some sanity
[27:08] check. So, now if everything looks good, I can trigger a production deployment.
[27:23] And then once I have a production deployment, um help. Once I have a production deployment, then
[27:35] in production things get slightly trickier, where you're thinking about two scenarios, both an online and an offline scenario. The online scenario is where you want your pre-program checks
[27:48] to run reliably using a scheduled monitoring job. But then you also have your offline deeper trace analysis, where you might actually go through
[28:00] certain traces manually, and you want to surface undiscovered failures. So, online monitoring of judges are not something we see many people doing, but
[28:14] it has been discussed um quite a few times already. So, in a sense it's not new. And for this one, we have a Databricks notebook,
[28:26] and we have the same judges that we've coded before, our brand compliance judge, our data privacy judge, and a response clarity judge that we schedule
[28:38] to run in production on a subset of our traces, so that we can audit that. So far, everything that we know could be a possible failure has actually been
[28:51] accounted for. But then we move on to the offline analysis portion. And if I go into my production traces, I see that okay, I'm getting
[29:05] thousands, tens of thousands of traces a day. I can't look at everything manually. If I do an eyeball, I think I see a failure here that we haven't coded
[29:18] for. Someone is asking about data roaming alerts, but we've never actually explored what tools we need to properly address such a question. So, I can see from a manual inspection
[29:32] that there seem to be some failures that we haven't accounted for, but how do I do this at scale? And this is where we see people like data scientists or product managers or software engineers
[29:46] start to use their LLMs to do large-scale trace analysis.
[29:58] So, if we go back to our original diagram here, we see that this is where we come up against the challenge of we need this multiple agentic systems to work together well to engineer a proper
[30:11] flywheel. And what we have here is a data scientist is using Claude code with custom ML flow skills to analyze their traces.
[30:24] So, this is not working, but this is an example of the Omnigen web UI, and the
[30:40] idea here is that I have Claude code configured or I can use code X or I can use also a custom agent that I have as an agent.yaml.
[30:53] And then I can configure traces to access the same workspace as production. And then I can ask my agent to use these
[31:05] custom skills to identify traces where we have not accounted for failures in CICD. And this is as easy as two prompts. So, the first
[31:17] prompt is, "What skills do you have?" And I see skills for analyzing traces. I see skills for retrieving traces, for computing specific metrics. And after
[31:30] that, I can send a prompt saying, "What kind of failures are we seeing in production that are not currently covered in CICD or development?" And because this Claude has access to both
[31:45] dev, staging, and prod, as well as our code base, it can look through the test that we've coded in CICD and dev, compare that to production traces, and
[31:57] give us a breakdown of what is not covered in production. And this is how eventually we are able to build a custom flywheel where a data
[32:10] scientist uses the same Omnigen web server to spin up a UI, look at what's happening with the customer support app, identify feature gaps, and then pass
[32:24] this on to a data scientist who can then use a custom coding agent to fill in this feature gap for us.
[32:41] >> And You got so I think um with gene demos uh we saw like for all these core challenges, we were the technical ones. We were explaining before, we had coverage with the Omnigen
[32:54] and Emma flow in order to ensure governability governance, reliability, uh observability, maintenance, and so forth. So, now going to that human challenge,
[33:08] um that's part of this per season phase, where um we have the organizational layer in order to ensure that the quality can scale across use cases. Because maybe
[33:21] you're successful in one use case, but if you don't take that into the others, then you have to restart again from the beginning. So, very quickly, uh there are like two things that are
[33:33] very important. One is stakeholder alignment. For this one, is a recommendation is to really ensure like map who is who, what are the priorities,
[33:45] uh have regular communication with them, understand um educate them because there are many expectation about agents or what can the agent do and what cannot, establish success metrics with each of
[33:59] them. And then this communication we move from educational to reporting on those metrics. And that creates like a very healthy um organizational habit for
[34:11] your use cases. And then in terms of the team, it's very important to map who is doing what. So, a very useful mapping is with um RACI, which is like say who is
[34:23] responsible, who is accountable, uh who has been consulted and informed for the different task. And this is something that needs to be revised periodically because as your project advances then this might
[34:36] change but it's very important so do have documented everything that is happening and it becomes like a automated part of your system rather than something that you are enforcing in meetings.
[34:48] And just bringing everything back so what we saw here today can be mapped into the three ways of DevOps which are flow which is like build and ship that will be like our per lab phase
[35:01] then the feedback way is where we are capturing the evaluation the feedback that will be the per race phase we saw and then the continuous learning is our long-term phase the per season where
[35:15] teams and systems are improving over time. Thank you very much for joining our session today. We have been writing a book the big book of agent ops where you can find all this
[35:29] content and more and it will be available as an early release after summit. Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.