Observability's Agentic Moment: Operating AI in Production with Genie Code

Summary

This video introduces a unified operational observability layer for AI in production on the Databricks Data and AI platform, spanning Model Serving, AI Search, AI Gateway, Agent Bricks, and Apps.
Genie Code goes beyond alerting to deliver instant root-cause analysis and agentic operations on demand—including automated model rollback without any manual configuration or infrastructure changes.
Observability is broken into four dimensions—operational, cost, quality, and governance—and this video focuses on the operational layer, covering metrics, events, and cross-request lineage across multi-step agent pipelines.

Observability's Agentic Moment: Operating AI in Production with Genie Code

Shipping AI products is easy. Trusting and operationalizing them at production scale is the hard part. Operational observability is now having its agentic moment. Join us to see how Databricks is reinventing operational observability for the agentic era — a unified, production-scale telemetry layer powered by Genie Code, our AI assistant, spanning every AI offering on the platform: Model Serving, AI Search, AI Gateway, Agent Bricks and Apps. Genie Code doesn't just alert you that something's wrong — it delivers instant root-cause analysis, pinpoints fixes for new error patterns and runs agentic operations on demand. Walk away knowing how to keep your application running smoothly in production — and how to get to the bottom of any issue in seconds when it doesn't.
Talk By: Anshul Gupta, Sr. Staff Software Engineer, Databricks ; Mingyu Li, Staff Software Engineer, databricks ;

Chapters

FAQs

What does operational observability mean for AI workloads on Databricks?

Operational observability answers whether a model or application is running correctly once it is in production, covering metrics, deployment events, error patterns, and root-cause analysis. On the Databricks Data and AI platform, this is distinct from cost, quality, and governance observability, each of which addresses different aspects of production AI management.

How does Genie Code help with root-cause analysis for production AI failures?

Genie Code analyzes logs, metrics, and deployment events to pinpoint the cause of production failures and surface actionable fixes in seconds rather than requiring engineers to manually correlate telemetry. It can also execute agentic operations on demand, such as triggering a model rollback without requiring manual configuration.

What AI products does the Databricks observability layer cover?

The unified observability layer covers all major AI products on the Databricks Data and AI platform including Model Serving, AI Search, AI Gateway, Agent Bricks, and Apps. This provides a single telemetry layer rather than requiring separate monitoring setups for each product.

What is cross-request lineage in multi-step agent pipelines?

Cross-request lineage tracks the full chain of calls across a multi-step agent pipeline, allowing engineers to see how an initial request flowed through each stage of the system. This is critical for debugging production issues in agentic workflows where a failure in one step may propagate through subsequent steps in non-obvious ways.

Full transcript

[00:09] Thank you all for joining us and hope you're having a very good summit so far. Um just to introduce ourselves, I'm Anul Gupta. I'm one of the engineering leads of the data bricks AI or I look into a lot of classic ML and agentic solutions
[00:22] that a lot of our customers use. >> Yeah, my name is Mu Lee. I'm a uh tech lead of the AI gateway and custom model serving team work on observability uh and custom model serving related
[00:36] projects. Right. All right. So today we are here to talk about observability. Um, we had a lot of talks obviously related to this. If you guys attended the keynote yesterday, we also talked
[00:50] about Genie Zero Ops which enables you to automate your post-p production activities. So, it's an extension of that. Uh, observability is what happens
[01:02] after your model, your app goes into production. And we'll cover a little bit today on how data bricks is thinking about solving this problem. what you can try tomorrow. Okay, so before I go into
[01:17] the details, I quickly wanted to give a little bit of brief because observability sometimes means different things to different users. Um we classify observability in four major
[01:30] categories. Operational, cost, quality and governance. So if you want to know hey is your model, your app running well once it's in production that's typically operational observability. Cost is the
[01:42] demographics and how much you're paying. Quality, are you getting the value out of it? Yes, it's running well. Yes, it's paying me it's costing me so and so. But am I getting the quality that I want from that model? And finally, how is the
[01:55] governance the audit logs looking like? Is it secure for me to run this in production? You have had a lot of talks hopefully some of you attended on the cost quality and governance area. We covered a lot in in the keynote as well.
[02:09] Today we will primarily focus on the operational side but I will show towards the end on how all of this come together and how are we looking at it together as well in data bricks.
[02:26] Okay, this remote doesn't work and sometimes it works too quickly. I'm just going to use the keyboard maybe. Okay.
[02:38] All right. So how many of you have are sort of some sort of engineers who have debugged a problem um by raise of hands through any part of their career? Okay, pretty much everyone. So you know when we have debugged the problem in the
[02:51] pre-agentic world, it needs you to do like a very complex ontology based on the kind of issue it is. You have your foundation of of observability. You need your metrics. You need your logs traces.
[03:04] And once you have all of that, you play a game of okay, you know, you change tabs, look at metrics, look at logs, try to find how you correlate them. And that's how you debug an issue that might be going on in production. For the easy
[03:17] ones, it takes you five minutes. For the complex one, sometimes it takes hours and days to even figure those things out. Right? So this is obviously super hard. It is time consuming and that is
[03:30] the problem that doesn't make sense anymore. Why? I think the key reason is agents can actually close the loop. Today our agents have gotten so good that they can figure out these problems
[03:43] if you give them all the data, all the context, metrics, logs, traces. It can actually figure out and come to the root cause much faster than we can and much easier than we can. Right? And why is it
[03:55] so important for a IML? Primarily because AI and ML are sprawling. So you are going to have more and more AI and ML applications models in your production. And when that happens, you
[04:09] need engineers to handle how to mature the models or how to mature your applications and not actually doing the operation spot. That's how you want your engineers to get the best value out of
[04:21] their time. Those are the three reasons how the our industry has changed in the last one year. Why agentic observability for a IML is becoming increasingly important
[04:37] and that I think is a transition that has already started. So how do we solve agent observability just like any other agent intelligence?
[04:49] I think you heard Ali saying in the keynote probably AGI is already here. Intelligence of the model is not a problem anymore. The context is the problem. If you really want your agents
[05:02] to perform well in observability cases, you need to give them the right context and they will give you the right solution. So obs agentic observability is a context problem and that's where we feel data bricks comes in pretty well
[05:16] because when it comes to observability it's not just your metrics and logs that matter for a real complete debugging you don't figure out your bugs because you have
[05:29] access to metrics and logs you also figure it out because you have access to your code what exactly is running in production and data bricks is the only vendor there where you have access to your data. It's living in there. You
[05:41] have access to models and apps running on top of that and the entire telemetry that data database provides on top of that. Databris has access to everything. And that's why our agentic observability
[05:55] will be able to figure out root causes issues much faster than let's say a tool which only has your observability telemetry and it's trying to play around with metrics and logs trying to find your issue. Maybe that will hint at the
[06:08] issue but to actually fix the issue to actually go deep down you need all the context and we have all the context. Not only that, we have a very powerful genie code layer. If you have used it,
[06:21] you know that it is extremely powerful in understanding a IML especially. That's 13 years of data bricks experience folded into Genie code which is not just the model but a lot of skills a lot of things that we have done
[06:33] on top of that to make Genie code understand your models and a IML extremely well and hence we think data bricks is extremely well placed to solve the agentic observability problem for anything that runs on data bricks
[06:51] there are a lot of customers today who use us in this regard I've coded one example on ModMed they actually run really large number of highly scalable applications in production which are super missionritical and they completely
[07:03] rely on our observability on making sure their operations in production are going well and as they do that they have seen how our observability has matured over the past six to six months to one year as the
[07:16] models have matured and as Genie code has matured and as our solutions are getting better and better and this This is our mission. If you have heard the keynote, our mission is to get to the point where operations is
[07:30] not your problem. It's our problem. You just have to maintain, monitor, and that's about it. But zero ops is our goal. And today I'll give you a brief idea on how we are thinking about
[07:42] approaching that problem. And what we have today, we want our agents to not only understand your problem, understand the issues that are coming in, but also monitor it, alert you, and heal itself.
[07:58] So how are we making it possible? We think about it uh or I'm going to explain about it in three different places uh three different phases. We start with how observability is centralized in datab bricks ecosystem.
[08:10] How it is all managed on top of unity catalog and you will see how we are actually using genie code as a single place where you can utilize all
[08:22] your telemetry and data to help you debug answers. Next, this is still a place where it's making it much easier for you, but you still have to go and debug. Our next
[08:34] step is where zerops really comes in. We want observability to be completely proactive. You don't really have to go to the UI to look at a metric to check whether your system is doing well. You need to sleep well at night and you need
[08:48] to be alerted and you don't have to worry about setting up those alerts, tuning those alerts. None. I think our goal is that once you have a system, you have out of the box monitoring system, you get alerted and in some cases
[09:01] autofixed as well when you're running observability on data bricks platform. And finally, we'll also see this happens not just at a single product level. Data bricks obviously
[09:13] provides a whole system for AI and ML starting from data to the models to agents and applications. So it has the entire suite of products and our
[09:25] observability extends across all of these products. So you can see lineage across all these products. You are able to use Genie code to debug problem not just for your model serving endpoint or your search endpoint but across your
[09:38] entire uh workflow of your production application. So let's start with the foundations though first. Um if you have used any of our AI products whether it's apps
[09:52] serving agents you have seen that all of them are built on these foundations of operational observability. We provide out of the box metrics for pretty much all our products. We provide logs for
[10:05] any custom code products that includes your deployment logs that includes your server logs for all the custom code that you have written. uh we provide traces backed by MLflow tracing for both agents
[10:18] and for any traces that you export out of your custom code and inference tables to record the payload logs for every single request that comes in into data bricks and events for your CRUD operations. So together these are our
[10:31] foundations and this has existed and being uh built upon for the past many many years in fact
[10:45] and Genie code literally sits on all of these. Every single one of these now resides in Unity catalog. All of this data actually goes into our tables and Genie code runs on that. So when you are on an endpoint page, your Genie code
[10:58] actually knows what endpoint it is, what are the metrics and logs and data associated with it and it can help you investigate uh any issues that come onto your endpoint and Mingo will help you
[11:10] show how it is actually in production today. Um yes uh now imagine I'm a data scientist and I'm responsible for a fraud detection model that will score um
[11:23] that will score new uh bank account applications and detect fraud actions. So this endpoint is in production uh model serving and from this endpoint I
[11:38] can go to this matrix page and I see there's a small amount of 4xx errors. So although it's not a total crash but for fraud detection model so every field request matters. So I want
[11:51] to figure out so what happened is it because of a new deployment is it because the model behavior or a bad client. So to start I can click on this ask genie button.
[12:03] So it will trigger the genie code to start an investigation on the endpoint based on the foundation on data bricks. And while it's running I can briefly
[12:15] show the foundation that just talked about. So we have deployment logs. It will show what happened to the endpoint and when. We have system logs, service logs that
[12:28] will show the logs of the running model. We have build logs that will show uh how the container is built and what happened to the environment
[12:41] from model. We also have inference table, trace logs, uh persistent system logs. They are all governed in uni in unity
[12:53] catalog and they are for the runtime logs for this endpoint. We also have lineage of the model. So from the model version
[13:10] it's also in unity catalog and I can see the upstream and downstream of this model. So which where is this model deployed to which endpoint and which which notebook actually train and
[13:22] register this model?
[13:39] >> and I think when this is running if you notice carefully the genie code was actually reading the notebook. It was actually reading the metrics and logs and it was showing hey I'm reading the endpoint metrics logs and the and the notebook itself and that's how it's able
[13:51] to figure it out. There are times it actually queries the inference table using SQL as well uh to figure out in that time duration what are the key things that happen inside that inference. Yes, this is running production and here
[14:06] genie code has figured out the yeah it went through the matrix inference table uh figure out the root cause and went to the source notebook and just the next step for me to take to mitigate this
[14:19] issue. That's awesome. What you saw right now was only for model serving endpoint, but this is all
[14:31] available in production. Not just for model serving, we have it for search gateway apps and a lot of our products and you can try it out today. Not only that, if you have been using
[14:44] some of these products, we have a lot of launches that happened either in the past month or basically launched today at DAS. I covered some of them uh which are some of our primary launches. Inference tables used to take a little
[14:58] bit of while for the inference to happen and when it shows in UC. Now inference tables are instant. Within a few seconds of the inference, you will see an entry in UC. Not only that, sometimes it can
[15:10] be costly to have all the data and you don't really need it. We have a sampling fraction that allows you to sample how many percentage of inferences should actually go and persist into UC tables. We have a lot of out of the box metrics
[15:23] much more finer grained in our model surveying and all the other areas of AI as well. You go to a supervisor agent, knowledge agent, you go to AI gateway, it has extremely powerful fine grain
[15:35] metrics. We not only that we have persistence across update of endpoints which used to be a pain point before. We have persistence of all the logs that are in our servers when you run your
[15:48] custom code whether it's in training apps or serving. These are some of the releases we did. You can also export your own metrics from your code or your logs traces which
[16:00] we don't capture out of the box and all of them will be stored in UC and hotel format. So we have made a lot of changes in our foundations to make sure you have any and all telemetry that is needed for
[16:13] an agent to perform the debugging and get better answers for any issue that happens in production.
[16:25] And how is it run running today? Today these are the amount of numbers that we have across all our customers. 9,000 plus customers use our observability platform. We have four nights of availability on our observability stack. We have more
[16:38] than 170 million time series that runs through every day across all our customers. 20 plus terabytes of data, 5 million plus traces and these numbers are constantly growing. And Genie code
[16:50] has become a way where people are actually querying these data points. That 400k number is although all the Genie code queries, we don't have a distinction of what is for observability versus not.
[17:06] All right. So this is what you all have today. But this for us is still scratching the surface. It's just the beginning of what we want to deliver. The next comes is if you notice in that demo, you still had to go to the endpoint page. You still had to click a
[17:19] button to figure out what's wrong with your error rate. That is not where we want observability to be. We want observability to be proactive where agent does all the work for you and you
[17:31] are fi taking the last final steps that's needed. So you saw the figure here. We are complementing what we had so far with
[17:43] alerts so that you get alerted when things is wrong when things are wrong instead of having to go to a page to figure it out. Not only that, Genie code is powered by skills. We automatically
[17:56] provide a lot of skills so that Genie code is extremely good at understanding AI and ML but you can do that as well. You know your model, you know your organization, the jarens that goes on in
[18:09] the code. You can add your own custom skills which are also governed in Unity catalog and those skills will also be used by Genieode in its investigation of any issues.
[18:21] And at some point once you have all of this it will be doing investigations on those alerts so that even before you come in you can actually know exactly what Genie code has done and you don't
[18:34] get alert you get an alert with a complete report you can set policies that can either auto accept the alert or require you to approve the action
[18:47] that is the place where we want to go to and that is what Mingu is going to demo right now on what it will look like very soon.
[19:04] Yes. So, here I have my email about an issue going on on my endpoint.
[19:17] So this is a different survey endpoint for bank account fraud detection and it shows our future vision of how the operational observability will look like for custom order serving on based on
[19:29] genie code. So on this page we have the elements that just talked about. We have the watchers, we have the skills and we have action policies.
[19:42] So for the watchers, it's not only an an inbox of the alerts. It actually is a very comprehensive workflow that will detect issues based on inference tables,
[19:56] metrics, deployment events, and even periodic workflow checks. and it will also investigate the issue um alert the users and based on the
[20:11] policy we'll do auto mitigation. So here we have four alerts of we have four watchers. Uh this trigger this request s watcher is in active state
[20:25] because there are some 5xx errors. This request latency watcher is also active because there's spike in the P99 latency. And here for this first single
[20:39] client error spike it's very important here because it has a green check mark. So this issue is triggered and investigated and mitigated automatically by gening code based on the pol policy
[20:52] set to resolve it automatically. Okay, let's go into one specific alert here
[21:04] on this event details page at the top of the page we have the summary of this alert. So when there's a issue triggered when when there's an incident so the most important thing to know is what
[21:17] happened what's the root cause how can I mitigate it so that's what we have here um and after that we also have the detailed steps of
[21:31] what genie code did what logs did it look look at what investigation step did it use to figure out the root loss and just the mitigation step.
[21:48] And because this event has all the context, you don't need to start from scratch every time to to ask questions or if you want to get more learn more about what happened in this incident,
[22:00] you can directly ask question from this event. So for example, I know I need to resolve this issue. there's 5xx error spike. Um, and the stress action is to
[22:13] roll back this model version to a previous model version. But I also want to know what's the impact of the roll back. So I click on that. Genie code has all the context. So it will go into the
[22:27] model version details and give me the comparison of these two models. So the fallback model policy has a slightly worse quality but during an incident the
[22:40] right trade-off is usually to uh mitigate the issue first to remove the production errors. So our start canary roll back to the fallback policy version
[22:53] and gen code will start to roll back start a canary and roll out to 5%. make sure that everything is healthy and they continue to roll back roll it back
[23:07] to 100%. And when I come back to this page, this issue is mitigated.
[23:24] So I can also add new watches. It's all backed by Genie code and a genic workflow. I can I can add a new alert say alert me when P50 latency exceed 5 seconds. Then
[23:36] it will automatically run a dry run to tell me how many alerts will be triggered during 24 hours. If I add the alert. So if I decide everything look good and
[23:49] I want to add the alert, I can click install.
[24:03] Skills are very important because all the genie code depend on the skills to investigate the watcher triggered events and potentially also add mitigations. So you can add
[24:17] predefined skills from your team. Those skills are all in UC or you can add your own skills from from plant text. And for action policies,
[24:29] that's where you want to add the guardrails. For each watcher trigger the event, do we want to automate based on some actions? For example, for
[24:42] 4xx errors, I probably want to allow auto roll back to previous version. And for a single client error, I probably want to allow auto li auto limit a
[24:55] single client. But from for both of these I don't want to delete the endpoint or I don't want to convert the endpoint to uh um to enable scale to
[25:07] zero or other functionalities. Yes, because data bricks has all the foundation data and has all the context, it's possible for us to add this agentic
[25:21] workflow and watchers to automated issues and free people's time from doing this work.
[25:39] So what you just saw is that agent is doing the ops work. You are doing the setup defining your business constraints. Hey, I want this to be my SLA. I want this to be my latency budget. Those are the things you're defining. You're giving it skills for to
[25:51] be able to debug better and with higher quality. And then you're leaving the job to Genie to Genie code. But what you saw was still limited to a single product. What we showed was for
[26:04] model serving endpoint. When you as a persona are actually working, you're working with multiple endpoints. Not only that, you're working with multiple different products.
[26:16] Perhaps you're using serving, but maybe you're also using a search endpoint to go with it. You're probably putting that in all in an agent which is working in an app. You want observability and across your entire ecosystem and again
[26:30] data bricks is in a great shape to actually make that happen. This is what we saw so far but Geniezero ops when it extends to all the products it will have a single view and you can
[26:43] do things like cross request lineage across all your products. If I have an agent which is running in an app which actually calls a model serving endpoint or calls an AI search endpoints whose
[26:55] embedding model is another model serving endpoint. These are all common setups. Today our debugging is at every different layer. But with data bricks when you have it on our platform you can do cross focus lineage. You can
[27:08] understand how your request is flowing. You can understand how the latency is in each aspect. But more importantly, Genie Code and the agent can understand all of that to make your life much much easier.
[27:29] So going back to what I was referring in the first slide, we primarily talked about operational and the foundations that are required to make operational function agentic. But I think the same applies to cost, quality and governance.
[27:41] Where do they all live? Cause primarily lives in system tables which is again unity catalog data. Quality lives for agents in agent tracing which again is mlflow tracing data. Our evaluation
[27:54] results live in UC. Governance is basically UC. All our audit logs, system logs, they all live in UC. So every part of the observability platform that you're looking for, the data actually
[28:06] lives in UC. And that's how even though we today we're showing across operational observability, we want Genie Ops to be able to handle this across all four kinds of observability. You can not only go to Genie code to ask alerts on
[28:20] it's working or not. You can set cost budgets. You can say that my endpoint should not go beyond this. My endpoint's quality if it goes below this. You can set alerts for that. That is our vision for Genius Zerops.
[28:42] And that's where we end. That's how we think observability's agentic moment is right here. And data bricks is best positioned to handle that. Thank you. Thanks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.