High-Throughput, Low-Latency: The Databricks Playbook for Production Model Serving

Summary

This video demonstrates how Databricks Model Serving delivers high-QPS production ML workloads — including a live demo scaling to 791,000 requests per second — on a unified platform that replaces the complex DIY serving stacks teams previously had to build and maintain.
The serving architecture includes 15-second autoscaling with vertical optimization, a declarative feature store with sub-150 millisecond streaming feature freshness, and end-to-end observability with metrics, logs, payload capture, and lineage through Unity Catalog.
The video previews a zero-ops ML agent for autonomous model management and introduces AI-native operations with Genie for managing endpoints and monitoring production model health without manual intervention.

High-Throughput, Low-Latency: The Databricks Playbook for Production Model Serving

Modern AI applications don’t just need accurate models, they need to serve them at massive scale, with low latency and strict reliability guarantees. In this video, we’ll go under the hood of Databricks Model Serving to show how we support high QPS (300K+ QPS), production-grade workloads on a unified Lakehouse platform. We’ll walk through the serving architecture, autoscaling strategies, and optimizations across GPU/CPU utilization, request routing, and caching that enable sustained high throughput without sacrificing latency or cost. You’ll see real patterns from customer deployments, learn how to design your endpoints for bursty and always-on traffic, and leave with practical guidance for running mission-critical ML and LLM workloads on Databricks Model Serving.
Talk By: Anshul Gupta, Sr. Staff Software Engineer, Databricks ; Mike Del Balso, Director of Product Management, Databricks ;

Chapters

FAQs

What is wrong with a DIY production ML serving stack?

A typical self-built real-time ML stack requires separate components for event streaming, streaming transformations, an online feature store, a model serving system, a load balancer, and observability tooling — each requiring dedicated engineering and maintenance. This video argues that maintaining all these components is expensive and complex, and shows how Databricks Model Serving replaces the entire stack on a single unified platform.

How fast is Databricks Model Serving autoscaling?

Databricks Model Serving can autoscale in approximately 15 seconds, with vertical optimization to right-size compute as traffic changes. This video includes a live demo demonstrating the system scaling to 791,000 QPS to illustrate the platform's capacity for both bursty and always-on high-throughput workloads.

What are declarative feature views in Databricks and how do they work?

Declarative feature views let teams define feature engineering logic in a specification-driven way, with the platform automatically managing computation and serving infrastructure. This video shows streaming features served with under 150 milliseconds of freshness, enabling real-time ML applications such as fraud detection.

What is a zero-ops ML agent in Databricks?

A zero-ops ML agent is an autonomous AI system that manages deployed ML models — monitoring health, detecting drift, and taking corrective actions — without requiring manual engineering intervention. This video previews this capability as part of the Databricks roadmap for AI-native operations on the Databricks Data and AI platform.

Full transcript

[00:09] Uh thanks everybody for joining today. Uh I'm Mike. I'm director of product management here. I work on uh machine learning products and data science products. Uh an >> I'm anul I'm one of the senior tails in the AI or I work on serving and agentic
[00:21] products and data bricks >> and we work together on a lot of um uh all of the infrastructure and products here to help people run uh missionritical machine learning applications in production. And so think
[00:35] of things that are high QPS uh things that are very important for your business. This could be a like a fraud detection application. And it could be uh real- time quoting application. It
[00:47] could be a real-time product recommener. You know, things that like ultimately affect your end customer experience and uh and are not the same as like these back office machine learning applications where you know you're bulk
[01:00] scoring um churn rates or something or predicted churn for your customers where if it kind of fails, you can kind of just like press retry and run it again. We're talking about stuff that's like live. It's hot. It's in the part of the
[01:12] customer experience. So, this talk
[01:25] uh uh and get to get them to production and that it takes a separate stack to run. Um, and also if you love giving your scientists a clear path to ship without you having to be in charge of
[01:39] maintaining all of these systems. So just tell me understand this group here uh because this is a little bit of a smaller group than the keynote this morning. Uh how many people here like run machine learning in production?
[01:53] Okay, cool. All right, excellent. That's helpful. Um,
[02:05] on the slide. I'm sorry. Uh, this is intended to represent uh an architecture of, you know, a typical like DIY uh real time production uh machine
[02:18] learning uh stack, right? So starting from I hope you can see my cursor too but um starting from the left there's all the data things we got to have right there's a Kafka cluster that's managing
[02:30] our events uh that's streaming our events we're ingesting there there's uh something to do some streaming transformations to turn those into features we have uh an online feature
[02:43] store um and then kind of going more in the middle we have a model serving system we have our model registry There's a load balancer in there. There's something to log all of this stuff. And on the right side, we have
[02:55] all of our observability stuff, our logs, uh different like different things to help us manage knowledge as well. So, there's a lot of stuff. And uh and it can be like kind of unwieldy and it's
[03:08] not uncommon for uh teams to have uh for for companies to have like huge machine learning teams to like maintain all of their custom infrastructure, right? To
[03:20] to run their models in production. I know some of the folks in this crowd and I'm familiar with the stacks that you've run today or have run in the past and have migrated away from. And it's it can be really expensive. It can be really slow. It can hold your teams back. It
[03:33] can hold back innovation. And uh it kind and it kind of sucks, but that's what we had to do in the past. Um building it yourself has stopped making sense. And so this is not this is not intended to
[03:45] be a a data bricks specific uh pitch, but uh it just doesn't make sense to DIY for most use cases anymore at all. Uh it makes a lot more sense to just buy buy some solution. And I'll explain why
[03:59] because better solutions exist. It's really like the people who know like you even understand all of the infrastructure that your team's putting together. Those people can work now on you can put them to to work on much
[04:11] higher ROI things for your business because you can kind of just like buy really good solutions for all of this stuff now and the stakes have changed, right? So, uh in the past we were using machine learning for frankly like less
[04:23] important things for the business. But as machine learning has matured and as our organizations have matured and we've begun um
[04:35] like advanced decisioning in different parts of the business, we've begun to rely on it more and use it in more critical parts of our business. So something that may have been um like a huristic driven in the past or maybe
[04:48] like a human-tuned algorithm in the past, you might just have a learned algorithm and it's making uh a decision in real time there. And so uh so your business depends on that and your teams have better things to do.
[05:01] You've you know this whole conference we've been talking about agents and and LLM and all that kind of stuff. Your company's also trying to solve those problems too. and uh you know they don't want to be solving the problems of the
[05:14] past. So one claim that we have in this talk is that production machine learning while it used to one make sense to build it yourself but also it made sense to
[05:26] have it as a separate stack that no longer makes sense. We really see production machine learning is just another workload that uh the data platform itself runs right it's just another thing that fits in the data
[05:38] platform and you know this is how data bricks thinks of uh the production machine learning stack everything that's involved in a live machine learning application and we'll walk through these
[05:50] components but just to show you it's one system you've probably seen many of these boxes before in different data bricks talks and stuff like that but it's basically just one unified stack to run your real-time ML
[06:03] and and we make this possible because we built for mission critical at scale right so high reliability but also being able to scale to the workloads the QPS that your uh real world applications
[06:16] need and we've built it all in the same substrate so you know these applications are not just uh you know model serving but you have a lot of things going on you have monitoring and feature
[06:28] management, feature store, all of that stuff is part of the ML application. We brought it all together into one system. And finally, uh if you saw one of the keynotes, the keynote that the talk that I gave this morning, it we really focus
[06:41] a lot on real real ML teams have to do a lot of operations to manage these systems. And so we're moving a lot more towards AI native operations as well. And so we think like all of this together uh really just makes an
[06:55] extremely compelling uh argument for just using the one unified stack for production ML and we'd like to show you what that is today and then also you know if we have time after just get your thoughts on uh you know
[07:09] how it compares to your needs and be happy to talk to you about this stuff to to understand um you know get some guidance on where we can go from here. So I'm gonna hand it off to Anchul to talk us through uh mission critical at scale and we're going to show you some
[07:22] demos along the way so you can see what you can do with the platform. Thank you. >> Just a show of hands how many uh of you have actually used data bricks model serving. Okay, good number. So you have seen probably UX um I think data bricks model
[07:35] serving was actually the first real-time product in data bricks back in I think 2020 2021. Uh and today it serves many many mission critical customers. Uh, Grammarly is my popular example which is
[07:47] publicly coded. We actually serve Grammarly for all their real-time needs, mission critical needs and they serve us at 200,000 QPS. Um, and the key advantages lies beyond just serving as
[07:59] well. Like the platform gives us a lot of benefits. So a lot of customers who serve with us actually train with us as well. And one of the things that's we focus a lot on is ease of deployment from training to serving. So it for
[08:13] those who has run medical machine learning in production they have seen that moving from training to serving is pretty complicated. You have to make sure the runtime environments are same etc etc. In data bricks it's a single
[08:25] click uh because you're running in data bricks we know exactly what you're running and a single click actually matches your runtime and make sure that everything is just working. We are extremely latency sensitive. This whole
[08:38] real-time infrastructure is built with minimal number of hops. so that you know we can have really less latency and today we have models uh which actually run less than 20 15 milliseconds. So
[08:50] I'll quickly show how that looks like. So this is our uh this is our MLflow registry. >> I think we got to zoom in a little bit. >> Yeah. So we this is Unity catalog and it
[09:04] shows all the models and you have trained the model and then you registered in MLflow. So once you do that all you have to do is click this button surf this model.
[09:18] You enter a few things like your name of the endpoint. Uh the model is already there. You select the hardware type. Uh we have a bunch of hardware types from CPU to GPU.
[09:31] This is a simple XG boost model. So I'll select CPU. You have the optionality of scale to zero. So you select minimum and maximum concurrency which are your floors for how much minimum number of concurrency that you want to support and
[09:44] max sort of handles cost controls. And once you do that you also have some advanced options which are more around observability. What kind of observability you want enabled and you just click create here
[10:03] and that's it. Within a few minutes, your endpoint will be ready to actually serve traffic. And not only that, this is how the endpoint page will look like. It can actually show a lot of metrics out of the box for your traffic like latency,
[10:16] response rate, throughput, CPU usage, uh all of those things. It gives you your event log for all the deployments that has happened. It gives you logs like your service logs, whatever is running
[10:29] in your code because these are still custom models that you have trained, deployment logs and you can see all of this available in UC. So when I open UC, your logs, traces, metrics, payload,
[10:42] they're all in a single place. So you get this out of the box as soon as you have created an endpoint. So I just created this very simple notebook for this endpoint. All it does is takes the endpoint, just generates my
[10:54] authentication and does a quick curl post call. Uh if I run this, this is actually hitting the endpoint that we just created and it's able to return a response on within 12 milliseconds.
[11:09] So that's how easy it is to move from training to serving. And this is something that we have optimized a lot since the beginning of the product. That was always our motivation that this this
[11:21] journey is extremely simple which was never done before in machine learning. Before this if you had to do all of this you had to actually not just match your
[11:35] runtimes you have to support your Kubernetes clusters uh deploy your API or the autoscaler config manage those clusters then you have to build observability by hand. So all of that is
[11:47] bundled into this managed offering of model serving. So like I showed not only do you get a lot of observability this data is the endpoint the models are completely
[12:00] governed in UC. So you get the entire power of UC governance. You can share it with different people. Uh we provide scale to zero and complete autoscaling. So if your endpoint is not in use, you're not paying for it. It's completely pay for what you use. And it
[12:14] is versioned. So whenever you update your endpoint, it happens with zero downtime. You can move from one model version to other model version by just creating pods and it switches traffic when the new pods are ready. Not only that, if you want to do testing like you
[12:27] know I want 10% of my traffic to go to uh new model and 15% 90% to my old version just to make sure your new version is matching well. Any sort of those kind of testing is also possible because a single endpoint uh that is
[12:40] exposed to the user can actually support multiple models behind it. any kind of models. So scale of this endpoint has grown a lot. Today our model sewing product
[12:52] supports north of 250,000 QPS uh at very low latency and it has 49s of reliability. We actually have Grammarly as the biggest customer who supports
[13:05] 200,000 in production. So I'm going to do a little bit of brave thing today and actually try to demo this live with 250,000 QPS. So wish me luck.
[13:20] So the example that I have is a Super Bowl. Super Bowl of course the event that we all know about. One of our customers actually run their Super Bowl uh models on us. And when Super Bowl is actually happening, the amount of QPS that you can see actually grows a lot.
[13:32] So I have an endpoint already running and what I'll do is basically I'll say hey I want to generate 250 QPS. So behind the scenes, my difficult challenge here was not the endpoint. Generating that much traffic or
[13:45] simulating that much traffic was also hard. So I actually had eight eight different compute machines sending traffic to the same endpoint. So I've kicked it off. It does take a few minutes because it scales up to the
[13:58] traffic. So we'll keep coming back to see how it's doing and hopefully it will get us to 250,000 or more. This is a minimum QPS. But I actually want to show how we make
[14:10] this possible. So this is the key part of our architecture uh for model serving u the center part that's where I want you to focus right now on the serving pod. So serving pod is our main engine. This is what we horizontally scale and
[14:24] all the data actually all all the requests go to the serving pod and this is where we run your model your custom code. It has multiple different parts. The main engine actually has the MLflow server, VLM, Triton. Based on which
[14:37] model you're running, we automatically figure out which runtime is best for you. It has hardware which you have selected at your model time. We support a lot of GPU types and you know a bunch of CPU types as well which are different
[14:49] memory, different cores. All of this is transparent to you. You don't have to manage any of this. All you do is send the requests and then we do a lot of observability exporting from within this part. So but the key part of this is the
[15:01] autopilot itself. So we have a patented technology on autopilot which does not only horizontal scaling based on requests. So we are able to respond to request spike within 15 seconds. Every 15 seconds we actually scale based on
[15:14] the incoming traffic which is a live metric. But the other thing that we do here is we also vertically scale our pod and this is not necessarily hardware scaling. We actually make sure that there are some models which can only run
[15:27] maybe four QPS per node but there are some models which can run 25 QPS per node. We don't know that beforehand because it's a custom model. But what we do is dynamically figure out the resourcing of your model when the
[15:41] traffic is coming. We learn from that. We adapt to that and our autoscaling actually changes how much a single node can handle for every model. So it adapts to your model. We have actually a blog published just on this autoscaler that I
[15:53] encourage reading. Uh every other vendor or when you do DIY, you have to manage all of this yourself for every single model, but we do it dynamically within the ecosystem.
[16:06] So let's see where we reached. We at 87K. So we'll come back. So that's the autoscaling part. It's not it's immediate on the request. It adapts to your model. Like I said, we are
[16:18] extremely latency first architecture. What is not shown here is the number of hops that we take to come here is very minimal. All you have to do is o rate limiting as soon as you enter the data bricks realm. And we have worm pools to
[16:30] make sure that when you actually autoscale the code latency is really low. Uh all our pods when they scale up it's multi-AZ placement so that even when one of your zones go out you actually have
[16:42] full 49 of reliability. So let's see where we are. We'll wait for a little bit more until it keeps going up. >> Yeah, it takes in steps because it sees
[16:54] the traffic go straight seven then sees then goes to the next one. Fanatics is the customer uh who basically ran Super Bowl on us and actually met a lot of this QPS with really low magnitude and
[17:07] you know very low latency as well. So they were very happy with how their event went. So we'll continue noticing this. Oh, there you go. It actually went far beyond. I love it. I said
[17:19] >> 791. >> Uhoh. >> Yeah,
[17:32] check the metrics page of this model. This was the model. Let's actually refresh to see even this shows that. And there you go. It shows around 791,000 kps. >> Wow.
[17:44] >> We weren't was not planning on that. Cool.
[17:57] So I just created white coded that graph.
[18:10] to highlight along the way. Uh so, model serving is a big component of uh production ML system
[18:23] store company called Tecton and uh we recently joined data bricks. Uh the big thing with feature store is that it's just as needed if you're building if you're running real time production
[18:35] applications. By the way, how many people here have something that they call a feature store or anything like that in their organizations? Wow. Okay. So 40% um so so uh you know the
[18:50] production machine learning system is not just a model serving system. It's uh you know models need your data. It could be the data about what the customer just did. It could be uh it could be data
[19:03] about the customer's background or the market or whatever it is. These are the features that need to go into the model as well. And so it goes it goes handinhand with the model serving system to create this uh single stack.
[19:17] And it's historically really hard to build uh uh feature features, feature stores. Uh we are we've had a feature store at uh data bricks for some time. Um, one of the things we've been doing
[19:30] over the past couple of months is uh building in a lot of capabilities that we built at Tecton, my feature store company into data bricks, right? And so some of the things that um that are launching uh in the
[19:43] next week on data bricks are a new framework to define features. So we now support uh super simple declarative features which uh can allow you to define features really simply and uh
[19:57] these can be like time window aggregations. You know all kinds of different um operations on your existing data and simply and get them to be uh very easily turned into production data pipelines that that can be whose data
[20:11] can be retrieved in real time and passed to uh a model. So I want to show you this way. Thought we got session expired. Okay. This is Anchul's machine. So it's
[20:24] g. Okay. So super. Okay. I'll tell you what I was going to show. So uh there's a there's a new um uh API that we have which we call uh declarative feature
[20:37] views. And that allows you to to pretty simply define a feature, what it is, what data it depends on, what its meaning is, what the transformation is that should happen to your raw data to turn it into that uh final signal. And
[20:51] it's really just like a handful of lines of code. So you don't have to define this whole complicated um data pipeline. And when you once you define these features, all of the operations you would want to have uh with a feature um
[21:04] are done are handled for you automatically. So things like um backfilling that backfilling the values so you can create training data from this signal right so you can create training data from this feature things like automatically
[21:17] uh computing this feature uh on an ongoing basis so ongoing materialization um a very important one is syncing that data into some online store so it can be
[21:29] retrieved in real time for inference because all this feature data you both need to use to train a model in the first place but then part of The real-time application of course is inference. So you need to serve it at inference time and so that's all handled
[21:41] automatically. Uh what happened? It was just it wasn't logging in. >> Yeah. >> Okay. So can I show that? >> Yeah. >> Okay. Sorry. Um uh and so the types of
[21:54] features that we support uh obviously are uh starting with just batch features. So you can define kind of any type of uh batch job, spark job and that will become uh a feature that's a first
[22:06] class entity managed in uh unity catalog in data bricks. But uh also this week we're putting into public preview second thing I wanted to demo for you but I guess it's not working. Uh uh
[22:20] which is streaming features. So features that update in real time and this is actually extremely powerful. So um why do we want real-time features in the first place? Why do we even use real-time recommendation or real-time predictions, right? We use real-time
[22:33] predictions to get a to get uh a signal very quickly, but if we're not updating the data that goes into the model really quickly, then there's not like a ton of value in getting a real time prediction,
[22:45] right? So we needed to have access to to fresh data as well.
[22:57] supported on uh data bricks. And what that allows is for you to use that exact same SDK which I hope to show you in a second um to define features but now you can define features feature
[23:11] transformations on Kafka data on event on event data. So, uh, you you're able to have your features represent the real-time activity of your customers.
[23:23] So, when your customer clicks into a certain category, a certain vertical on your website, you can have your recommendations like updated in real time. And this is the key to unlocking real time um, uh, really like reactive
[23:37] applications. And historically in data bricks, you've been able to build um, uh, streaming pipelines. Uh but data bricks just launched this new real time this new streaming engine
[23:49] called RTM real time real time mode and that's used to power this these streaming features. So that allows instead of uh it taking a couple of seconds of delay for from an event to
[24:03] your features and your predictions reflecting that event. Now we can do that in less than most recently what we just measured in our most re recent benchmarks 150 milliseconds. So from uh the time when a
[24:18] customer you know clicks onto your web page with some new information all of your recommendations or all of your decisioning can be updated 150 milliseconds later. So it's really powerful and it's extremely powerful
[24:30] particularly for fraud detection. Um, so yeah, real-time mode, uh, right? And it's directly integrated with Lakebase, the online store, uh, and it's always
[24:42] fresh, right? Uh, and so it's super important for fraud detection. That's like the the the top use case for this kind of stuff. And this this is not just like hypothetical, right? Coinbase is one of uh the customers that uses this
[24:56] and they run you can imagine like fraud detection at scale uh extremely missionritical for their business and it's built on the engine that I'm talking about today. So if you have real
[25:09] time decisions you have decisions that you think uh would benefit would be improved by a better understanding of how your customer what your customer's very recent behavior was. We now have a
[25:22] really easy way for you to uh integrate that data into your models and into into those decisions. So you think I shouldn't was it like it wasn't logging in or something? >> Not this one, but we can try the next.
[25:35] >> Okay. Okay. We'll go to the >> I lost access to the notebook. >> Oh, you lost.
[25:47] the way, there's an integration between the feature store and the model serving system that Anel just showed you.
[26:02] we'll automatically set up the feature store for you. And you can instead of having to interact separately with a feature store, give me the features and then let me go make a prediction, let me let me call the model server to get a prediction. Uh data bricks will handle
[26:16] all of that stuff for you. So you just say give me a prediction and it will know to talk to the feature store and and retrieve all the features and it'll hand handle all of those handoffs for you. So it's really simple. It's just a single request.
[26:30] Okay. So that brings me to part three which is AI native operations. So we built all this cool stuff. We have uh models uh that can make uh can do inference in real time. we have uh it all connected to our real- time feature
[26:42] store which is connected to all of the real-time data sources in the business. So now we're able to make reactive applications uh that are predictive based on our customers behavior. Uh but
[26:54] we need to keep the system running and uh there's a lot of work involved in keeping this system running. So there's a couple things that uh are important to do. A lot of things can break in machine learning models. Um, one of them is, uh,
[27:09] data. There's all kinds of data drift, train serve, skew, all kinds of things could go wrong with, um, any of the metrics in the models, even operational metrics. And so we've been hard at work over the last uh many months to build in
[27:23] a lot of AI capabilities native into every part of our machine learning stack so that you can use AI agents to help you uh debug any errors that come up along the way or even find if there are
[27:36] problems. Right? So, what? So, we're short on time. So, I'm going to show you um I'm going to show you uh one thing here.
[27:49] What was the one that was most instructive to show on show? Was it this one? I see. Okay. So, okay. So, look. So, I want to figure out some of these metrics, right? This is So, this is the uh metrics page on my serving endpoint.
[28:02] And so, I can see, you know, latency. I can see response uh rate, error rates, CPU percentage, all of these different things that give me a sense of the health of my of my model in production.
[28:15] And uh what previously, you know, if something looks kind of weird here, like here I see error rates. I don't I don't know if that's good or bad, but previously I would just have to kind of
[28:28] like look at this chart and make a human judgment of is there a problem here or try to figure out like what went wrong if something looks weird. And so we've added um we've integrated Genie here. So for every one of these charts now you
[28:41] can uh press ask Genie and what it will do is it will it understands the context of this uh of this serving endpoint. It's connected with MLflow. It knows
[28:54] what models involved. It knows the features. It also is connected to all of the logs, the operational logs, and all of the metrics associated with this. And it can help me uh debug and figure out what's going on. And often it will
[29:07] actually do some really deep analysis. It'll run some SQL queries to go read uh all all of the metrics offline. It'll it'll do a variety of different things. Um okay so uh let me show you another thing
[29:21] that we're building in here related to observability. So one all of the inference tables are uh enabled by default and inference tables bring all of uh the predictions we make and all of
[29:35] the information from our model making uh doing inference back to the same govern tables. So uh your teams can read them, your agents can read them. Everything's going through AI gateway. There's I'm
[29:47] sure you've seen some talks about that. You've seen that mentioned uh lately that brings a layer of governance and observability to all of these models. There's monitoring on Delta or anything
[29:59] in iceberg built into the platform. So there's a whole bunch of data quality monitoring you can built in you can you can use built in. And of course there's lineage. So one thing that's really important is if there's a problem with your model, you find something weird
[30:12] with a prediction because there's all the lineage through Unity catalog uh uh in data bricks, you can go from a prediction to figure out, you know, what model made this prediction and which
[30:24] features powered that model at that time and then which data was uh was that feature using. And that
[30:37] which models might be affected or this thing looks weird maybe there's some problem upstream
[30:51] errors right so I just showed you like a metrics graph and and it showed okay you have three you had some level of uh some percentage of your requests are errors
[31:03] and we clicked genie to allow us to to help us like debug the specific metric that was in that chart, which may or may not have been a problem. Maybe it was expected to have uh uh that number of
[31:16] errors that were shown there. But there's sometimes when
[31:28] Is that better for everybody? So, uh this is uh an endpoint. And I see it says, "Hey, uh, endpoint update failed. Your recommener system didn't deploy. The model server crashed. There's some
[31:40] problem with it." Uh, so previously, this is another thing where I would have to go. Okay, let me see. Let me just go click into the logs here. Service logs. All right, let me dig in
[31:52] here. This is starting to get into I have to understand what's actually going on under the hood. And I got to spend a lot of time on this. And I'll go read the build logs as well. And this is not even available right now. So what I can
[32:05] do now is uh press diagnose error. And this also invokes Genie code to help us uh uh diagnose this. And what it's doing,
[32:18] it's actually showing when I clicked this earlier right here above. But what it's doing is it's retrieving all the information about uh about this endpoint. It's going through all of the logs.
[32:30] It's going through the build logs. You can see going through the service logs here. It's
[32:45] Yeah. It will tell me, hey, this is actually why your model was unable to deploy, right? And let's see what it says. >>
[32:57] processes at startup. So it sucks that these type of errors happen in the first place and we want to make these errors go away and autoresolve and avoid these errors even happening. But we're but
[33:11] with these types of capabilities we reduce the amount of knowledge and expertise that a user of these systems has to have for them to be effective with them. So you can just press this button and it'll point you in the right direction. Uh you don't have to, you
[33:24] know, be an expert in in debugging this stuff. So we're really excited about that. And one of the things uh that we announced at the keynote this morning, we
[33:37] that's all about is uh you know here in this demo I showed you me going in and like clicking a button and hey there's a problem and then I'm trying to solve it and I'm taking a bunch of actions. What
[33:50] zero ops for machine learning about is about is it's a model that autonomously runs your machine learning. It's an agent that autonomously runs your ML systems for you. So what that means is it's continuously monitoring all of these metrics. So all the things we just
[34:03] saw, this agent is continuously looking at them and it's identifying when there's problems, saying, "Hey, this thing looks fishy. Let me look deeper into it." And then once it finds something that uh that it identifies
[34:16] actually as a problem, it starts doing a root cause analysis. And then it starts building a solution for and it will it'll run a whole data bricks notebook create a whole it could even build a new feature retrain the
[34:29] model with the new feature with a new training data set and it will bring to you an updated model and it will say hey do you want to deploy this to production. So it brings you all the way to the point of the problem being solved
[34:42] but it still leaves you in the loop for you to press approve once it gives you that uh solution proposal. So, that's something that's coming later this summer. Um, uh, but that's where a lot that's where we intend for a lot of the
[34:54] like AI native operations to go is for them to get more proactive, more autonomous, still giving you the control, but requiring you to do less and less and have less expertise. And this is all
[35:07] part of making machine learning easier and easier for you to scale to the folks in your organization. Okay.
[35:19] Uh I think I also have a talk tomorrow dedicated to observability and AI products called this observability's agentic moment where we'll cover a little bit more auds that you can join. >> Yeah, if you want to dive okay so he's
[35:32] underplaying it. Anchel's giving the talk tomorrow about the uh uh observability based in this whole system and he's the lead architect who's been building all of this. And so if you're interested in diving deep into this uh
[35:46] uh that's the talk to go to tomorrow. What's the talk called and when is it? >> Observatory's agentic moment. >> What time? >> I think around two. >> Okay. >> 150 at the very >> 150. Yeah.
[35:58] >> Around two. >> Thank you.
[36:10] workflow. um and they take advantage of the scalability of the system um take advantage obviously of the reliability. This is their core product that they run on the system and uh you know they they
[36:22] are using this observability all the time. And so the point here is that we want to reduce the situations where you have to like build and maintain and operate all this crazy stack on the
[36:34] left. And uh you can get this pretty clean unified stack uh uh from data bricks which you already have. If you already have data bricks, you already have this and it's really easy. It's
[36:46] just a couple clicks to put your models in production. And uh and if there's a reason why this is not a fit and it wouldn't work for you, we'd love to hear from you. And there's a bunch of things that we weren't able to to hit on today. Uh this
[37:01] model serving system supports GPU serving. So you can support you can uh serve uh deep learning models, LLMs. We have uh VLM, we have uh Triton serving.
[37:14] Um and then one other thing is we now have uh real time or instant inference tables. So uh you know the data from inference used to take some time to actually land back into the governed
[37:27] tables that your your teams or your agents can can use. That's all happening instantly now through the new technology that we launched called zerobus which is real time ingest. So now uh you're able
[37:40] to have we really are focused on agents and your agents can instantly figure out and see the impact of any changes that they're making. Uh so we're really excited about that.
[37:52] Um okay, anything else? That's it. Okay. Thank you everybody. >> Appreciate your time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.