MLflow 3.0: AI and MLOps on Databricks

Summary

MLOps evolution to GenAI: MLflow 3.0 provides first-class support for generative AI apps and agents alongside traditional ML, standardizing workflows from experimentation to production at scale.
Real-time production tracing: Monitor all production agents and models in real time with comprehensive tracing capabilities, regardless of deployment location or architecture.
GenAI evaluation suite: Implement LLM judges, human feedback collection, and sophisticated evaluation datasets to ensure generative AI quality and trustworthiness throughout the lifecycle.

MLflow 3.0: AI and MLOps on Databricks

Talk by Arpit Jasapara, Software Engineer, Databricks ; Corey Zumar, Staff Software Engineer, Databricks
Ready to streamline your ML lifecycle? Join us to explore MLflow 3.0 on Databricks, where we'll show you how to manage everything from experimentation to production with less effort and better results. See how this powerful platform provides comprehensive tracking, evaluation, and deployment capabilities for traditional ML models and cutting-edge generative AI applications. Key takeaways: Track experiments automatically to compare model performance Monitor models throughout their lifecycle across environments Manage deployments with robust versioning and governance Implement proven MLOps workflows across development stages Build and deploy generative AI applications at scale Whether you're an MLOps novice or veteran, you'll walk away with practical techniques to accelerate your ML development and deployment.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How does MLflow 3.0 support generative AI applications?

MLflow 3.0 extends the established MLOps platform with first-class support for GenAI, including real-time tracing, prompt registry for version management, and sophisticated evaluation frameworks.

What is the prompt registry and why is it important?

The prompt registry enables versioning, management, and comparison of prompts across your organization with governance through Unity Catalog, supporting reproducibility and governance.

What MLOps capabilities are essential for managing the entire AI lifecycle?

A comprehensive MLOps platform must support experiment tracking to compare model performance, a model registry for versioning and governance, deployment tools with robust version control, and production monitoring across environments. MLflow 3.0 integrates all these capabilities alongside generative AI support, enabling teams to move from experimentation to production efficiently while maintaining governance through tools like Unity Catalog.

Does MLflow 3.0 provide integrated experiment tracking, model registry, and deployment in a single platform?

Yes, MLflow 3.0 on the Databricks Data and AI platform consolidates the entire AI lifecycle in one system. It combines automated experiment tracking for comparing model versions, a unified model registry with governance through Unity Catalog, production deployment capabilities with versioning, and real-time monitoring across all environments. This integrated approach eliminates the need for multiple disconnected tools and reduces operational complexity.

How does MLflow 3.0 evaluate and improve the quality of generative AI systems?

MLflow 3.0 introduces sophisticated evaluation capabilities specifically designed for generative AI, including LLM judges that automate quality assessment of AI outputs and human feedback collection utilities for domain expert evaluation. These methods address the unique challenge that AI responses are free-form and nuanced, requiring more than simple right-or-wrong metrics. The platform natively integrates these evaluation mechanisms with versioning and monitoring to continuously improve agent quality in production.

Full transcript

`0:00` Introduction to mlflow 3

Right. Good afternoon. I'm the engineering lead for MLflow at Data Bricks. Yeah. And I'm an MLFlow engineer. Uh I've been here at Data Bricks for about four years. been here a lot longer. So today we're really excited to walk you through all of the advancements that we've made in MLflow 3 for Genai and for MLOps. I'm going to start off with a little bit of historical context. So at this point, MLflow has been standardizing MLOps workflows for about seven years.
First created by data bricks in 2018, it focused initially on experiment tracking for models and model management at scale. In 2023, with the rise of generative AI in the mainstream, we also provided support for experiment tracking of LLM fine-tuning and pre-training. And now with MLO3, we're announcing a major evolution in the platform, providing first class support for generative AI apps and agents alongside all of the existing MLOps capabilities.

`0:28` Evolution of mlflow

And we're doing this at production scale. Throughout the talk today, we're going to cover some really exciting capabilities, including real time tracing for all of your production agents, regardless of where they're deployed or how they're deployed. We'll take a look at the prompt registry for versioning, management, and comparison of prompts across your organization, including governance with Unity catalog. And we'll also take a deep dive into the generative AI evaluation suite that becomes natively integrated in MLflow 3.

`1:09` Mlflow 3.0 features

This includes LLM judges and customizable and sophisticated human feedback collection utilities. To date, MLFlow has become by far the largest and most successful open-source MLOps platform with more than 5,000 organizations building and deploying models at scale on the platform, a vibrant ecosystem of more than 850 contributors and more than 30 million package downloads every month. And after this talk, I'm really excited to see what you all do with MLflow 3 for Genai.
But before we get into features, I want to answer a very naturally occurring question that's probably on some of your minds. MLFlow was originally built as an MLOps platform. Like why would this make a good platform for Genai trying to do many things at once? To answer that question, I want to take a look at the MLOps life cycle beginning with data collection proceeding to data processing and featurization followed by model training and evaluation and culminating in deployment.

`2:15` Genai lifecycle challenges

If we take a look at this life cycle, we immediately notice it's quite complex. You have multiple people collaborating, data engineers, data scientists, ML engineers with a complex stack of tools at each stage. To successfully get a model from development through production, you need to connect all of these disparate sets of tools. You need to enable these developers to work seamlessly together. And this has been MLFlow's bread and butter since 2018.
Now if we take a look at the generative AI life cycle, we do notice a couple differences. The first is instead of data scientists building models, you have software engineers building applications. These are multi-step agents that consist of external functions, business logic, and context retrieval from vector databases and data indices built on top and around that model. When it comes time for evaluation, unlike traditional machine learning where you often have concrete ground truth labels, generative AI outputs are free form and they're nuanced.
There are thousands of different ways to express a correct answer or an incorrect answer to the exact same query, which means that you need more sophisticated evaluation mechanisms. You need LLM judges. You need you need domain experts evaluating and reviewing the outputs of these agents to ensure that they're meeting quality benchmarks. And at deployment time, what we also see is that because you're deploying a multi-step agent, you need observability into each individual node in that application or agent to ensure that everything is working properly.
And because the outputs in production are free form, you need guard rails on the responses so that you can ensure you're not responding inappropriately or leaking sensitive data. So while the overall structure here looks quite similar to MLOps, there are you know additional tools in the ecosystem and there are additional challenges presented by multi-step applications. However, you're still going from data collection through deployment.
You need that data to evaluate and improve the quality of your agents. You need these multiple personas to collaborate effectively. You need life cycle standardization which led us to explore MLflow for generative AI. In addition to looking at the life cycle, we talked to hundreds of customers. We explored the ecosystem of other platforms out there and we identified five core requirements for Genai platforms. First and foremost, and probably most obviously, intelligence.

`5:05` Core platform requirements

Your Genai platform needs to offer access and integration with highquality LLMs that produce great responses. Just as importantly though is full life cycle support. To move from data collection to production, you need a comprehensive suite of tools for observability, evaluation, and asset management. This includes tracing, LLM judges, feedback, monitoring, versioning, and prompts management and beyond. Interoperability is also super important.
Generative AI apps and agents are fundamentally software projects and you can build software using this wide variety of programming languages, all sorts of authoring tools and frameworks. And so your Genai platform needs to meet you where you're at. It it can't force you to change your stack, adopt tooling, rewrite new programs. And so that's critically important. Open standards are also fundamental. If I'm going to adopt a geni platform, I need to be able to revisit my decision 6 months, 12 months, two years in the future.
And I can't do that if my data is locked into proprietary formats. In the context of generative AI, this means MCP, it means open telemetry, it means integrating with open standards that ensure you can move your data from one platform to another if you so choose. And the final requirement here and perhaps the most important requirements beyond intelligence is security and governance. See, generative AI applications and their data are more sensitive than ever.
Which means that you need enterprisegrade security and access management. You need encryption, audit logs, network controls, identity integration, and more to prevent disaster. So with these five requirements identified, we took a look at the landscape in particular four categories of Genai platform and we decided to see how they stack up. Where do they satisfy certain requirements? Where do they miss? We looked at foundation model providers like OpenAI and Enthropic.

`6:58` Landscape analysis

We looked at the hyperscalers AWS bedrock, Azure AI foundry. We looked at agent observability startups that a lot of you have probably heard of. Langmith, Langfuse, Arise and software observability platforms that are moving into the genai space, data dog and others. All of these check the box in intelligence. They provide access to LLMs or they integrate deeply with LLMs, even the software observability stack. But when you start looking at these other requirements, things break down pretty quickly.
If you're looking for full life cycle supports, the best you're going to get with foundation models is basic observability and very limited evaluation. And many foundation model providers don't deliver those tools at all. Software observability platforms, as the name would imply, they're focused on observability and tracing. They don't do so much in terms of feedback collection, prompt management, versioning, any of that.

`7:44` Mlflow for genai solution

Interoperability is actually fairly good with foundation model providers. They're giving you REST APIs. The agent observability startups optimize for it. And software observability platforms aren't particularly opinionated about the app that you're observing. But the hyperscalers still try to lock you in. They want you to use particular sets of frameworks, SDKs, and web-based workflows to get your agents into production and realize full benefit.
Track record is also fairly mixed on open standards. While MCP is the shining example from Anthropic for foundation model providers leading the way in open standards, you can point to many others both across foundation models and hyperscalers that don't integrate with any particular open standard which means if you choose to use them, you're locked into one particular platform. It's hard to migrate. And finally, from a security and governance perspective, there are a lot of boxes checked by hyperscalers, model providers, software observability platforms that are reputable, but agents observability startups, they recommend that you self-host.
That's the solution for enterprise security. They're putting that burden of managing and securing a production scale platform on you, the developer or the business owner. So, what we see is this landscape is far from perfect. There really isn't a particular category of platform that checks all the boxes yet these are fundamental to your success securely at scale which led us to the question can MLflow do better? Can we take this MLOps foundation that we built and extend it for generative AI?
And we began with the open core of ML flow and we introduced several valuable capabilities for Genai that make it a complete platform including tracing which provides rich Genai observability across environments. Evaluation delivering human feedback and AI judge metrics to give you a complete and concrete sense of agent quality and performance. Versioning enabling you to track, manage, and compare prompts, configs, and performance over time. and the AI gateway which gives you complete control and auditability of every LLM API call across your organization from development through production.
Just as MLflow was originally designed for compatibility with any model, training or authoring framework, MLflow for Genai is compatible with the entire ecosystem. Pick any programming language, pick any authoring framework or tool, MLflow supports it providing the tracing, evaluation and beyond. Not only is MLFlow built as an open source platform itself, it also integrates with open standards. We're MCP compatible, we're open telemetry compatible.
If MLflow is integrated into your stack, you have the freedom to choose other tools in the future. You are not locked in. And this platform standardizes the full endto-end life cycle from data collection through deployment of generative AI apps and agents. On data bicks, MLflow is even more powerful, offering that enterprise governance to secure your data at scale, integrating with the rest of the data bricks data intelligence platform, MLflow underpins the newlyannounced agent bricks feature, AutoML, Genai fine-tuning, and much more.

`10:53` Enterprise governance

And with data bricks, AIBI and SQL, you're given consistent SQL access to all of the agent performance data and traces across your organization. Now that I've set the stage for ML flow and explained why a historically MLOps focused platform is an excellent solution in this case for Genai, particularly on data bricks, let's dive into the core features starting with the most critical one, tracing. Traces are just as important to understanding and improving the quality of Genai apps and models as training data sets and test data sets are for classical machine learning.

`11:40` Real-time tracing

Without traces, you have no way to understand the performance of your Genai app and you have no way to improve it. Traces come in through just about every portion of this endto-end life cycle. For example, if I'm developing a Genai app or agent, I need tracing available so I can debug quickly and understand what's going on uh at 10 layers down the stack. And that allows me to get to production faster. Once I move into production, I can't just rely on enduser feedback to catch 100% of errors.
You know, there are edge cases that users are not going to report or even egregious problems that may go unnoticed, which means I need to collect all of the requests and responses and intermediate states and analyze them automatically in real time using sophisticated LLM judge algorithms to be confident that my application is not leaking sensitive data or delivering inappropriate responses. After I hit production, I still am going to need to improve.
My first, second, third version are often far from perfect. That means collecting those request response logs, collecting those traces, using subject matter and domain experts to collect valuable labels to supplement those traces, and building evaluation data sets that I can use to test new versions. As I test those versions and grow my evaluation data sets, quality gets better over time. And ultimately the cycle repeats itself.
I run evaluation, I perform development, I promote to production and traces remain central to the entire process. Now fortunately MLflow makes tracing and observability extremely simple for Genai. There's one API call that you can drop into your existing agent code in Python and get automatic deep instrumentation with more than 20 Genai frameworks including OpenAI, Anthropic, DSPI, Langchain and more. This in particular is a lang chain example.
We see that I'm importing MLflow and calling its auto logging API. Then I can just run my agent or app as is. This will collect traces and seamlessly transmit them to the production scale MLFlow tracing back end where I can then analyze them in the UI. Now, what if I'm not using one of these frameworks or what if I'm extending those frameworks with custom or external code? MLFlow solves this use case as well. With simple decorators and powerful Python context managers, as well as full TypeScript integration, I can instrument every part of my program whether or not it comes from a particular framework.
Here's an example where I'm decorating a tool call for address validation. And I can immediately see that the inputs and the outputs from every invocation of that tool are made visible in the MLflow UI. Once I've insert I've instrumented my app, I've promoted it into production. It's serving traffic. It's collecting requests and responses. These all become visible in the MLflow UI. I can see rich visualizations including retrieved documents that are rendered beautifully giving me a very straightforward mechanism for debugging bad documents or retrieval issues in my agent.
I can also take a look at in-depth stack traces from MCP tool calls. So any tool call that I'm making with my MCP integrated agent shows up here as well. I can easily identify it by icon or by search API. And this really facilitates quick debugging. Beyond that, these traces don't stop at just execution flow, inputs, outputs, latency. You can also seamlessly send these traces, the inputs and the outputs, and everything in between to domain experts for labeling directly from our UI or API.
It's as easy as a click of a button. And as you run automated LLM judges that are integrated with MLflow, which Arpit will be discussing shortly, all of those results are propagated directly to the trace, you can see them side by side. It gives you a complete picture of overall quality while also giving you a detailed breakdown of everything that happened. So you can map the quality results to particular portions of your trace.
Tracing is deeply integrated with the MLFlow experiment UI, which some of you may be familiar with from the MLOps offering for models. We've made the experiment a one-stop shop for everything generative AI. You have your traces in there for observability. You have your evaluation results, labeling, session management, monitoring, versioning, and prompt management all in one place. And this extends across environments.
For example, as I move into production, I can see all of my production logs alongside the dev and staging traces, analyze them, build dashboards, and curate evaluation data sets, all from a single centralized pane of glass, which makes it much easier than it ever was before to analyze and understand agent performance at scale. You can also analyze these traces with data brick SQL or with the MLFlow Python SDK. And MLFlow comes with built-in notebook integrations that make it very simple to view traces without ever switching tabs or windows.
Simply execute your agent code in the notebook. We'll show you a full interactive trace that you can use for debugging. It makes iteration a lot faster. Now that I've covered tracing and demonstrated how fundamental tracing is to agent performance, I want to move on to prompts. Prompts are also a major determining factor in how well your agent or GI app performs. So, it's important to track and version and group these together over time so that you can better understand the trends in performance.

`17:06` Prompt registry and management

Are particular types of prompts performing better than others across your agents? Are there particular failure modes or certain prompts that are responsible for regressions in production? You need those answers and you need prompt management to provide those insights. In MLflow, we offer a Unity catalog integrated prompt management solution surfaced directly through that same experiment UI. You can govern all of your prompts with a standardized set of permissions across dev, staging, and prod.
And you have total lineage from individual prompts to all of the agents that consume them and all of the all of the traces, all of those execution workflows that use a particular prompt. making extremely easy to reproduce and diagnose issues related to bad prompts or to understand performance trends. These prompts can be seamlessly integrated into your agent code. Use a simple one line of code MLFlow geni load prompt.
Specify the name and the version of the prompt and pass in your template variables. For example, a query and that's all there is to it. This particular example demonstrates how I can load a prompt used by an LLM for product Q&A in a chatbot and then just pass it to an LLM of my choosing, for example, OpenAI's GPT40 Mini for inference. And these prompts can also be used to populate descriptions for tool calls. You can use separate prompts in the same application and load multiple versions at the same time.
It's a very powerful abstraction. I'm also really excited to announce that MLflow is the first prompt management solution to natively integrate with prompt optimizers. Some of you may be familiar with the DSPI framework which provides sophisticated algorithms for automatically improving prompt quality. Removing the need for this manual human in the loop prompt engineering action and pattern that many of you are spending a lot of time on.
With a single API call, you can specify a prompt in our prompt management solution. Specify the LLM that's used to perform inference with that prompt. Give us a data set to optimize over which you can easily derive from your traces whether in dev or production and then a collection of evaluation metrics also built into MLflow that we'll be covering shortly. MLflow will automatically generate candidate prompts using the DSPI optimizer.
It will store those prompts as improvements are made back to the prompt registry. You can then compare the results over time alongside their performance characteristics and automate away all of the manual prompt engineering. Here's an example of the comparison UI. It's probably a little bit small and hard to see, but the point is that it's very easy to compare prompts across versions. We highlight the differences and we make it very visually apparent what changed from one version to the next.
We also recognize that GNI versioning and application packaging goes far beyond just the prompt. You care about a whole host of other hyperparameters. You're interested in tracking which particular LLMs are being used with a given prompt. You're interested in knowing which version of a vector database you're using to pull retrieve documents, the maximum number of tokens that your end user response uh can possibly populate.

`20:09` Agent versioning

All of these variables need to be tracked together along with source code and the performance metrics that you obtain by evaluating these different versions over time. This is how you can gain confidence that a particular version is good enough for production or perform accurate comparisons. MLFlow also makes this really easy. You can package up all of your parameters, links to prompts, code, and link those metrics in addition to linking traces.
So coming back to tracing in addition to that oneliner for auto logging I can add a second line called set active model. I'll specify an identifier whatever I choose it to be for my version. Every trace that I create whether this is a production environments or just working in a notebook is seamlessly linked to that particular agent version and I can actually see all of those traces directly in the MLflow UI alongside the quality feedback.
I can send those traces for labeling. I can compare those traces to other versions and I can query all of this information through APIs as well using the simple MLflow search traces capability. Pass in the ID of my agent and I'm off to the races. I can build evaluation data sets on an agent versionby version basis. Now that I've walked through tracing, prompt management, and agent versioning, I'm going to hand it over to take you the rest of the way, beginning with Genai evaluation.

`21:45` Genai evaluation

Cool. Thanks. Appreciate that. Before I jump right in to what's going on basically with evaluation and everything, this is an open session. We have two mics there. So if anytime anyone has any questions, please don't hesitate to ask. I know uh covered a lot of awesome stuff which I hope folks are excited for. Yes. Great. Uh cool. Yeah, we'll do one quick question and then I'll move on to the content. Go ahead. might be for you by the way.
Yeah, it makes sense. Yeah. So I had a question on like production workflows where you may not have a experiment available that would enable tracing. So is that really like handled by the inference table connected to like a model serving endpoint? Is that how how that would work? You got it. Exactly. So the question being in particular if I'm deploying on data bricks like do I get my traces flowing directly into my MLflow experiments without having to you know go through a bunch of overhead you know adding configurations environment variables all of that and the answer is absolutely uh you give us your agent or Genai application code.
You specify an experiment either through our UI or API, the data just starts flowing in. We also persist that data to inference tables or um managed catalog tables in Unity catalog and you can analyze those with SQL. You can view them in the MLFlow experiment UI as well. I'll be covering that in a little bit as we actually go in and talk about the deployment step. Before I get there though, did a great job walking us through, you know, how basically data collection, data prep, building works, especially with versioning, providing prompt versions and getting all of that.
But now we go into one of the more trickier parts of Genai, which is evaluation. So what makes this so tricky? Well, compared to classical software, which is generally much more fixed and structured input, how do we evaluate free formed inputs and outputs? So in classical software, you would write and run code locally. You have unit tests, you have QA testing and production telemetry. So for genai the first part at least we can do with prompt engineering and vibe check where you can see okay yeah I gave it this prompt seems to answer reasonably but for unit test and quality basically QA testing uh we need something more and that's where we'll go into basically MLflow evaluation and how basically you can use expert feedback as well as scores to make this happen.
So as well as actually using this for uh production telemetry which we'll talk about with monitoring. So agent evaluation which hopefully folks are familiar with is actually now directly a part of MLflow 3. So before we have this MLflow evaluate call that is currently used for both traditional ML and was used for agents as well. And we realize there's a lot of stuff there that maybe makes it a little complicated and stuff but we all we really need are three main things.

`24:14` Automated llm judges

The prediction function. Oh pressed the wrong button. There it is. Uh the prediction function, the data and the scores. So the actual function is just the agent you're evaluating. So in this case it's GPG40. You have basically certain requests. Pretty simple, pretty self-explanatory. And then from there we can talk a bit about the evaluation data set. In this case we have both inputs and expectations because we want to have a general guideline for what the expected response should look like.
But now the interesting bit that we get into is the actual scores themselves. So for basically this MLflow evaluation, you can specify a score that can return any of a boolean, a numeric, a string, an assessment, and write any kind of Python you want, which makes it really easy to do any kind of custom logic that you may want to do for your basically your LM and make sure it's being scored on for each trace that comes in.
But we realize folks might be like, well, I don't know where to start. How does this look? So, we also have a bunch of predefined LM scores that you can just use right out of the box. The first four don't actually require any ground truths. Most of them are either related to the user's input or are basically objective requests like safety. The last two do require basically a ground truth and then you can use that to compare something like for example if we go back a couple slides to the expectations.
This can be used as a ground truth to make sure that when I say hello world you get something along the lines of a response back introducing the bot introducing itself. So that's within correctness and then retrieval efficiency will be especially useful if you're running any kind of rag or you're retrieving any kind of documents and you want to make sure that they contain basically all the necessary information the user may have asked for in that request.
So with that we can talk a bit more about how you can establish without expectations and with expectations. So with these judges that you're basically using these LLM judges, you can actually specify guidelines and say, "Hey, the outputs must be professional and just give it certain kind of norms to adhere to or you can give it direct expectations. For example, the expected response that we saw and you want to make sure that it actually adheres very closely to the response that you're expecting.
This can be especially useful for chatbot scenarios where you know what kind of responses should be done about your company's knowledge base and Q&A. Uh so with expectations makes it very powerful to have an understanding on how your your actual app is performing on basically general uh requests. So how do you actually run this evaluation? Pretty simple. Like I said, you just basically call geni evaluate. You give it that data.
You give it the agent and then you give it those scores. And you can see that basically now this is what you'll see in your experiment UI. And right along those traces that mentioned now you get basically a bunch of assessments and you can see over here that we have for two right now is correct and is professional. So actually let me go back there. So one thing that you can see every time he uses laser he keeps moving the slide forward.
Maybe no laser then. So we can see basically for the first one what is the capital of Mexico? Mexico City. Excellent. that's both correct and is professional. When we saw with the hello world, you know, I wanted to introduce itself in a reasonable way, which in this case it says what's up dog. Uh I mean some may argue that's correct. I think it's not and then I also don't think it's professional. Um whereas the next one the hello world and then it asks you basically how it can assist you.
Great. Both correct and professional. And the last one, what's the capital of India? It says Mumbai. I know a lot of folks like to think that but it's actually Delhi. So that's not correct but it is professional. Shout out to all my Mumbai folks in the audience if there's any. So moving on to basically the expert feedback bit. Those scores are great. They make it very easy now to have automated evaluations that you can run and have a reliable sense of basically what's going on with each of your requests.

`28:20` Expert feedback

But you may also want a human to actually go in and say, "Hey, are these looking good? What does that look like?" And we provide two avenues to do this. So one, we have the review app. You can just send this over to your expert. They can play around ask it basically questions and then it can they can also provide bas answers on the guidelines did it perform as expected uh other basically assessments and feedback and so on and so forth.
You can also start a labeling session on pre-existing traces where the expert will then go and say yeah the score did a good job here or actually no this isn't great and all this as mentioned will be visible on the trace and then can be used to improve your model again. So it makes it very easy to collect both automated LMS judge feedback and custom expert feedback as well. So with that what you can also see is that we also make it easy then to take these evaluation traces and form data sets.
So evaluation data sets are very powerful. A lot of times you'll create an app and maybe just to start off and test oh I want to know basically this Q&A chatbot if it does well in a production environment or even just in my dev environment. collect a bunch of traces and now you want to generalize to a bunch of different basically use cases. Well, you can do that pretty easily with evaluation data sets. With evaluation data sets, you can create a new data set.

`29:22` Evaluation datasets

You can provide basically the schema where you wanted to have it in Unity catalog getting all the benefits of governance lineage and everything else. I'm sure you've heard about Unity catalog at this point. And then you can populate it pretty easily with all the traces that you've already collected. You can also populate it with other custom ways, code, and so on and so forth. MLflow in general is a very open platform as mentioned generally if they have more custom use cases even with the scores for example you can always go and make it much more custom than just what we defined we like to provide easier journeys because we know most folks will want to basically do it which is why we have the export traces here such that you can populate that evaluation data set so now let's put it all together and you can see basically over here you have everything that you kind of need to see on your evaluation you can see when it passes relevance ments.
You can see when it passes safety and you can click into the trace. You can see each assessment, a detailed view, the input output, what tools it called, everything that you kind of need to know about what happened with this request and is my app responding correctly. Now, that's great for one evaluation run, but as mentioned, you may have versions. You want to know what's going on across evaluations. No problem. We can make it very easy to compare these evaluations as well.
So for example in this case you may see that oh initially in one of the versions you basically had some kind of issue with uh there was always a supervised reroute directing happening and we want to minimize that. We want to make sure that we have the knowledge we can answer a lot of these questions. So we released a new version we had a new basically uh version that actually fixes that and we started a new evaluation run.
Now you can go and directly compare. All right, is it supervised routing? Is it not? And you can see with the assessments, it tells you true, false, and you can see that the new one is performing a lot better. It makes it very easy to understand. You look like you have a question. I'm going to answer that really quick. Go ahead. Um, so for your relevance to uh pass fail, do you have the ability to set thresholds for that one?

`31:41` Production monitoring

Do you have the ability to set thresholds for when like it's pass fail? Yeah, absolutely. And you can ab that's the whole point with all these assessments and scores. You can set your own thresholds and actually it's almost like I planted him that it brings me a perfect segue to my next actual uh slide where you can set up production grade monitoring such that basically you can make sure that on all your real-time traces coming in you can automatically perform evaluation using these scores and have them already pre-populated for assessments such that when your expert needs to go and check they can see everything has already been graded.
Setting up monitoring is fairly easy. You can see that you basically just save the judges. You can provide certain guidelines and then you can basically specify a sample rate as well and you want to see these monitoring dashboards. You can see on all the new requests coming in how it's performing. you can see okay what's the latency how why is the quality good and everything and you can then set up alerting as well with custom thresh thresholds and say okay yeah I want to know when my uh app is not doing well when it is doing well and as soon as it's not I need to be able to go in and maybe roll back revert do other things and you know manage it in deployment so and with that actually let's talk a bit about deployments because evaluations now looks like we have a pretty good handle on what to do but how do we actually get this agender model into production So we are introducing a new thing called deployment jobs which is a first party integration with data bricks jobs to allow agent and model life cycle management.

`32:57` Deployment jobs

So a lot of you I'm sure are familiar with a data bricks job. This is an example of a simple data bricks job that we set up to manage the model life cycle evaluation approval deployment. So generally what we basically realized is when folks create a new version and register it to UC a lot of these things that we saw with like evaluation automatically happening when you set up monitoring you'll also want to run all these things evaluation approval and then eventually deploy automatically as soon as you have that new version.
So as soon as basically you realize this you can go to your UC model and set up a deployment job maybe giving it a template like this or whatever custom template you want and basically say that yeah as soon as this new version is registered evaluate it and then after that I want my expert to go in and check is it good is it not and we make this really easy another great part of MLflow 3. 0 know is that actually now we have centralized performance data in UC and I'll show that on the next slide and what we've basically done is taken your metrics params and everything from all your different environments and brought it into UC as well as your traces so now you can see very easily in one spot especially if you're expert do I approve this version or not do I want it to deploy to production or not what is going on and if it's good then great should be very easy to do so let's take a look at an example of what this looks like so over here you can see there's metrics and parameters s as well as an activity log detailing what has already happened so far.
This one, all right, I think it looks good. I did my due diligence. Let's approve it. And as soon as you approve it, it'll go and it now also emits a new log letting you know who approved it, when they approved it, and basically what's going on with each task. It lets you know when each task passes, fails. Um, and then you can see basically it moves on to deployment. So making it very easy to now almost have a basically a CI/CD uh architecture within data bricks.
Very lightweight and easy to get started. And if in general if you want to see what's going on with your deployment state and basically see what's going on with your activity logs, you can take a step back and look at the model itself there. For each version, you can see the model serving endpoints, the deployment state, as well as, you know, tags, aliases, and everything. And you can take a look at the activity log to see what's going on.

`35:01` Enterprise rollout strategies

Maybe you had a rollback and then you can see, oh yeah, okay, this person approved this rollback and that makes sense. And you can go follow up and see what happened. You can even click into the traces yourself and see ah okay this is why it wasn't performing well enough now I can see why we had to roll back all your questions about deployment state can now be answered very easily in one spot in data bricks so with all that we've made it very easy to have an endto-end AI infrastructure on data bricks itself so I gave an example of that deployment job that was pretty simple with just evaluation approval and uh deployment a lot of times we have much more complicated rollouts though for example So in this case I give an example of a canary roll out where we do say a 1% roll out then we do some more additional testing and metrics do 10% roll out maybe then we have another human in the loop manually verifying that this model and app are actually ready to go to production and then so on and so forth.
So with that and all the traces that basically mentioned, especially with the Unity catalog governance bit, you can see now that loop that we talked about is very easy to set up within data bricks, especially when we talk about these uh these things at the bottom that nice my captions are cutting off, but I'll just go through each of them. It's basically prompts like mentioned. You can see with the prompt versions, you can decide when you want to basically uh improve these prompts.
You can use the optimizers and automatically make sure that the latest prompts are being used in evaluation, the scores and the feedback and evaluation as well to make sure that everything is behaving appropriately with your model as well as monitoring to understand what's going on. The last thing is a thing that we're also excited about called job triggers because a lot of times you may have actually generalized use cases across multiple apps.
For example, your legal team has mandated that every single model that basically, you know, needs to be deployed in this schema, I want to make sure that it's passed the legal check. No problem. We can set up a trigger that says every time a new model app agent is created in a specific schema, I want to run this job that will automatically perform this check and make sure that basically then in my deployment job, you can just check, has this check been done?
Great. You can have this run concurrently. You can have this run with the deployment job. It makes it all very customizable and very easy to set up any kind of infrastructure that you need. One thing that also alluded to with basically AI gateway and basically have over here is Mosaic AI gateway. It learns on all of your agent and LM deployments like we mentioned MLflow very interoperable you can use with anything right.

`37:39` Mosaic ai gateway

So if you have external models open AAI anthropic so on and so forth you have data bricks hosted models with you know llama and mosaic or you have your agent and chain that you built with langraph lang whatever or mlflow then basically you can use all of that with AI gateway and instantly get all these things you get basically usage tracking inference tables that we alluded to and all these features right out the box and this makes it very easy so that in fact if you want to set up a federated infrastructure with even other connectors.
For example, let's say you have a Salesforce agent. Sure, you can make that happen too. You can use basically MCP maybe set up a tool that calls that and now you get tracing. So you can also look at that trace and be like, hey, what's going on with my entire, you know, multi- aent architecture? Oh, looks like this agent was returning this response. Now you know exactly where to dive deep and dig into. So we've made it very easy to manage this entire infrastructure.
But I know we've talked a lot about AI and stuff. What about normal ML and traditional ML? Well, yes, we made a unified platform. Almost everything that we mentioned today that works great for Gen AI will also work great for machine learning. One thing that we've done, we realized how important versions are. We also pulled out the concept of a model from the runs and made it its own dedicated concept to make it very easy to track the life cycle of the model.

`38:52` Unified platform approach

So clicking into this model itself, you can see this one that has been moved to UC, you can see all its metrics. You can see all the parameters still there's deployment jobs there's activity logs all these things still work for your existing models today in MLflow uh in the traditional ML side of things so this centralized performance data is now basically all also governed by UC so you have the lineage you have all these other benefits that I mentioned before and it makes it very easy to understand what's going across all your environments and I think that makes it very very powerful and one of the common things that we heard with customers saying oh I don't know what happened with my experiment runs.
I don't know what which model is being deployed. We made it very easy to now know exactly what's going on, especially once you bring it to UC. Speaking of pulling out the model concept, we see basically we also have a lot of customers who are doing deep learning. In deep learning, right now all you would have is one single MLO run that contains a bunch of checkpoints and then you maybe you took one, you was the best one, you registered it to UC, but then you know the next person who wants to audit it looks at it and goes wait which checkpoint was it?
Why would why did we pick this one? What's going on? We've made it easy and it pulled out all the checkpoints as well into their own models. So now you can see that the top one was the best one that we thought was good enough to move to UC and you can click and understand what's going on with that life cycle as well as the older models, the other ones that didn't make the cut, why didn't they make the cut and you can see what's going on.
Similarly, I alluded to this. We also have deployment jobs and activity logs for traditional ML as well. uh you can do approvals, you can do evaluations as we mentioned u you know all these things that we built for genai we've also made sure traditional ML supports it so I know I talked about the first four but here's just an list of like all the things that you know can be done both for genai and machine learning uh mentioned that you can set up custom traces a lot of folks think traces are just for genai but actually you can use them for traditional ML as well you can set them up for example on your deep learning model to understand what's going on at each point which functions are being called and get an understanding about what's going on in my uh deep learning model providing a layer of explanability that was actually you know pretty hard to get before.

`40:36` Deep learning and ml features

Along with that, as I mentioned, the job triggers you can use to set up all kinds of custom infrastructures. And we also now moved on along with moving the centralized data to UC, we've also made it such that you can actually have all this data available via DBSQL and you can utilize in your AIBI dashboards, making it very easy to understand what's going on with your latest models, how they're performing in, you know, important loss and accuracy metrics, parameters, so on and so forth, as well as traces.
And we already have all these pre-existing ML features like MLflow 2X has evaluations monitoring that already work with traditional ML. So now we just made it one easy way to do both Genai and traditional ML seamlessly. Along with being able to do these things with ML flow, we've tried to also make it very easy to do AI and ML in general on data bricks. So I'm sure you've heard about the agent bricks announcement. Very big, very exciting stuff.

`42:01` Databricks ecosystem

But one thing you'll see is for example when you're reviewing there's traces and those are the same traces that you see in the rest of MLflow 3. You have assessments, you have basically expert feedback. All the power that you have with MLflow 3 is also now being used across. So it's one centralized way you can go see what's going on both in agent bricks and with basically MLflow3. Everything is very easy now to basically reason about and utilize in all your AI apps.
Similarly for fine-tuning we can still basically use the experiment UI and get all the other you know benefits from traces versions monitoring labeling so on and so forth as well as you'll notice we have this training tab because for fine-tuning you'll have training runs so then you can actually go and check and as well look at the life cycle of each of these runs and the versions that are basically pulled out from there.
Similarly for AutoML, this is also built on MLflow 3. We have basically now all these things that you can use that you were now basically using for AutoML traces, models being pulled out. You can all basically use everything with, you know, getting started really quickly with AutoML. I know you've been saying you can do all this really quickly and everything, but I really want to make sure that point gets across. So I'm emphasizing that because everything that you can do with AI and ML and data bricks is now enriched by MLflow 3.
And here's an example of some of the other things that you can do with that as well. So I mentioned the first four, but if you were using infrastructure as code data bricks asset bundles, MLOp stacks or the new flavor MLOp stacks, LMOP stacks, all this is also enriched by MLflow 3. You have traces, you have deployment jobs, you have activity logs, you have the dedicated log model concept, centralized performance data, everything.

`43:35` Conclusion

You can set up evaluations, everything can be done through that as well. Similarly with vector search you have your vector search you have your retrieval you can see what's going on with tracing there as well you understand what documents did it retrieve why did it retrieve those how long did it take to retrieve those all that's available mcp like I mentioned if you me make any kinds of tools you have other things you want to call you want to maybe set up your data bricks agent as a central orchestrator for multi-agent architecture no problem with MCP you can set each of these things up as a tool in unity catalog getting all the benefits of unity catalog as well as now being able to see where it basically each tool is being called, each agent be called and all the benefits of that your a your experts can go in and label that with feedback.
Just one second, let me finish the slide and then I get to your question. Um and then basically with batch inference as well and AI functions with AI functions, we make it so I'm sure you heard about the exciting lakeflow builder announcement which makes it all SQL and you can use AI functions with that. Great. You also get traces on all of those because those will use basically model serving. We have inference tables.
We have traces, everything is integrated so that now if you're even doing batch inference on a bunch of basically, you know, queries and prompts that you have ready to go, you'll get automatic traces on them. And if you set up monitoring, boom, you also get evaluations on that. It's that easy. And a bunch more that I haven't gone into in here, but I want to make sure that I can answer a couple questions. I'm almost done to with the next slide, but let me answer uh one question before I move on.
Yeah, go ahead. Yeah, I'm really excited to see MLOps stacks listed here and you alluded to some templates earlier in your presentation. Um, do you think that you'll th those templates for agent deployments? Um, I I mean is it integrated within MLOps stacks or or another template? Is that what you were alluding to? Yeah. So, very uh exciting stuff. Um, basically we have a new flavor of MLOps stacks called LMOP stacks.
U we can maybe rename that to agent opts. We can see that that part's uh TBD, but we have basically this uh new functionality available today. Uh if you don't see it, you can always contact your data bricks representative to get started with it. And that should set up basically a bunch of components like a feature store index and all the way ending with the data brick app that deploys your agent. Uh and then yeah, that should all be integrated with mlop stacks.
Today it's not a native option, but soon it will be. So right now it's still like in beta, so you'll need to again probably sign up for that. Awesome. Thank you. Cool. So before that I answer the next question I kind of want to just circle back to what was mentioning with you know what's going on with Genai platforms these days. So we talked about the foundation model providers. We talked about these hyperscalers, the agent observability startups and the software observability platforms.
All of them come pretty close but you know there's something that's just generally missing. ML flow on data bricks we've made sure to build from the ground up to adhere to all these core principles. We saw that we have intelligence. You can use our data bricks hosted models. You can use your own models agents. You can use any model that you want to bring in from anywhere and just connect it as an external model and you have basically access to all those full life cycle support between prompts, traces, evaluations, everything else that we've talked about today.
That entire life cycle that we mentioned is now covered really well and end to end making it very easy to understand what happened from data prep all the way to deployment interoperability. We showed it with basically all these icons. We made it very easy. So then you can set up all kinds of architectures that you want with anyone else. We're not locked into data bricks. Data bricks has always been really good about that.
Everything that you have basically is yours and you can always bring in more use all the tooling as well as getting the power of data bricks. With that we have open standards as well with MCP and everything else. And lastly and I touched on this especially we have security and governance right out of the box. Unity catalog is very powerful. We have guardrails, we have gateway, we have all these governance tools and all these things that make it such that MLflow really is the best genai platform out there.
We make sure that we adhere to all these principles and we're only continuing to iterate to make it very easy for everyone to be able to do app development and genai on data bricks especially with the coming AI wave. Hopefully, it's easy for everyone to get started today. So, with that, please get started and thank you very much. We can now move to some Q&A.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.