Comprehensive Guide to MLOps on Databricks

Summary

MLflow 3.0 for AI lifecycle: The latest MLflow iteration handles the complete machine learning lifecycle—from development to deployment—with special emphasis on generative AI and LLM management.
LLMOps and AgentOps patterns: Arpit and Eric cover advanced practices for managing language models and AI agents in production, including versioning, evaluation, and deployment automation.
MLOps stacks and promotion: Learn governance patterns, CI/CD automation, and promotion workflows that ensure machine learning models transition safely from development through staging to production.

Comprehensive Guide to MLOps on Databricks

Talk by Arpit Jasapara, Software Engineer, Databricks ; Eric Golinko, Staff Data Scientist, Databricks
This in-depth session explores advanced MLOps practices for implementing production-grade machine learning workflows on Databricks. We'll examine the complete MLOps journey from foundational principles to sophisticated implementation patterns, covering essential tools including MLflow, Unity Catalog, Feature Stores and version control with Git. Dive into Databricks' latest MLOps capabilities including MLflow 3.0, which enhances the entire ML lifecycle from development to deployment with particular focus on generative AI applications. Key session takeaways include: Advanced MLflow 3.0 features for LLM management and deployment Enterprise-grade governance with Unity Catalog integration Robust promotion patterns across development, staging and production CI/CD pipeline automation for continuous deployment GenAI application evaluation and streamlined deployment
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are the key improvements in MLflow 3.0?

MLflow 3.0 simplifies LLM and generative AI management, providing enhanced versioning, evaluation tools, and deployment capabilities specifically built for modern AI applications.

How do teams implement LLMOps in production?

LLMOps follows similar principles to MLOps but tailored for language models—version-controlling prompts and models, evaluating outputs against quality thresholds, and deploying changes through governed promotion pipelines.

What is MLOps and what capabilities does a modern MLOps platform need?

This comprehensive guide to MLOps on the Databricks Data and AI platform covers the full machine learning lifecycle: data preparation, feature engineering, model training and evaluation, model registry and versioning, deployment and serving, and continuous monitoring. A modern MLOps platform must integrate these capabilities seamlessly while providing reproducibility, auditability, and scalability.

What does a no-pipeline data architecture look like end to end?

This video demonstrates a simplified, no-pipeline MLOps architecture on the Databricks Data and AI platform. Rather than managing separate data pipelines, ML pipelines, and serving infrastructure, the platform consolidates these into unified workflows. Data scientists work directly in notebooks with access to data, features, and models in one place, dramatically reducing operational complexity.

What is the total cost of consolidating data science and MLOps tools onto one ML platform?

This video analyzes the cost implications of consolidating fragmented data science and MLOps tools onto the unified Databricks Data and AI platform. By eliminating licensing fees for separate tools, reducing engineering overhead for integration and operational management, and improving data scientist productivity, organizations achieve significant cost savings while improving reliability and time-to-value.

Full transcript

`0:00` Introduction to MLOps

Uh I'm an engineer here at Data Bricks. Been a data bricks for a while now, around four years. Uh and my co-speaker today is. Hey everybody. Nice to meet you. Good. Cool. So today we'll be talking about a comprehensive guide to MLOps on data bricks. Now, we're basically going to be going into what MLOps is. Basically, what an LLM ops and agent ops can look like in this world of Gen AI. MLflow 3. 0, the latest basically iteration of MLflow that we've launched specifically built for AI and MLOps at scale.
Touch a bit upon MLOps and LMOS tax, the prime offering that basically data bricks has to get your LMLOps infrastructure started off the bat. and then some Q&A at the end. So, I think it's a pretty packed agenda. Uh, we'll try to make sure we kind of cover everything. That being said, if anyone has any urgent questions, there's like, oh, RP, what you're saying up there is completely wrong, I disagree, please raise your hand.
I would love to hear that. Uh, I'll try to, you know, if I spot any questions in the crowd, I'll try to answer them, but otherwise, we can, you know, have them all during the Q&A at the end. Cool. So with that what is MLOps? A lot of you are I'm sure familiar with data ops, devops, model ops. Uh and the idea is that MLOps kind of combines all these processes. It's basically bringing you know CI/CD and what you kind of do generally for your code to uh the ML side of things.

`1:20` What is MLOps

Uh and I said this last year and I'll say it this year. I know folks are taking photos but these slides will be available at the end. uh if you want to save that precious memory you know for kids uh picture of kids and whatnot. So cool. So for yeah mlops basically uh we have this big book of mlops um and like I said this slide will be shared later that link is clickable but you can also just search it on Google and find it and this basically uh book goes into very depth on how to set up your uh actual MLOps infrastructure.
But basically the set of processes that you really want to kind of think about are how is my data basically being fed in. How are folks be able to improve their model? That's the DevOps side of it. And then the model's life cycle itself. How does that get deployed? And all three of these basically combine to make it so that you have a very stable, efficient and risk basically tolerant system. So this life cycle I touched on a little bit but basically this is kind of your uh at a high level what your life cycle generally look like.
So you can basically start at any point but we'll start with generally you know where most folks would kind of think about it which is at the training step. So at the training step you already have some amount of data someone has come to you with a business need or you discovered that hey we need to basically build this. You'll use your favorite platform listed a few there uh to basically build uh your actual model. At that point basically the data scientist will also perform some evaluation and see okay how is this model actually doing?
Is this any good? Is this ready to go? That point the ML engineer will take over and say yeah okay cool it's been validated. That's where the handoff occurs. Data scientist does a bit of validation. ML engineer does a bit of validation. Then we go into deployment. We actually deploy this model get it out there in production. Folks are using it. people like it and you're starting to collect the data on how this model's performing again and that's where we go back into data collection.
The data engineer has then set up some pipelines hopefully using lakeflow and then now you get basically that data prep done. You can do a little bit of feature engineering there to clean it up and again feed it back into your model and this cycle will continuously improve your model through various iterations. Now why should I care about MLOps? So one of the major you know factors in MLOps is it helps reduce risk. Imagine if today uh all the engineers in the world suddenly were to lose GitHub in every single version control.
It would be a mad dash right like folks can't even imagine what that would look like. Everyone just committing code willy-nilly. How does that look? So if you think about that's kind of the state that we have right now with ML. Data scientists will be like here here's my notebook. Oh here's my notebook. Which notebook do I use? Which is the latest one? Is this the better one? Is this the better one? have to go run validation every time.
It's very very basically complex. So you want to make sure that this MLOps infrastructure that you set up will help you reduce that risk and basically make sure that you're both compliant and technically you have the right you know setup models to deploy to production. Now MLOps will also improve your long-term efficiency through automation. A lot of things that you know you don't have to do. Maybe you're up at 12:00 a. m. because you really need to push that next pipeline to the region, you know, in some uh Asian region basically like, "Oh, Japan went down.
I need to stay up." We can do all of that through MLOps because now you can have staggered deployments, automated rollouts, all these things that you normally have on the code side, you can bring to ML. So with that kind of high level overview, we're going to go into some of these best practices that can help you achieve these goals. Now one of the best practices that we have is environment isolation. So you'll have your assets and generally we can divide assets into three sections which is code your actual you know training notebook inference notebook whatever you have uh data your features your labels your actual data that you have hopefully in unity catalog actually probably in unity catalog because we'll talk about why that's important uh and then your model your actual model that you built maybe through sklearn it's your malflow model um and the main thing with these assets is the first thing is they need to be developed and you want to do this in your dev environment you want to test it in your stage ing environment and then deploy it in prod.

`5:05` Best practices and patterns

And it's important to maintain a good separation here because you want to basically keep the access level in dev open. Everyone should be able to see what everyone else is doing. You should be able to work on everything. Um but on the flip side with that comes basically a lower trust in the quality of the models, quality of the data and quality of the assets in general. On the other hand, with prod, you don't want to touch that.
No one should have access to your prod environment except maybe a few administrators and your service principles. So then that makes it so that the trust quality and basically test testability of your assets is very high as well as then you want to make you're knowing that like your customers are getting a safe basically model goes through a bunch of checks and balances to make sure it actually gets to production and it's completely locked down and there you know like I said there's trade-offs there's basically pros and cons on each of these and that's why it's important to make sure you isolate these environments in the right way.
Now another basically best practice we'll go through here is deployment patterns and commonly we'll see basically two deployment patterns deploy model and deploy code. Now the main kind of you know challenge of MLOps actually resides in this where you know if it was just the same as software engineering why didn't we already do it why didn't we just start using GitHub right off the bat and the tricky bit is that models will basically change each time you iterate on it right each time you basically give it new data it'll get better it'll get worse something will happen so inherently models feel like this almost like blackbox where you give it data it changes and therefore how can you have reproducibility across different environments And you know that's actually what we want to kind of focus in on.
So with the deploy model approach that's kind of the approach that's kind of taken where okay I'm going to make it in dev then I'll do some staging validation and you know I'll improve the model using that and then you know continue basically doing that in prod as the prod data comes in we'll continue to improve the model. Now this kind of it poses a risk because now let's say something happens to this model some weight goes bad or something how do you grow back?
How do you revert? How do you go back to that last iteration that was good? Uh what were the weights? We don't know. And that's why it's important to maintain reproducibility by what we recommend actually doing is the deploy code architecture where there is basically certain levels of code. And there is as long as you're using the same data in the same basically you know hyperparameters and stuff you can actually achieve a pretty high level of reproducibility in these models.
So then you run that code in dev then you take the code and you actually move it to staging. Then you do it with staging data. You validate it. Now maybe yeah with the staging data it actually performs worse. That means that the model actually isn't good. You should then it doesn't generalize well. So then you do need to go back and you know go back to dev and try to make a model that works and so on and so forth. Then the same thing with prod.
So deploying basically the training code across these environments will make it so that now if you need to basically revert it's as simple as rerunning the code and having it go onto another container. So going a bit into this deploy code uh you know kind of idea. So in the development environment you'll train your you know you develop your training code you develop basically ancillary code and you promote that code.
Once it goes to the staging you'll test that model training code and the subset of data. You'll test the ancillary code as well and promote that code. Then finally you'll train this basically model on production data. You'll test the model and you'll make sure that it still performs well with the production data which should ideally be a superset of your training data and what you have in production and then deploy the model incillary code.
And the advantages of doing deploy code will be you have automation because now you can do automated retraining in lockdown environments. You don't have to worry about basically doing that manually. You have data access control. So only the production environments actually really needs the read access to the production training data. You have reproducible models engineering control basically over the training environment.
And that's going to make it very easy to, like I said, roll back and revert if you need to. And you have support for larger projects. Now you don't have to worry about, you know, data scientists changing each other's notebooks, moving that around. Whatever you check into basically, you know, your version control, that will be the source of truth. So this next slide is a bit much. I'm sure some folks will take pictures of this one because it looks complicated, but that's kind of the idea, right?
Setting up an MLOps infrastructure is difficult. And you know I'll go through some of the important pieces here but hopefully the point that you know we'll show with some of these products is we make it a little easier which is set up these uh basically this environment especially with uh MLflow3 and MLOP stacks but um one thing I kind of want to mention is um you'll notice at the bottom there we basically say unity catalog and the nice thing basically that we have with Unity catalog is you have complete control over your governance of your data.
So what does that mean? So you can see actually in the development environment there's basically an optional prod catalog data. So some folks will be like yeah you know what my production data I need to make sure it stays in my production environment only my production environment can modify it but I want to make sure my models are you know being developed right off the bat with the best data and that's going to be what we're getting latest from our customers in pro in prod.
So it's like okay no problem just take that catalog share it to your development workspace and now you have that data to train with as well. So you can see basically that this architecture can also uh evolve and adapt based on your needs but generally kind of convey some of the best practices that I talked about. I'll pause there for a quick sec to see if anyone has any questions before I move on to the LM op sides of things.
All right, cool. So what changes when we start putting LMS into the mix? Because now it's completely different, right? You can't do deploy code. Maybe it's the how are you going to take your LM model that you spent millions of dollars to fine-tune and deploy code it through the different environments or how do you know the reason about what does it mean to train this like you know model well you'll see actually a lot of it stays the same uh the main thing that you yeah you're you know thinking about is the building tuning step which has now replaced your training step but you'll still basically need to be able to you know actually create that lane chain lane graph whatever the actual code that will take care your model, you'll need to perform some kind of evaluation to make sure it's actually good, deploy it somewhere, then you'll start collecting data on it.

`11:30` LLMOps and AgentOps

You can feed that back into your model, improve it, and you know, the cycle continues in a similar way. Now, obviously, if it were that simple, LMOPS would be a solved thing and much more of a common industry term or agent ops. Uh, but it's not, you know, uh, obviously there's some pitfalls here and one of the biggest pitfalls is actually evaluation. So how do you actually have automated reliable evaluation on something that doesn't have structured input and output?
That's the tricky bit right. So with that we'll go a bit into especially what MLflow 3 has but generally we have a few options there between expert opinions feedback random basically vibe checking u as well as LM judges and some scorers that you can set up as well. deployment will be largely the same except you know you'll want to set up also some guardrails to make sure that you know your LM is not being misused as well as you know wanting to make sure that it scales well but this is pretty similar to what we have in the ML ops world so I won't go too much into that data collection largely stays the same and then for data prep you may you know put your stuff into a vector index that's the only real difference and then kind of going back into it for building and like you know tuning like I said it's just your whatever your link chain your rebuild your tools whatever else you need to do but all that inherently will still be code even if you're using GP40 you're still going to need some code to call that and the model itself may change behind the scenes especially if you're using a proprietary closed model rather than your own oss model but that's kind of you know hopefully negligible in terms of what your actual use case is because generally you won't need that model to be able to distinguish you know the right dictionary word you I wanted to make sure it takes care of your use case and has good understanding of your context, you know, your documents, your data, whatever it is.
So with that, we're going to talk about a bit about MLflow 3. Before I go into it, any more questions about this is kind of the overview of MLOps section and we're going to go into how data bricks does this well and better than anyone else? No. Okay, great. So what is MLflow 3? MLflow 3 actually let me get a quick show of hands. How many of folks here are familiar with MLflow? Lovely. This is the that's what I like to see.

`13:52` Introduction to MLflow 3.0

So MLflow you know you all know and hopefully love and it has been pretty good in basically the ML world built by our CTO himself mate. It has worked really well to make it so that you can have reproducibility and have very easy comparisons on what's going on as you train your models. But how do we take that and make it, you know, easy to do it in the Gen AI world? Because like I said, the Gen AI world, you have tools, you have all these other things coming in.
How do you take your prompts and, you know, I have different prompts. I want to use them all over again. Um, what's going on in my actual model? Before you could just say, oh, input 0 point whatever, output 1. 25. Great. That looks great. Now, is it good? Is it not? So what we're basically introducing with MLflow 3 is a bunch of new kind of you know uh concepts as well as redefining what we think especially with that first bullet point emphasis on the model.
So currently in MLflow we was the emphasis was on your runs because you would train your model and then evaluate your model and it's all about how each run would basically look at it and we realized especially in the LM world the model is actually much more important these days. It's much more model centric. So we've kind of put an emphasis on the model and made it very easy to track your model's life cycle. And we'll be doing, you know, some slides as well as a demo.
So we'll go more into detail on that later. But that's the main idea is we want to make it so that now it's super easy to understand what is going on with my model. Then the other basically thing that we've done with now having a central model life cycle is well we can also centralize all the performance data and have it be governed by UC. So your metrics, your params, your traces, now all of that is in one location, very easy to see and you can share that across environments and have all the other benefits of UC lineage etc.
Tracing both for traditional ML and Genai. Uh let me get another show of hands. How many folks are familiar with MLflow traces? A few less I figured. So tracing and I'll go into that a bit into the next slide. uh tracing is pretty helpful because now we get basically a lot more understanding of what is actually happening behind the scenes in your model. Generally your like uh sklearn model is a bit of a blackbox. You just you know have some weights and it happens but your LLM generally your agent is not.
It's going to be calling certain tools. It's going to be doing basically these functions and tracing makes it so you can see all of that and you can actually bring it back to traditional ML. For example, your deep learning model may be doing some stuff as well and you want to actually uh track track when it calls certain you know functions and does whatever. So you get all of that. We also have something called deployment jobs and activity logs.
And basically this is a closer integration with our workflows product in our jobs product to make it such that you have a a dedicated deployment life cycle and a manage model management life cycle as well as auditability on what's going on with my model. what's happened in each deployment state assessments and evaluation. So we can now actually go and have those expert feedback go in see what's going on with my model.
Uh okay, this is better, is this not good? You can have these experts label it. You can do automatic evaluation. I think we mentioned that DBSQL support. Now your metrics, your params, your traces, you can take those and actually put them into your AIBI dashboards. This is really great. Now you can show look my loss is decreasing and my accuracy is increasing. So you can basically show all these things now using all the you know power of DBSQL as well prompt registry which is going to let you basically store reuse prompts and as you can tell by my windedness I'm getting tired of listing all these things there's a lot and you know I would highly recommend all of everyone to go check out our website mlflow. org There's a link at the end as well as you know going to the data bricks section on MLflow as well because managed MLflow is also getting all this and going to be really great.
So that's MLflow 3. How does it work with MLOps? Why does how does it improve basically MLOps before we go into LM LM side of things? So for MLOps, we basically make it very easy to go through one of the most standard flows. So let's say for this model life cycle management, we have your model creation. data scientist creates the model and performs some comparisons. Now they want to register the model version that they deem best.
Today that's a little difficult. Uh what does that mean? Which model do I register to? This cat this prod catalog this dev catalog. How does you know folks know where to get it picked up? A lot of folks had to set up some custom infrastructure using jobs or ADO or whatever to basically say oh yeah this is the model you definitely want to deploy to and this is how it's going to pick it up and so on and so forth. We make it a lot easier to do that now. basically with a deployment job and your deployment job will basically pick up that model run on the new model version run that evaluation step to create that evaluation run in data then the job will pause waiting for approval then the approval will go in say hey looking at all that centralized performance data this looks great let's go ahead and let's move it to production so something before that was very complicated so how do I get my actual you know model version that the data scientist has made all the way to production in a safe and like you know echo lockdown way is hopefully now a lot easier.
So what does that look like? So then in the MLflow slide or in the experiment side you can basically see that now like I mentioned we put an emphasis on models. Uh so the runs tab you're familiar with but in these models you can see that we have these models you can still see where they're logged from and you know we can see as well that oh they've been actually moved to the UC as it's been seen by the three three level name.
So if we were to click on one of these, we could see here's my model version that was now deployed to UC and you can see, oh, there's all my metrics. There's my parameters as well as an activity log. What's been happening? Looks like evaluation succeeded. And you can see with the data set, oh, that evaluation run. Yeah. Okay, that's basically good. That's my score. Now I can look at that score. I can scroll up and down, decide what I want to do, and say, you know what?
Yeah, this looks good. I'm going to hit that approve button. And now let's move it along. So moved it along. It succeeded and now you can see that it's all tracked in your activity log for good auditability purposes and generally making this flow very easy and hopefully very simple to use. So let me give you an example of what a deployment job could look like. These are some templates of in this case the top one is what the exact deployment job we used for that model and the bottom one is one maybe you can set up.
The nice thing about deployment jobs is they're flexible. Folks will have all kinds of different things. Some people will like the top one where it's like yeah do an eval. Let me just approve it real quick and then you I want you to deploy it to my model sering endpoint. Some folks will be like no no I want granular roll out. I want you to roll it out a bit maybe 1%. And then I want to do more checks and then 10% and then more checks and then I want to get legal involved and then more checks and then I want to make sure there's a bias and fairness check.
Sure, whatever you want, you know. Uh and that's kind of the beauty of it. All of that will then be then, you know, visible in your activity log. You can see what's going on, who approved it, when they approved it. You know, all these information will be there. So, great auditability as well as being able to see in a very clean way your deployment state on the actual model itself. So, you can quickly look at all your model versions, be like, "This one's deployed, this one's not.
I see what's going on here." As well as what's going on with each version. So for the basically you know the traditional ML side of things, MLflow 3 has made it very easy with centralized performance data as well as everything else the deployment jobs and activity log to now have a complete MLOps life cycle right off the bat. That's a pretty simple case. What if I want to do something more? Well, the idea is it's it's customizable.
So let's say now you basically want to do deep learning. Well, we've pulled out the concept of a deep learning checkpoint into its own model. Before you'd have just have a training run, you'd be like, I think it was version number, you know, checkpoint number 12. I like uh let me register that to UC. Oh, where'd it go? I have to go to UC. What's going on? We've pulled that all out into dedicated like models. So now when you're doing deep learning as well, you can you see, okay, this is my favorite model.

`21:47` Advanced MLOps with MLflow

I can see all the metrics. Here's my charts comparison across everything. As well as now I can set up custom tracing. I can say I want to basically know when this model you know passes this certain checkpoint or when it basically calls this function I want to log this trace and going back to this you know the simple example that doesn't wasn't exactly code deploy it's just taking the model and you know maybe in a single workspace deploying through all the regions for a lot of folks getting started with MLOps it's the easiest way to get started we want to make sure that even folks who are like I just spin up a workspace my data scientist created a model well you can get you know governance and lockdown right off the bat so you can even do it without having to worry about setting up a full deploy code architecture and be like yeah I'm just playing around with this great you can still have reliable governance alles and approvals but for the code deploy approach how would this work well you can set up a deployment job in each environment you can basically say okay now once my data scientist has basically registered this model to this one it'll kick off this job that job can basically go and then now run the next one and staging you can set all that up and you'll get you know basically activity logs across all of it the nice thing about deployment job is you can then use the same model or different models uh in different environments.
Use the same model, each deployment job is workspace tied. So then you can have different basically, you know, activity logs showing up from all of your produ like you know your production, staging and dev environments basically on the workspaces that you want to. So you can see what's going on or you can isolate them. And something that we have that's coming soon is actually job triggers. So deployment job makes it so you can basically have one centralized job that manages your the life cycle of your model.
But a lot of times folks will have different jobs for different things. Oh, as soon as someone registers to basically this staging catalog, I want to immediately trigger like a bias and fairness check. Sure, but job triggers that's doable. What we'll basically make it so is so that each time you know certain events occur, you set an alias, you set a certain tag, you set basically, you know, create a new model or model version in specified schemas and cataloges, it can auto trigger a job.
Now you can see how this basically makes it very easy to set up your entire infrastructure inhouse without having to go and say oh let me go set up ADO let me go set up GitHub let me go you know transfer my tokens over hopefully this makes it a lot easier the other thing that basically we'll talk about and with MLOp stacks and we'll come back to it is you can also integrate this with MLOp stacks as well so it actually is not a replacement necessarily it can be on the earlier stages but it has a synergy with it once you have a more advanced architecture you can have a bunch more deployment jobs running for each project.
So that makes it hopefully a lot easier to manage like I said the entire infrastructure within data bricks. Last thing I want to mention is the AIBI dashboards are DBSQL. So like I said the metrics and performance data and all that stuff is going to be in DBSQL. So you can set up these AIBI dashboards very easily and see exactly what's going on all your models across all the different environments. All right, before I move on to the Genai side, any more questions?
I know I'm pausing really less because I'm hoping there's no questions so I can keep going, but if there are no. Okay, great. Oh, coming. I'm kidding. Go ahead. We're We're getting a mic over to you. And I'll repeat the question, too, so that for the recording or folks uh watching online, they can hear it, too. I'm a little bit confused about the direction we're headed to. So is is the idea I don't need GitHub workflows anymore or my Bitbucket workflows.
I'll be managing everything from the data bricks environment with MLflow. Is that like the target state? So yeah, that can be a target state. Uh we know that folks already have set up their workflows and everything in Bitbucket and everything. So we don't want to necessarily say that they can't use these with as well because generally even if you set up something in Bitbucket and stuff you'll probably still have a data bricks job at the end that needs to run.
Now instead of you know having that data bricks job and go call some MLflow commands and do everything you can just basically say I just need to I all the only reason I have this bit bucket flow is so that when my data scientist checks in a PR it just automatically trains that model. Well, you can just do that now with the deployment job, right? So, the idea is not that necessarily replaces it. But for folks who are getting started, who haven't actually set up the Bitbucket and GitHub workflows, yes, I imagine actually for them hopefully this should actually replace much of that functionality.
They can now just set up everything inhouse. Um, at some point if you know you decide, yeah, I want to migrate things over, you can definitely do that. Then we also have data bricks repo integration. So, if you still want to keep your stuff in Bitbucket, GitHub, whatever it is, you can. or you know you can use a different version control uh because now you're not necessarily locked into that one. That answer your question.
Great. Cool. Okay. Sorry, one more question. Similarly to her question, I was curious if um how this integrates with data bricks asset bundles and like those sort of workflow deployments. Yeah, exactly. Great question. So you can set up your deployment job via data bricks asset bundle as well. So let's say you have um your entire mlop infrastructure set up via mlop stacks. Um mlop stacks use a data bricks asset bundles and you know I'm sure maybe you have your own custom way but I'll use stacks as an example because what we do is we declare an experiment in code.
We declare a registered model in code. We declare some jobs in code and then we kind of say now all that is deployed but you don't have your model versions yet. You don't have any of that stuff because you deployed your infrastructure. you haven't actually done all the training yet, the infrastructures code is separated from your actual dynamic basically training workflows. And that works perfectly with deployment jobs because then you can still declare these are my notebooks.
This is exactly how I wanted to, you know, train, evaluate, and do everything. Just have that all in code, deploy it, have the deployment job ready to go. So as soon as you even train your first model version, it'll immediately trigger and fire off that deployment job. Answer your question. Cool. Now going into a bit into genai and lm ops with mflow 3. 0. So this is now basically the genai experiment and you'll notice it actually looks a little bit different than what we just saw for our mlflow traditional experiment.

`28:15` GenAI and LLMOps

There's traces evaluations uh labeling etc etc and you can see now basically the first tab is traces and we're really focusing on tracing. So u I'll come back to that in a sec but let's go into first the first step of that cycle that we talked about which was the LLM building and tuning. So here's some popular basically you know platforms to get LLMs or build your own LLMs on or you know building basically wrappers around LLMs and you'll basic have it in your code you'll create some basically you know notebook and generally have this ready to go.
Now one thing basically that you'll see then is with tracing what can we do? So if we basically click into this trace, we can click into what happened with this input ML flow. So you can click in and see okay this is the input this is what the system prompt was. This is the user. These are the components that are basically used. Key components as well as you can see what's going on with okay this was the completions.
Now you know what basically happened there and I'll go in there's another basically example later on that's a bit more complicated when you have a bunch of tools. Which tool was actually called? A lot of times for example you realize that your LLM isn't working as expected simply because the keywords that you provided wasn't using the right tool. Tracing makes it very easy to see that. How long did it spend on something?
Maybe you accidentally put a time. T for 10,000 seconds. I don't know why you would do that, but if you did, you'd be able to see that in your trace now because now you can see in that function that took really long time to basically run and so on and so forth. makes it very very basically easy and powerful. So that basically when we go back to this, you can see all these traces. You can know exactly what's going on in your model. you can see what's being inputed, where it's basically spending time as well as when we go into basically eval traces and then feed them back into your model as data.
So going a bit into evaluations. So this is an example basically of your evaluation. So we have basically certain labels here that we've added. Oh pass, you can click into it. You can see that trace basically you can see oh these are the assessments that you know your experts added here. Here were the inputs and here were the outputs. Um and you know you can kind of basically see very easily and compare um especially if you were to then go into the chart view what's going on uh how are traces basically behaving now that's kind of interesting because you know generally when you would think about each of these requests they could be very different requests someone might say what is mlflow someone might say uh my dog is my dog's tummy is rumbling I don't know why they'd use your you know chat app for that but they might now you can see then basically with all these different kinds of inputs how can we generalize how can we have charts And that's the idea, especially with assessments, you can basically set up your own custom scores as well as use basically our built-in and you know generally recommended scores to see how it's uh how the model is generally performing on a wide variety of inputs and outputs.
Speaking of scores and monitoring, so on the left we have a code snippet example of basically what it could look like to set up your own evaluation with a custom score. So we have this new endpoint malflow geni evaluate that makes it very easy to then add certain guidelines as well as you know putting in your eval data set and you can see and you know on the right there basically this is what it would look like as you basically add uh certain you know assessments and basically you're set up your monitoring so that now okay not only you can go and compare when you're actually you know building and tuning your model and improving it you can also even see when it's in production when your model starts to degrade and why maybe it has degraded especially with the latest versions.
Again, something that we have in MLOps sides, but now we can also have in the LM op side. So, this makes it very easy to understand and know when something goes wrong and get alerted on your model. And the nice the interesting thing about the main kind of you know difference between genai and MLOps development is a lot of times basically you'll need to iterate on your model as basically you're deploying it as it's in production whereas often times you can do clear boundaries in ML like yeah that one was deployed sucks let's go on and basically say the next version will be better with this basically you can say now okay I actually want to collect these traces yeah these traces now I'm going to use that and feed that back into the same version of the model and you know improve it and then you know it's kind of an iterative loop.
So that is obviously an option. We also I'll talk about this in a sec but we also have versioning and I'll kind of go into that so you can actually have dedicated versions of your uh LM as well. But before I you know get ahead of myself this kind of shows the automatic evaluation side of things. This shows how you can set up that auto eval with scores and you can see that okay um now with my deployment job this can actually run automatically and you know going kind of tying back into how we were using MLflow evaluate on the traditional ML side of things this will make it so that you don't actually have to worry about you know manually necessarily doing that first pass on your LLM to see if it's actually performing but generally after this you know these scores and this you know setting this up you' be like okay well I want to actually see you know if my experts agree that this model is good and we have uh two ways to basically make it very easy to have an expert feedback.
One, we have basically this review app. And this review app, you can basically go in when you ask it certain questions, you can say provide your feedback. Performed well, didn't perform well. This will get logged as a trace as well as basically everything that you provided feedback on will also get added as assessments. You can also create a dedicated labeling session and send that over. And this labeling session, you can go in and actually take existing traces and say, "Yeah, I want to go and, you know, label these, these are good, these are bad, so on and so forth." and makes it very easy for your experts to provide their feedback and then you can use that to improve your model.
So we talked about the building side, we talked about the evaluation side. The next side is deployment. So like I said, you can still use a deployment job maybe, you know, we're keeping it simple with just the evaluation approval. Obviously make it as complex as you want. And then after that for the deploying step, that's when we integrate into AI gateway. Now AI gateway is very powerful for folks. I'm sure hopefully you've done any kind of model serving.

`34:12` Deployment and monitoring

It provides a bunch of basically things. These are just a subset with the usage tracking, payload logging, guard rails, AB testing, granular limiting and that comes on all of your LM deployments. So if you're using an external model, sure yeah you get it on OpenAI. You're using the data bricks hoster model, yeah, you can get it on meta and if you want to use an agent and chain, you can get it on that as well. So all of these will then basically have this you know dedicated deployment uh capabilities and you can just have that as a single step in basically your data bricks notebook something as simple as agents. ep deploy.
And then when you're like, I don't want to use this model anymore, no problem. Just switch it out. And it makes it very easy to create that same deployment job template for maybe several different cases because now all you just need to do is switch out the model. Everything else stays standardized. But kind of getting a bit further into Mosaic Airway, especially the payload logging inference tables bit. This inference tables are great because now we talked about this with like traces.
Now you know not only were you basically those offline traces that you had during your batch in EVEL now you can basically do it during online as well you'll be collecting basically all of your you know logs in real time and you can get all these traces as well so you can see and you know basically on the top example that oh someone queried it what happened you go into the inference table now you can actually see that trace pop up and that makes it very powerful because now it goes and feeds back into that whole loop again now you don't have to worry about oh I need to go and get all this basically production data hopefully my you know eval data was good but you know I don't know how it's going to go and then when you're collecting production data make sure that oh maybe I need to trace it somehow maybe I need to log it it's all taken care of and then you can take those traces you can go in and basically you know set up labeling sessions have your experts basically go in and then say yeah these assessments are good these assessments are not and then you you know if someone were to basically say hey what is ML flow you can even add a new assessment actually useful data and you know going back to the guy who has that dog with the stomach issue you can be like yeah I don't really care about this response don't ignore it so you can make it very easy to see what's going on there as well as have generalized statistics at the bottom you can see basically this inference table it's governed in UC it's a UC table now you can share it you can show it can send send it across to maybe like there's a regulatory org you send it there they'll you know do some checks maybe you have a AIBI workspace you can you know share the table there they can set up their own AIBI dashboards on this data and say yeah you know I want to see how these traces are generally doing what kind of you know questions are we getting all that is now very easy and doable and I'd say I cycle back to this with versioning so we do basically have you know uh model agent versions as well so like I said you can improve the agent you know with the traces or if you want to do a clean cut and say no I want to change you know how my lang chain looks I want to change which tools I'm using you can set up different basically versions as well.

`37:12` Versioning and registry

And we make that very easy like I said by pulling out the model and you know agent concept into these dedicated versions. You can share these versions and then you can treat them as you know how you would with model versions and that makes it a very very you know clean and easy way to reason about agents rather than being oh did this improve which tools is using oh someone improve this agent you can just cut them off and have basically dedicated versioning similar to everything else.
On the other hand we have something that you know we're actually launching in beta called prompt registry. We actually already have this on the OSS side and we're introducing a managed version that will be governed in UC and the advantage basically to your prompt versions now can be yeah you had basically certain prompts and you want to iterate and improve them over time but you can go back and see when I trained basically this model or fine-tuned it rather um which prompt was it using and you can see how it performed with assessments and evals and traces and be like oh this prompt was performing this way and it's all connected easily through lineage whereas now you can see this new prompt oh yeah it's performing much better.
These are the kind of feedback that we should be getting and this is how we should be structuring to get the right response out of it. Prompt versions and prompt registry makes it very easy to basically do like fine shot and few shot tuning on your u on your actual model as well as improving it. Okay, I think I just like you know yapped for 40 minutes there. I know I took a couple questions but I want to make sure I pause for a second if there's any more questions before we go into the demo.
No. Is this stuff? Oh, we have one question. Um, I want to double click on the prompt development process where domain experts and the person who is kind of developing the prompt. How easy are Sorry, could you bring the mic a little closer? Yeah. Mhm. Uh so I want to double click on the prompt uh development process where domain experts and the person who is actually writing the prompt. How easy or difficult will it be using the view which you were showing previously to collaborate between these two uh uh this one right?

`39:18` Demonstrations

Uh and the screen previous like where you were showing um developing the prompt but like getting the feedback This one exactly over here. Okay. Yeah. Um so the idea is that we've kind of integrated it into all these things. So we make it very easy. So you know to develop your prompt. Um the idea is that you'll have the you can set up templates with these basically the prompt registry. So you can have you know maybe uh let's say you have a guardrail like a simple guardrail a system prompt and you want to make sure that everyone in your org is using that system prompt.
What you can then do is you can say and you know basically this catalog and this schema you set up basically template prompt. Everyone else will then basically start from temperate prompt use that and then basically inherit that or if you want to or you can then extend it copy it and then set up your additional prompt information beyond that. So the idea is that yes everything you're saying it should be very easy to do.
You should be able to see other folks's prompts. You should be able to, you know, use these template prompts to get started with writing your own and then you should be able to evaluate these prompts as well and then see if you need to improve them or make them, you know, uh, more like clarified. Make sense? One more question up front. okay, I'll just repeat the question. Go ahead. Some of us are interested in this particular because um some of us want to use it not as a one-off thing for someone answering a question but as a corporate thing where you can have a sanctioned place to go and uh do operations that result in multi- aent stuff how will that get integrated is there a plan and yeah I mean the beauty of this is that uh this works with your you know single agent multi- aent all these tools right so and let's say we set up you know we data that has MCP support.
So let's say you set up basically your MCP tools and you have your catalog that you you know set set up your server. Now you basically have you know certain connectors to let's say other organizations you have a federated architecture you have multi- aents you have agents in other orgs and like other you know basically uh connectors maybe you have a salesforce agent so on and so forth how can I use all this and have get all these functionality right and the idea is that basically this all this will still work as long as basically your top level agent is still within data bricks and basically using mlflow because you'll still get the tracing functionality of it as long as it's there you'll get the evalu ations experts can go in and feedback except now with the beauty of tracing is instead of just seeing a tool you can see oh Salesforce agent you gave it this uh you know response and I gave it back you like wait that's wrong why is you know my Salesforce agent behaving this way and then you can go and basically debug in there and figure out that was the issue rather than your entire agent architecture so all this works with multi- aent just as well as it does with single agent one question in the corner there and this probably be the last question before uh we go into the demo.
Want to make sure that has enough time. Hi. Um my question is on the version. So in the prompts as well as in the model you showed that there is versioning. How do let's say I came to version seven but now I feel that my version two was better than ver version seven. So how do I revert back to version uh two is it an easy process? Yeah, that's the beauty especially if you were using a deployment job is now it's as simple as now you just take basically version two and rerun that.

`43:05` Troubleshooting pipelines

So if you saw on I don't want to click like 40 screens back because this clicker is a little slow but basically you saw that there was a redeploy option on every single basically deployment job after you deploy it. So what you can always just do is go back to version two. You just click on it. Like in here you can just click on it and you see and you're like yeah I just want this hit redeploy and that's it because you set up your code in such a modular and easy way that it's easy to roll out new versions as well as revert.
Okay now we're going to go into basically presentation. But before I do that, how do folks generally feel about the content shared so far? Excited. Oh, there we go. You get a round of applause to start you off. Right on. We'll hear. Let's see. Let's move the advances to one. Cool. All right. So, I'm gonna switch over. Hey, everybody. Um. It's nice to be here with y'all. Thanks for attending the session. I'm going to go through a couple demos.
And did you already fantastic? So, we just heard um Arpit's uh talk. He gave us some great tools and great suggestions on ways to do best practice for MLOps that we can extend to LLM ops and to agent ops. And so I want to provide a couple a couple demos. The first one is going to be an example of me uh doing the best that I can to adopt those principles that he just provided of having separate environments, setting up CI/CD, doing uh deploy code, all those amazing things.

`44:12` MLOps stacks demonstration

But we're going to look at it through the lens of traces and evaluations because I want to show uh what happens in production or in real life when we actually catch mistakes. How quick can we go from catching that mistake to rectifying it in real time? So let me first start out by showing my uh environment. So I have three separate cataloges um that are representing our different environments for dev stage and prod. In dev our data science team is doing their kind of work their their day-to-day instrumentation trying different things out um kicking kicking the rocks around and everything like that.
So we can see that the schema is set up uh in the same way for all these and specifically in the genai schema. I have a model that uh that I deployed into production oops into into prod. Yeah. So here's my my mlflow d uh 2025 application. I really want to show it to you guys. So let me go to the playground and rip out a couple prompts and let's see what happens. And so the model or the agent that's deployed is like a very simple canonical example of just an um LLM with a system prompt just and deployed as an agent.
So let's show what this looks like. Okay, so here we go. I'm in the playground. I want to go ahead and use this and let's just ask some general questions about MLflow. So, what flavors are supported in MLflow? Let's see what we get back. Okay, so I didn't add streaming in this. Uh-oh. So, when we read this back, you be wanting to know the flavors of ML Flow. It looks like I got hijacked by a pirate. What am I going to do?
So now when I when I see this, this is something that's being exposed to our customers because this is in prod. So when they're asking questions at my endpoint, they're getting back this like wacky pirate response. This is a this is an issue, right? So what I want to do is kind of start to use these these tools that were are part of MLflow 3 now that allow us to see tracing and evaluation. The quickest way that I can do this and diagnose maybe what has happened is I go back to the catalog and I go back to the model view of um this prod model.
I want to find out first and foremost was this always deployed like with this pirate voice or was this ever deployed actually normally. So I can look back at the original version and I go ahead and study the trace of the um of the deployment and I can see back. So you'll notice like a couple different things. I'll just pull your attention to the tags column in at the very bottom here. You'll see that these are actually uh traces that were gathered from my evaluation when the when the code was actually deployed.
And these request IDs are actually being pulled from the inference table from the endpoint itself. So we can view these um kind of as they're coming in if you will. So I'll go ahead and look at the theuh detailed view of this and I can see hey someone just asked what the MLflow flavors are and originally they got a response that looks normal. So somewhere along the lines something happened where someone uh one of maybe my colleagues maybe it was even me probably um who had made some mistake and deployed this into production right so let's even go back in time like one step further and again this is the power of having isolation of environments I want to look back in my experiments and I know that because I'm doing CI/CD and deploying code that each time that my code is landing an environment I'm running a workflow it's executing it's registering a model.
It's doing evaluation. It's doing all this stuff that's going to give me telemetry about what I'm doing. So, I can look back at my staging experiment and I can see that um I have some evaluations that I had set up for this. Um specifically, I was looking at the safety of the responses that are coming back and also the correctness which is shown in the evaluation and I'll I'll show that piece of where that came from. But again, this is in the staging environment.
So, I want to see I look back at the trace here and I get back the same kind of thing. um I wanted to know about MLflow tracking and I got back like a wacky pirate response, right? So again, knowing kind of the nature of how we move things from dev to staging to prod, I know that for each every single time that I'm I'm getting one of these traces or one of these deployments or the endpoint or anything like that, I have like a a workflow that's kind of like tethered to it that I can go back and reference also.
So I'm going to go back to um our workflows and I'll look at what happened in dev my deployment. So originally, you know, the first couple times you can see these are, you know, if like me, I have to fiddle around and I have to see what works out for me and then I can finally deploy something successfully. Right? So I'm going to go to the code really quick and take a look and see because let me go back and show you where this is coming from.
Right? So I have a a repo that I created that has my CI/CD code, my GitHub workflows. Let me just give a little context. So what this is doing is this is just defining for um each step along the way, every time that I'm trying to put um merge code to one of the environments via the branches, the this is going to do deployment for me. Then as far as my dab is defined, I have two key components to it. I have the data bricks yaml which defines the actual workspaces and maybe like any variables that I'm adding or pushing down to workflows and things like that.
I have that file defined and I also have the resources um in my deploy. So this is the actual definition of my of my jobs that are being or my um notebooks that I'm really putting into production. So there's really only two places that this can happen, right? Because I only have two notebooks. I have this example that I'm trying to share and then I have this deploy. So let's go back to the workflow for a minute and I look in the dev.
So I can I could look back in each of these um tasks and see which notebook this might have been caused from. Let's look in the deploy first going backwards. Right. So I'm just loading in some uh libraries. I have my widget that's grabbing which environment that I'm using. Um and I'm doing an agent deployment. really pretty straightforward stuff. Nothing that's going to infiltrate to make it look like a pirate, right?
So, it's probably coming from the place where I'm registering the model itself. So, let's go back there. So, now I'm going to the register model task and I could look at the notebook where this was executed. And again, this is just a simple uh like langraph wrapper pretty much for an LLM with a system prompt. So, I'm looking I'm looking here, you know, we're looking at this. Where did this happen? What what's happening here?
What's going on here? Here I am. Uhoh. Here's the problem. So, I'm going to put in my system prompt again. It could have been me. Um, that we want our responses to be that of a nasty pirate. So, that's how we're starting to get these responses back that got pushed all the way from dev to stage to prod and we're able to see it through the trace and the evaluation and ultimately the work the workflow itself. So, we got to go ahead and change this, right?
So, I'm going to go ahead into um my notebook here. I'll create a branch. And then we'll make this quick change and then we'll show the CD. Okay. So, let's make this change. You definitely don't want to be the pirate anymore. Okay. So, I'm going to go ahead and commit this change. And then I'm going to go to GitHub and I'll open a pull request for this. So, let's go here. And we can see that this uh this updated my MLFlow pirate.
So now I'm going to start the process of kind of going from environment to environment to environment. And this is going to start with taking my feature and putting it into dev. I'll open the request. I'll ask our pit if he'll approve my pull request. Good to go. We got the stamp. We're going to merge this in. And then this is going to kick off my action. There we go. Okay. So, so basically this is going to give us the tools.
So, from what I showed in the demo, just a quick recap of it is we saw that there was an issue in in my production endpoint. We're able to follow through traces and evaluation from each environment, specifically where this is rooted from. I committed a change onto the dev branch and then started to push the code from left to right along the line to get the pirate the pirate out of our our nice waters. Um, I have another demo that we prepared too.
So, let me let me go ahead and skip to that really quick. And before I before I go to the next demo, is there any questions on that specifically? Yeah. Thank you. Um my question is regarding when you were showing before when you saw the pirate language and when you went to the evaluation said correctness wrong but that's very subjective. I mean what if I want that so at this point your judges are broken because this is what I wanted.
I mean what if I wanted a pirate sites but I was wondering about your scoring too. So this is where there have a question mark. No that's a great question. So let me I kind of glossed over that. So thank you. So let me let me show you what this would look like potentially. So this is where the idea of assessments comes in. So if I go back to that rightness or wrongness, right? You can see on the right hand um has the correctness the safety and then in my evaluation that I would have defined I would have had like a set of expected facts that we thought subjectively um were the correct ones.
But what we can also do if you're an expert in the field, you could go ahead and add your own assessment to this where you can add your feedback. You know, pirates are awesome or whatever, whatever the case would be, right? And you'd be able to add that assessment be part of your trace. My actual question, I could hear you. Sorry. Uh yes, but doesn't explain why correctness is no because I'm little you don't have any additional constraint.
Yeah. So that definition I'll show you where this is coming from and thank you for bringing this up. So let's let's have a look. So if I look back in my notebook where my um where this is coming from all the way through this here I have a little piece and in some arpit slide where there was there was um pieces defined here about this but evaluation right. So part of this evaluation is I I define maybe as an expert what the eval evaluation should be.
So if there's a specific um manner in which like a question should be answered or there's guardrails maybe there's something that the company is it has to have a specific type of language or guideline if you will you can enforce that in your val so that way when it's passed to um the genai evaluate you invoke that scoreer and you could create custom scorers as well so you'd be able to kind of enforce to make sure that your um expert opinion is aligned with those types of judges.
Yeah. So, my question is related to that system prompt you had. Um, some of this previous stuff that they were showing before this was how the system prompts are supposed to be able to be logged directly on MLflow. You're supposed to be able to track them. But what you're showing here is you actually had to go into the notebook to find that the system prompt was changed. So, is that something else that you're supposed to be tracking that somewhere else to track that system prompt?
Yeah, I mean I I wouldn't I wouldn't take this as as certainly like this is more for illustration more than anything, but I the way that I've seen prompt management is probably the way that our pit suggested in the way that we have the prompt registry is going to be the better way to do it that's in beta right now. But I've seen like in um examples that I've done to either have like a um a config file that has the prompt itself so it's separate kind of decoupled from the actual execution code so you can manage it in its own life cycle. maybe we'll do one more and then we'll we'll save some time for Q&A at the end.
Yeah. Um my question is basically uh when you found the issue and you saw that it used to work uh properly before maybe uh from the engineering side or maybe like the probably the common common um workflow will be that hey recover first and then fix the issue. So most likely you want to uh uh redeploy an older version. Um I'm I'm not sure if you're going to show that u or the same site on the g you probably do a revert instead of like trying to fix it.
You're saying in instead of instead of fixing the problem and then redeploying to just redeploy it. I'm I'm not I'm not clear on the question. I'm sorry. No, you saw that this is a regression happened, right? It used to work and now it is not not working properly. then most likely you are you'll be um I mean the most likely workflow is that hey uh use the last known good version. So instead of fixing just redeploy the version that worked before.
Yeah that would be an option too right I mean kind of like what what our pit just uh his example before if if this was a deployment job which it definitely could have been uh you just redeploy the version one. Okay let me let me just continue and we'll do some more Q&A at the end if that's okay. Um, do you have the clicker? Do I have the clicker? Let me switch first. Yeah. Thanks. Okay. So, for the next uh the next kind of demo that we want to do really quick is talk about MLOps stacks.

`59:34` MLOps and LLMOps stacks

So, MLOps stacks is a an infrastructure that's provided. Um, it's customizable that it helps us to manage the life cycle within data bricks and follows the uh best practices here. So we're able to create production MLOps infrastructure on data bricks um by following like a fantastic template. So this integrates um fairly readily with common uh uh providers that are doing CI/CD like uh GitHub and Azure DevOps and this helps us to manage assets as infrastructure as code through data bricks asset bundles.
So this allows you really to focus from a data science perspective to really focus on doing ML the nature of your of your job and really helps to um get started with project development by using the templates that are available and for those who are MLES this really helps to just identify how to set up where to set things up properly to make that uh life cycle a lot smoother. Uh so then once that uh template and CI/CD is set up then there's a solid template of being able to walk through and makes the life cycle of development a lot faster.
So let me advance here. I'm going to keep this paused for just one moment. So I'm going to show a uh a demo, but you're able to follow this on your phone or on your uh web browser if you will. So I'll keep this up for just a minute. Okay, great. So, we should be here. If you if you're following the link, uh then you should be at a a site looks like this. So, we'll go ahead and explore this. Uh I'm going to just talk through this.
Uh but please take the time as as you would need to click through this on your own. But basically, this is a a preset up version of what the MLOps stacks deployed um version looks like. If you were to set this up from scratch, there's like a couple um items within like the CLI that you would add like what's the name of your project? Where's the host of your uh data bricks workspace? What do you want to call the project?
What's your provider? There's really about like I'd say like maybe five, six, seven um pieces of prompting that needs to be done to fill out. And then this pretty much just creates this uh fullbodied repo for you. So the re the read me that's in here covers like how to how to do this um and the different components of what you'll need. So let's go to the next part. So here, this should look like a familiar uh illustration that Arpet shared in his graphic before, but this is really about like enabling like data science teams to really work in a more streamlined fashion and not have to get hung up with some of the things that sometimes can frustrate us.
Um so by streamlining the template for CI/CD and the way that code can be deployed um through different environments just streams a lot of work. So the graphic here just is showing that um a reiteration of what we saw earlier. Okay. So to start the new project that's it's showing us here that you really are just doing like an init of a cookie cutter um template and then filling out some of these attributes that I had mentioned before of uh what's the project do you want to call what's is this a CCD only um what's the root the directory there's just a couple pieces that we need to fill out really to get us started but it's pretty simple stuff okay so if I so after filling out the um the templated piece of what you need.
This now is actually a deployed version of ML applax, what it looks like in in a uh an actual workspace or an a GitHub repo. Okay, so let's take a look here just at the structure of MLOps sacks. Like what do we get? So, the nice thing about this is once we're filling out like the template, we get like all these um things that are pre-populated for us and we get a lot of information up front to like help us walk through how to walk through this as if you're a data scientist, if you are an MLE, like what tasks you need to do, where you need to uh complete the tasks specifically, what you can do, the help pages that you need to find, everything is kind of like contained like in a in a nice delivery for us.
So you can see the code structure here has to we follow our ML code we have the re uh resources and then CI/CD. So we have the structure of this is is spelled out here for us. So like like we had mentioned this is part of an extension of using a dab. So you'll see some of the similar kind of like terminology that if you've used dabs you'll see it in ML applax. So you'll see that there's the data bricks yl and there's also the resources that have the uh specific things that we're using to deploy.
So in this specific case we might have some requirements of our of our modules. We have the data bricks yaml which is that definition of uh where the uh assets need to be deployed the different workspaces. We have a folder for training validation monitoring and deployment which is going to contain like the individual uh pieces of code that do those specific tasks. And then within the uh tests, if we're doing like unit tests or integration tests, those would be placed there.
And then there's resources for the specific um tied really to the specific task of what we're doing. Oops, I lost my button. Kidding. So yeah, please take a minute to read these on your own, but I'm just going to kind of like zip through just for the purpose of time for us. So like I had mentioned for each of these specific components there's a kind of a step through of how to set this up. So if you're a data scientist it walks through like how you would uh commit code open a pull request and then move that along the process.
If you are a MLE or MLOps person, you might be the person who's responsible to find service principles, create secrets and things like that. It tells you gives you instructions exactly how to do that. um which is fantastic, so here there's going to be a little bit of like assumption that we've assumed that we've already completed some type of work um up to the to the next piece. So, let's kind of move it on and see what actually starts to happen once we start to work with this.
So, let's suppose here that um I switched to the dev branch. I executed I I write some code. Um the pipelines themselves have been have been updated. Uh we go ahead and we click on the project name to find the training code itself. Now in the training code delightfully named train um we're able to go ahead and make the edits that we want to our specific code base that we want to execute. So suppose um in our specific example we're training a light GBM model and maybe I want to change some hyperparameters and I want to um change regularization or something like that.
Um I go ahead and I I would go ahead and I'd make my change. Right? So we can see that this is is registered in in Unity catalog and it's using the uh the light GBM flavor. And then finally once that experiment is executed we're able to see all the information back in the experiment uh view of this we get the metrics that were are meaningful and that we care about and we get the summary of all the metrics that are logged in the um in the model run itself.
So one of the things that we want to call out here is like if we look at the R squ because this is a regression problem for our light GBM. We look at the R squ this turns out to be like almost 0. 93 which is fantastic. So we decide that we actually want to move forward with this code. So then once we go ahead we we might uh provide an alias for this to let us know that this is in fact the model version that we want to promote and we want to use or the code and then go ahead and create a pull request for it.
Okay. So we create the pull request and then finally this is going to start to um trigger my integration uh testing from the CI/CD. Okay. Okay, so this is going to trigger uh the CI here and this will do within the CI it's going to validate the bundle and then finally uh deploy it for us. Okay, so we can see this and if there's tests that are being run, I'd be able to include that in the CD script itself. Let's view some of the tests.
So now we could see what's happening uh in the inference as this is running. So the inference test is uh is uh running for us. We can just check out to see what's actually happening. And this is running right now. more. Click on it. Click on that. Click on that link. Click on the run. URL. Click on the URL. U this one. there? Oh, so now thank you. So now this is run and we're able to see that this actually kicked off that job.
So it's doing the training, it's doing the model validation and the deployment itself, right? So we're not going to go through necessarily each of these, but we're able to see that the action kicked this job off for us once we merged. Okay, so our test run for inference has been completed also. And let's take a look here. We can see what this looks like. We can merge it. All right. So now we're merged to main. We're in prod. let's take a look and see what's happening in here.
Okay, so we can see the individual actions that are being carried out on our behalf. And you notice in this in this case uh differed from this is using uh service principles here and we're validating within the uh the YAML the file for staging and we'll go ahead and deploy that to staging now. Okay. So we'll navigate uh to the jobs. We can look for our our um our job and we'll get to see our jobs that are showing up.
Okay. So now we go ahead and uh want to release. Okay. So now we have our our release branch that we're going to pursue. Okay. Stop there. Yeah. Okay. All right. So, we are going to move to the next piece. Got it. Thank you, for those demos. Um, sorry for like speeding that along. We wanted to make sure that we have also time for some Q&A at the end as well as these last bits. Um, and yeah, you know, with those demos, we only showed one journey.
There's a many, many more journeys possible. So, that question about traces, if we could see it through system prompt, that is an option. revert via deployment job that is an option. We just wanted to make sure that we covered at least one journey so folks can at least get an idea of what it can do. But all those customizations you're thinking of odds are yes you can probably do that. So now going into basically again thank you for kind of showing us what MLOps stacks looks like especially today easy way to basically then manage and have basically very very strict guardrails around your enterprisegrade architecture for MLOps.
So if you want to do this now also for LMOPS we're adding a new flavor for that actually called LMOP stacks. Now out of box basically we want to make sure that you can also have everything that you saw all streamlined. So that entire graph that we saw in the beginning that was very complicated looking and rather than having to set all of that up manually you can basically use stacks or lmop stacks deploy the right bundles and get all of that right off the bat.
And that's really nice because having that template, it just makes it very easy to even refer to if you want to do your own thing or customize it. You can start from there and go from there. Um whereas otherwise, like I said, you'd have to set all this up kind of out of the box. We've tried to make that easier with deployment jobs and everything, but naturally, yeah, you know, you'll still have to go and create a model.
You'll have to go create experiment, you have to go and create, you know, like your job. Having all that in code as a source of truth as a dab makes this very easy. So with that with LMOP stacks what you get is all those things we talked about with the MLflow 3 all those benefits as well as being basically able to use Unity catalog for registration for the governance for tools you have basically CI/CD versioning and making it very easy to basically use serverless as well.
So what are the actual components that we're now adding introducing? So you have all the components that you normally have in MLAP stacks but we're also going to make it very for you to ingest and prepare your data in these certain notebooks. We also set up a vector search index for you. So you can have basically that vectorized version of your data. Agent tool calling. We show an example. We have a template. We register one example you know tool for you.
You can change that whatever tools you may actually need as well as then performing automated automated evaluation with LLM as a judge and then having deployment management with data bricks model serving. So custom, foundation, external, agent, whatever endpoints you want to use with AI gateway, all that is available with model serving and all you have to do is then basically switch out the model if you want to use a different one.
Also, we use data bricks apps to then automatically deploy basically that app to a front end. Um, so that now you can basically have that entire thing from code all the way to deploying a nice app that you can share with customers all right out of the box. So you can initialize that and get started with it right away as well as having some testing and CI/CD to make sure the entire thing is governed fairly well. Okay, so with that now we can finally move into the Q&A section.
But before I do that, because I imagine this will take up the rest of the time, I'm going to go through actually the ending slides real quick. We have some other basically product leg sessions for AI at DICE. This is the one we just had today. There's a few more as you can kind of see. I can see some folks running out right now to either grab these or lunch reasonable um and tomorrow as well. So we'll be actually doing a basically a deeper dive into MLflow 3 tomorrow uh and seeing you know a lot more features on that.
So we can come check that out as well as some of these other talks. Here's a QR code to basically try out MLflow3. Yeah, you can basically get started. You can scan that. I'll wait a second for folks to basically scan that. Actually, you know what? I can just leave that slide up and that's fine. And yeah, one quick reminder, please take your surveys at the end. We do check them. It does help inform what kind of content we should be doing.
More demos, less demos, more yapping, less yapping. Let us know. We can improve the talk based on that. And with that, we can now close it out and do Q&A. So, thanks everyone for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.