Skip to main content

From Training to Production: MLOps for Deep Learning on Databricks

Summary

  • Deep learning MLOps on the Databricks Data and AI platform is demonstrated using a dental X-ray object detection use case with a Vision Transformer backbone, where distributed multi-GPU training is managed through Databricks AI Runtime's on-demand GPUs, pre-baked CUDA, and NCCL configuration.
  • MLflow 3.0 tracks epochs, hyperparameter sweeps, and evaluation metrics at scale, and an AI-assisted hypothesis generation workflow helped improve model performance from 0.3 to 0.58 mean average precision in the demo.
  • A production promotion chain uses dev and production schemas in Unity Catalog with automated evaluation, champion-challenger A/B testing, and drift monitoring to ensure models stay reliable long after initial deployment.

From Training to Production: MLOps for Deep Learning on Databricks

Watch: From Training to Production: MLOps for Deep Learning on Databricks
Deep learning breaks most MLOps playbooks. Multi-gigabyte weights, GPU-heavy training, distributed checkpoints, and fine-tuning loops don't fit the lifecycle patterns built for classical ML. Teams end up stitching together fragile pipelines to move models from notebook to production — and paying for it in downtime, drift, and missed retraining cycles.
this video shows how to run deep learning MLOps end-to-end on Databricks: distributed training with MLflow tracking, model registry workflows for large models, GPU-aware serving with traffic splitting and autoscaling, and automated monitoring that catches regressions before users do.
You'll leave with a reference architecture for deep learning MLOps on Databricks — and the patterns to ship models that stay reliable long after launch.
Talk By: Michael Shtelma, Lead Product Specialist - GenAI, Databricks ; Puneet Jain, Lead SSA, Databricks ;

Chapters

FAQs

Why is deep learning MLOps harder than classical machine learning?

Deep learning models have hundreds of millions of parameters, require multiple GPUs for distributed training, and produce large checkpoints that don't fit the lifecycle patterns built for classical ML models like gradient-boosted trees or random forests. The size and compute requirements make distributed training, model storage, and production promotion significantly more complex to manage.

What is Databricks AI Runtime for deep learning?

Databricks AI Runtime provides on-demand GPU access with pre-configured CUDA and NCCL libraries, removing the environment setup burden for distributed deep learning workloads. It allows data scientists to trigger distributed training runs from notebooks or a CLI using a YAML configuration file, as demonstrated in this video.

How does MLflow 3.0 support deep learning experiment tracking?

MLflow 3.0 on the Databricks Data and AI platform tracks training epochs, hyperparameter sweeps, and evaluation metrics at scale, enabling teams to compare runs across multiple experiments in one place. The platform also supports AI-assisted hypothesis generation that can suggest training improvements based on current experiment results.

How do I manage model promotion from development to production on Databricks?

This video demonstrates a deploy-model approach using dev and production schemas in Unity Catalog, with an automated evaluation and approval pipeline that gates each promotion decision. Once in production, champion-challenger A/B testing and drift monitoring continuously validate model performance and surface regressions before they impact users.

Full transcript

[00:07] Let's get started. Today we will be talking about operating deep learning models at scale using data bricks and uh data bricks runtime. So I'm Michaela lead product specialist for geni a datab
[00:19] bricks and I'm with my colleague Punit. Punit would like to introduce yourself. >> Hey everyone, I'm Punit. I am a lead specialist solutions architect focusing on ML and AI based out of UK London. >> Thank you. Uh okay. So let's get
[00:35] started. Um basically training your model is just very small part of the whole operations of all model operations and uh of the whole ML Mlops or deep
[00:50] learning ops uh thing we need to do. So in reality we are usually starting with the data preparation we are doing the training we are after that we need to evaluate our model and what's most
[01:03] important then we deploy it and then basically the most interesting journey starts because what we need to do we need to uh understand how we are going to bring the new version to production. So basically we would need to go through
[01:15] the whole uh through all those stages again and uh it's very like it's not very simple but it's uh manageable for a small models for some I don't know eg boost or random forest models but it's
[01:29] significantly harder for the deep learning models. Why? Because deep learning models tend to be bigger. We are talking now about hundreds of millions of parameters. Sometimes uh we are talking about uh billions and
[01:42] trillions if if it's about lms but we are not going to talk about that during this talk. So yeah, I would say this basically the size makes it complicated. If we are talking about such a size, uh
[01:55] it's also going to use multiple GPUs. So we would need a distribution. Uh that's what we are going to talk about today. But we did not want to make it a very theoretical or abstract talk.
[02:08] We wanted to pick up a use case which uh kind of tackles on all those uh areas and issues. So we have uh picked up the dental X-ray data set and what we're
[02:21] going to show today how you can use that data set to uh train a model which can predict uh basically do the diagnosis and uh
[02:33] find the uh boxes where we have one of those uh um diseases or conditions carriers uh region.
[02:45] uh and we have a bunch of other classes. Okay, but uh if we just do a very small step back, why actually deep learning problems are so hard. Um first here uh
[03:02] we have just 700 images. It's a very small data set. Imaget was train was trained on millions of images and here we have just 700. Right? So it basically means we will not be able to train a new
[03:14] model for that. We would we would need to reuse something. Uh also our images are quite small uh and it's low contrast. It's grayscale. So it's uh let's say bit hard images to work with
[03:28] and uh that's going to be our um issue. So how uh do we solve it on a very theoretical level? uh because you don't have so many images, you would need to go for some pre-trained
[03:41] backbone. Um currently vision transformer as the let's say state-of-the-art u backbone architecture. Uh what vision transformer
[03:53] does it basically gives us two things. First uh an image embedding which like describes the whole image image as a whole and also it gives us a grid of patches and uh for each of those patches
[04:05] we would have also a vector describing them. So basically patch embeddings but that's not what we actually need at the end. We need uh bounding boxes for each uh
[04:19] so for each uh right. So how do we get there? Uh for that we are using uh feature pyramid with the retinaet. Um and okay another important thing I
[04:32] have forgotten to mention. So uh those uh grid of patches it uses this it uses one uh like zoom level and it we cannot uh find equally good
[04:48] big and small objects using that scale or that zoom level. So what feature pyramid is doing here it generates four different zoom levels which allow us to
[05:00] use each of those zoom levels for uh objects of different size. So basically, sorry. Yeah, if we have something huge, we're just going to have like a really
[05:13] zoomed out uh view and we build bounding box there. And for some very very small objects, we would be using like very zoomed in uh and that's basically what feature pyramid is doing for us. And
[05:26] then the retinaet is going to output a list of uh bounding boxes and classes. And then we just need to uh filter out uh duplicates. And uh at the right hand
[05:38] side of the slide you can see the relevant papers. That's what we are going to do. But now let's discuss what kind of backbone we can use. So um there are bunch of them. The one of the best
[05:52] right now is Dino V3 coming from uh meta. Um it's uh basically v vision transformer pre-trained uh uh in a selfs
[06:04] supervised way on millions of images. That's one uh very good contender and another one is Nvidia C radio which actually was distilled from Dino V3 and
[06:16] a bunch of other models. So we have decided to try both and we have uh wanted also to show that if you do the architecture right you can make it
[06:28] pluggable right and you can later switch it and just retrain the whole thing and uh uh that's going to work. Okay, that's what we are building today. And now let's make a step back and uh discuss a
[06:42] little bit why distributed deploying is hard because uh still we we have not so many images but we have a big model. So we would need multiple GPUs and we would need distributed deploying here. So at first very often it's hard to procure
[06:55] GPUs. Uh it's hard to make sure that you have the right uh version of software. So I mean here like a driver, CUDA version, NCCL version, the PyTorch that
[07:07] correspond and works well with uh those uh libraries and of course you would need to manage environment. You would need to make sure that like you run it in the right way. um you can distribute
[07:22] and at the end of the day you have uh right GPU utilization because you don't want to pay for GPUs which are doing nothing and uh data bricks uh has introduced
[07:34] runtime that's exactly the tooling or the platform that solves all those issues. So what it does with uh database air runtime you can on demand request
[07:47] any number of GPUs you want and uh you get the GPUs which already have pre-baked the like pre-baked version of the right packages you have the right driver they have an links and the
[08:00] switches so basically it's really tailored towards large scale distributed uh deep learning workloads and of course it's like house native you can get your data from the UC and so on. So how does
[08:13] it look at in you just request your uh GPUs using CLI or in notebook and we uh provision GPUs for you from our uh GPU cluster and we just run your
[08:28] workload. Why it's important and why it's actually better than the conventional approach. What I see very often that customers go into multi-year
[08:41] uh contracts uh getting very nice discount for the GPUs basically for the reser GPU reservation and what we see after that that due to very low
[08:53] utilization rate actually like I have seen customers with 50% and numbers around that at the end of the day it it uh becomes even more expensive than if you would like just get the GPUs for the
[09:07] time you need, run your thing and then uh give them away. Okay. Now, uh let's discuss how we actually can uh run your distributed
[09:20] training workload on a runtime. So, as I mentioned, we already have here pre-baked uh right like we have the image which already has everything you need. If you
[09:32] are using PyTorch with let's say latest version of PyTorch with latest CUDA and uh standard things like if it's we are talking about LM like flash attention and so on it's already there you don't
[09:44] need to install anything you can just use it out of the box you can also use composer for the distribution we have ray ray is like the first party citizen of air runtime uh we have hugging faces
[09:57] hugging face transformers and libraries which usually see with that and it's already works together. Um you can start uh training directly
[10:10] from notebooks. You can just connect to accelerator you want. Uh you can also run it from jobs and we have CLI uh SSH support. So you can even run your
[10:23] um ID on a runtime and like use it for interactive debugging like of course we have CLI which you can also use in CI/CD but I'll show the CLI uh in more detail a bit later. Okay. So how can you really
[10:37] run it? On the left hand side you see example um which you can uh run a note you can use a notebook. So you can just uh put a decorator on your training
[10:49] function and in this decorator you just say how many GPUs you want and what kind of GPUs. So it can be H100, it can be A10 and on the right hand side you can see the YAML configuration which we can
[11:02] use the CLI. It's something that uh we use very often. I think it's like really nice way to interact with. And here you just uh you can either select your
[11:14] GitHub repo or you can just use the project you have in your current folder. It will be uh compressed uploaded and all the machines that you are using for your training will automatically see it.
[11:26] So you don't need to do anything for that. Okay. And last piece here why governance matters. actually you want to see you want to your
[11:38] um engineers, data scientists to use the data they actually need and you want only the right people so people like which are working on this project to
[11:50] have access to the data. Um what we see in reality what happens that teams are really waiting a long time to get access to the data and uh after that for some pre-processing data is being exported
[12:03] outside of the security parimeter. So that's something that's actually should not be happening with uh sensitive data and on data bricks air runtime is can
[12:15] utilize unity catalog to 100% so you can uh get your data directly from the UC tables you can get your data from the UC volumes all that data is protected and you can give the access to this data
[12:28] only to the right uh team service principle which is used to run the training job. Okay. And now Punita is going to talk a little bit more about optimizing
[12:40] our uh model. >> Yeah, thanks Michael. So um we talked about how what are the pains from a hardware perspective uh when it comes to setting up a deep learning pipeline and
[12:53] how do you follow MLOps practices. So AI runtime is one of the key cogs that enables you to sort of think in a serverless realm. you just get GPUs provisioned instantly. You don't have to worry about all the pesky CUDA and NCNC
[13:05] NCCL connections. Um you can request any number of GPUs that you want on demand. But what we're now going to focus on is what about all the other loops. So we've talked about the train but we still have
[13:18] the evaluation, the monitoring, the serving and then how can we make that entire loop more robust. Right? Um before uh before we start on all of
[13:30] these components uh Michael showed a slide where we had a lot of decision criterias right um we had uh foundational backends do we go with Dino V3 do we go with um C radio uh there was
[13:46] another slide where do we go with full fine-tuning do we go with pre-training do we go with Laura um and if and then I start talking about all the hyper parameter optimization problems. You know that entire surface just starts
[13:59] exploding. So what do we want? We want a very efficient way in which we can just hypothesize all of these parameters, run them at scale and then automatically
[14:11] track and train which is working, which is not working and have that incremental growth from baseline to state-of-the-art. Right? That's the dream. But how can we make this dream a reality? Uh that's where three key
[14:25] components come in. Number one is jobs. So now I'm going to switch my screen.
[15:04] Sorry guys, one sec. Just figuring out technical stuff.
[15:22] Sorry. Thank you guys. Yeah. So what you can do over here is uh using asset bundles you can come up with YAML templates and in each template you can specifically run a configuration. So
[15:34] with the entire AI runtime capability. It does not bottleneck you from only running one trial at a time. You could spin up three or four jobs at a time and all of these jobs will provision their
[15:48] own infrastructure under the hood. Now, it's as same as paying maybe waiting 15 hours to run 15 trials from one box, but meeting one hour and running 15 trials concurrently, but you pay the same
[16:00] because everything is running on a permputee basis, right? So, this enables us to run all of our validations at scale. And the best part is all of this can be essentially something that an AI
[16:12] can trigger on its own because asset bundles as well as datab bricks job CLI are already part of the AI devkit that is going to be triggered. Michael is actually going to do a walk through um
[16:25] in some time. But I just wanted to sort of show that this entire iterative approach is really efficient if you have YAML jobs set up where you can effectively track and tune all of
[16:38] the runs. So in this scenario if I try to open up one run. So here you can see I go into the sequability uh you can see I specify the
[16:50] configuration to run the configuration runs and then it tries to register the model within Unity catalog and we'll come to Unity catalog in one moment but let's go back once. So how do I start
[17:04] running all of my concurrent trials? You have a combination of jobs and you have a combination of asset bundles.
[17:17] Next, how do I start tracking these? This is the relevant element, right? I need a way in which if I'm running all of these trials at a scale, all of them can be tracked. And this is where MLflow 3.0 comes in. So, MLflow 3.0 O has been
[17:31] reimagined not only for LLMs but also for deep learning as well wherein now you have the ability not only to checkpoint the final run but also checkpoint each and each individual
[17:44] epoch if you wanted to. You can come up with your own customized solutions as well. Um again going back
[18:04] So here you can see for the purposes of this talk what we did was set up our hypothesis we started at zero uh and then just use wipe coding and LLM to continuously feed
[18:17] all of the telemetry coming from MLflow back into the system. So now let's see what sort of telemetry can we capture right. So in this you can see we are not only capturing all of the metrics but we
[18:30] are capturing all of the hyper tuning parameters as well. Um going into a little bit more detail you now not only get to track the performance of the metrics but because now we are running
[18:42] in deep learning and GPU systems we also want to understand what our utilized percentages are. And you can clearly see in this scenario the utiliz utilize
[18:54] percentage was at a maximum. That means we are throttling our CPUs at the best capability possible. And our entire sweep run is set up correctly. Now coming into the metrics, you have the
[19:06] classic train metrics, but then you also have your loss metrics that you track over time. But you're not only tracking one particular data point, you're tracking the entire angle. Now just
[19:18] think about it. If you pass all of this parameter back into an LLM and ask it to propose hypothesis of what to optimize next, what would be the answer? Well, we asked the same question.
[19:30] So, the first element was the first premise. Does do we does the baseline model actually have enough knowledge so that
[19:44] we don't need to pre-train the weights? Can we just get away with adding a retina on it? Answer is no. We tried keeping it frozen, adding a Laura weight and adding a full fine tune, but we
[19:56] found that full fine tune actually remarkably improved the results. Now again, all of this is empirically based. Nothing is just a hunch. In a proper MLOps paradigm, you have hypothesis. You
[20:08] test those hypothesis. So the question becomes, is your pipeline able to run your hypothesis at scale? The next element was how can we get from 0.3 to the state-of-the-art which is
[20:22] 0.58 which was what the kegel winners had run with an ensemble of four models. And you can see here this is not a fluke. Oh I just ran a bunch of parameters and got to that level. It's a
[20:36] systematic understanding of running trials, feeding that back into an LLM, generating hypothesis of what next to run and then getting to the baseline. So you can see we started with broken
[20:48] recipe where everything was random, nothing was working. First made the leap from frozen backend to fine-tuning, got the initial bump, then fixed the anchor assignments, finally
[21:02] added augmentation, and then looking at the MLflow pipelines, tried to increase the hyperparameter configurations as well. So what you need to understand from an MLOps perspective is is your
[21:14] pipeline able to do exactly that? create a journey of how you went from zero to hero or how you were able to achieve the state-of-the-art run. But the story does not end here.
[21:27] Our objective is to move from dev. Yes, we soda soda model state-of-the-art model, but does it give the same performance in production? And this is where we learned a painful lesson as
[21:39] well. The moment we went from 0.53 and put it first into a uh model serving endpoint, it actually crashed back to 0.03. We fixed that bug and then we
[21:51] again tried to get the sort performance. It actually again crashed and we found another bug as well. Which then begs the question, okay, you've trained your dev model. How do you add checks and balances and have an entire promotion
[22:05] logic that can safely guardrail from this happening in production? So it becomes the question what are all the other auxiliary tools that we need in the pipeline. Now I'm going to jump a
[22:19] little um a poll very quick poll. How many of you have seen this? How many of you have actually looked at the big book of MLOps that data bricks has published? Superb. I see quite a lot of hands um
[22:31] you know hold up. Now my point over here is the M book big book of MLOps is amazing and one of the things that you immediately notice with this diagram is we actually prescribe deploy code which
[22:44] means we say you have a workspace environment where you do your dev testing you have another environment where you do UAT and we have another environment you do the production but if I'm running H100 clusters for 16 hours
[22:58] the costs actually prohibit me from doing this exact paradigm So we need to flip the script a little and as in the big book of MLOps it's written you do not always need to go into a deploy code approach you can do a
[23:10] deploy model approach as well and historically it had been very challenging to sort of get that setup done within data bricks but now that has actually changed so what's the key factor it's essentially ensemble of
[23:24] these six capabilities we've talked about serverless GPUs uh AI runtime we've talked about ML flow 3.2 which gives you the proper tracking and training but we still had a issue with
[23:36] how do you move from a dev experiment to a production experiment where you add all the checks and balance uh checks and balances and the approval logics in place.
[23:48] So unity catalog is that final frontier which helps us do that at the bottom. It allows you to then take a prod model, serve it at scale with model serving, um
[24:00] create embeddings using AI search and if you want to do data quality modeling within uh AI search, you have inference tables as well and we'll just get into a little bit detail in that as well. But
[24:12] what I wanted to do was right using all of these six capabilities, how can we move into a gardenwalled MLOps paradigm, which is do a sweep, log the model. As
[24:27] soon as the model is registered, I have a bunch of evaluation pipelines which grab the model, run it as if it was running in production, tell me if I've beaten my champion. If I have beaten my
[24:39] champion, do an approval process flow, trigger a email. Once someone approves it, move it into a production schema. From there, deploy to an endpoint where
[24:51] I can either do AB testing to do a staggered retrieve or replace the current existing champion and before doing that do one one final smoken nest to get that to run. Right?
[25:04] So, let's see how we went about implementing that.
[25:21] if I go into my catalog you'll see that I have simulated
[25:35] There you go. >> It's also different. >> Yeah, this is >> cool. So you can see what I did. I simulated two schemas. So you can see this as my dev schema. In this schema,
[25:48] all data scientists, ML engineers, scientists have the access to write create models. Whenever I do a sweep, that sweep will automatically register a model within Unity Catalog. And it's
[26:02] simple as mlflow.log model. You specify the Unity catalog hyphen name. Now for people who don't know Unity catalog 10-second overview, it's the governance framework within data bricks which not only provides permissioning for tables
[26:15] but all of your ML and AI assets as well. So what happens now is you can see I've created two model pipelines C radio and the Dino V3 which were my two premises. I could have had one model but
[26:27] just for the purposes of this talk I segregated those out. Now when you actually go and click into a model, what you will see is as soon as you register a model, we now have something known as deployment jobs which get triggered as
[26:42] soon as a model is registered within the Unity catalog experiment. And let's look at a couple of runs. So if I look at a run over here, you will see it kicks off an entire evaluation, approval, and
[26:56] promote chain. Right? So in the evaluation chain you can see what I have done is I have added a guardrail to say does it actually per
[27:15] model on the evaluation data set. If not fail the experiment if it does then go and ask for an approval flow. So if I then go back
[27:28] you can see over here let us say you were able to beat the soda model. You then have an approval check process integrated wherein the originator can actually go in and he can actually start seeing all of the
[27:41] data sets the source run as well as the evaluation value from within unity catalog itself. So if you wanted to do the dev prod u segregation you could do it now with unity catalog because mlflow
[27:54] 3.0 logged run essentially works cross workspace as well and as it's as simple as going in to a previous run and then saying approve. Once you do the
[28:07] approval, you will see that the entire process succeeds. And what it would do is coming back to my production schema,
[28:22] it automatically registered the model by the name of champion candidate and it automatically adds the tag champion. As soon as a model endpoint is running, I have hit 10 correct individual results
[28:34] and match them with my ground truth. But that's not only it. I can then actually create an entire pipeline. If you just go into this pipeline, just
[28:47] doing a quick check. Yes, I can actually create an entire pipeline where I not only deploy the champion, I then repute my embeddings. I recalculate my vector index and I have my drift and I have my
[29:02] bas uh baseline drift detection metrics integrated for continuous online monitoring as well. So it not only updates my model endpoint but also initiates all of my auxiliary categories
[29:15] as well and all of this is automatically triggered because these are version controlled. Roll back is as simple as changing the alias to a previous model and then going into that particular
[29:27] experiment and then just redeploying the pipeline as you can see on the top left. So this allows us to now come back and actually
[29:42] run this at scale. So now you can see we've completed all the steps. We've done the sweep. We've logged the model if it's better promoted
[29:55] to the dev environment. Um and one last thing uh very quickly just to sort of round out the entire conversation. You can see over here that the lineage is absolutely maintained between the dev
[30:09] and the prod environment. So on the right you can see whenever I promote an existing model from my dev schema to my prod schema the lineage is automatically captured and you can exactly see from where it was copied what was the
[30:22] underlying sweep run that triggered it and you can get all the telemetry and parameters out of the entire equation. So everything is governed automated end to end. You can keep on generating hypothesis, rolling that end, test it
[30:35] again different data sets. All of that is within Unity catalog completely discoverable across everyone. Now I'll hand it over to Michael um to sort of do a live demo of how to trigger this entire pipeline end to end. Thanks
[30:47] Punet. Okay. So now we will take a look at how we can also use uh a runtime CLI to trigger pipeline or to basically trigger a training run. So
[31:00] let me
[31:12] uh here uh you can see that we can just uh write a uh air run minus f. We just uh put their path uh to our yl configuration and we just need to select uh the right workspace. So I will just
[31:25] trigger it and it's going to request uh GPUs on demand and do the training. So as you see at first it's going to package my code like basically the whole
[31:37] uh repo uh transfer it to the workspace and uh kick off the run. Okay. So while it's doing that I think we can uh take a look at the oh as you see uh is we already have the run. We can also click
[31:51] on this run and you will see and then like we automatically get the link to MLflow. You can see we have here a run. Uh it's a job run. So we can click here and
[32:03] that's going to be an MLflow run. And here in artifacts we are going to get uh the logs and the artifacts. We're also going to get the model matrix here. But now it's empty. It's not
[32:15] running yet. Okay. So let's uh get back to the ID. Uh maybe let's make it big and let's close that. So what I'm uh going to show you now is how you can
[32:27] configure uh the job using the air runtime CLI. So what you can do you can put here uh experiment name also name for the MLflow experiment. You can put the environment
[32:40] variables if you want. So here like I have activated NCCL debug. Uh sometimes uh it's like nice thing to see. Uh what's really interesting uh you can uh
[32:52] select number of GPUs you need you can select the node type you're going to use here we are going we are using uh machines equipped with eight H100s
[33:04] and there are a bunch of other things so uh here we have code source you can either go for snapshot basically means it's going to compress what we have here or you can say g commit you can specify
[33:17] the exact g commit you would like to use. You can also specify the G branch and then we have parameters right those parameters uh they are custom they those parameters will be uh
[33:32] processed uh by your job and we have a comment right so here I'm just using directly torch run you see here a bunch of uh variables those variables are uh set by
[33:47] the runtime time and if you're using not torch run but maybe accelerate or something different you still can use those uh variables and now let's uh really quickly see how the CLI looks
[34:00] like like if we if we go down you can see that I'm just creating trainer and pasting the configuration
[34:16] and here it's uh basically relies on this hyperparame ameter pass that's the uh environment variable which will be set also by the runtime and it will have the yaml with your custom parameters okay so now let's see how our run is
[34:31] doing as you see something is already happening so we have our GPU we have installed all the required packages and it's already running so let's check out
[34:46] Here we might have some matrix. Oh, we don't have any matrix yet. Okay, but we have some locks. Basically, it's all going to come. I don't think
[34:58] you have time to wait for everything. Basically, it's running. Okay, so uh coming back to the slides.
[35:16] Uh we need we were going to Yep. Sorry uh my bad. So very quickly uh this >> yeah so uh apologies. So as you've seen the entire AR runtime paradigm um if we now do a very high level caveat of our
[35:29] typical envelops template these are the pattern deviations that you can do and now within unity catalog you still get the governance layer that you require um rather than being very rigid and doing the entire retrain across every
[35:42] environment. So in terms of environment we saw you can have either were two schemas, three schemas, three cataloges, two cataloges. You can completely customize the process. I went with the
[35:55] one that was very tricky which was one workspace and two cataloges. But with delta sharing you can have multiple workspaces as well. Um promotion gate as I said I we try to re-evaluate everything on the evaluation data set.
[36:08] You could have your own test data set segregated to make sure that there is no online training offline evaluation skew. Um and then finally monitoring because this is a foundational model caveat you if you have the ground truth you could
[36:21] just use uh data model monitoring within data bricks or you could come up with your own custom logic as well. Uh finally all of this is essentially available out of the box um in terms of
[36:33] you can package all of this up into what we call asset bundles. So it's as simple as data bricks bundle deploy which can be integrated within your CI/CD system or any automation software that you
[36:45] have. You could manually trigger it as well set up the entire scaffold or you could have the entire CI/CD best practice logic integrated into the architecture as well and
[36:57] >> okay >> I'll just hand it back to Michael. >> Thank you. Thanks Pony. And I think uh very interesting point we have uh a repo in GitHub where you have all the code
[37:09] which we have used and which we have just shown. So feel free to clone it run it on datab bricks air runtime. You can also easily apply to a different uh object detection task. You can easily
[37:22] swap the backbone if you get a even a better one. If tomorrow an even better one comes, you can easily swap it and use a better one. And yeah, feel free to use it for your projects. Thanks
[37:35] everyone. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.