The Right Model for the Job: A Practical Framework for AI

Summary

This video introduces a three-step framework — profile, evaluate, and govern — for systematically selecting the right AI model for each enterprise task across customer support, document processing, and analytics workloads.
The shelf life of a top-performing LLM has compressed to approximately two months while open-weight models now trail frontier performance by only about 3.5 months, shifting the economic calculus toward open-source for many production use cases.
A live demo shows fine-tuning a Qwen model on a single A100 GPU on the Databricks Data and AI platform, achieving performance matching frontier models at approximately 40 times lower cost for a high-volume classification task.

The Right Model for the Job: A Practical Framework for AI

Not every problem needs the biggest model — and not every model works out of the box. With frontier models, small open models, and fine-tuning all on the table, the hardest question isn't "can we do AI?" It's "which model is actually right for this?"
In this video, we walk through how to match models to use cases across real enterprise workloads. You'll see when a frontier model earns its cost, when a small open model wins on latency, and when fine-tuning turns a generic model into a specialist that knows your business. We'll put the tradeoffs in action — quality, cost, latency, control — using concrete examples across customer support, document processing, and analytics.
You'll leave with a repeatable framework for choosing, customizing, and deploying the right model on Databricks every time — and the patterns to keep those decisions sharp as new models land.
Talk By: Nuno Castro, Director AI FDE, Databricks ; Viktoria Semaan, Principal Technical Evangelist, Databricks ;

Chapters

FAQs

How do you choose between frontier and open-weight AI models for enterprise tasks?

This video recommends profiling tasks into categories — easy and repetitive versus complex and ambiguous — and matching model capability to task requirements. For high-volume, well-defined tasks like classification, open-weight or fine-tuned models often match frontier quality at a fraction of the cost, while complex reasoning tasks may justify the premium.

What is the three-step model selection framework presented in this video?

The framework consists of three steps: profile (understand the task characteristics and volume), evaluate (compare models side-by-side on accuracy, speed, and cost using MLflow), and govern (manage traffic splits and model routing in production through the AI Gateway). This structured approach helps teams make repeatable model selection decisions as new models are released.

How does MLflow support model evaluation and comparison in Databricks?

MLflow on the Databricks Data and AI platform enables side-by-side comparison of multiple models on accuracy, latency, and cost using the same evaluation dataset. This video demonstrates using MLflow evaluation to compare frontier, open-weight, and fine-tuned models so teams can make data-driven selection decisions.

How much can fine-tuning reduce AI inference costs?

This video demonstrates fine-tuning a Qwen model on a single A100 GPU for a high-volume classification task, achieving results that match frontier model performance at approximately 40 times lower cost. Fine-tuning allows a smaller model to specialize in a specific task, dramatically reducing token costs for production workloads at scale.

Full transcript

[00:09] Before we begin, just a quick disclaimer that any forward looking statements we they're not guaranteed of product maps or features. This is based on market insights, trends. So, uh please don't
[00:22] take it as a guarantee. uh and if you could at the end of the talk complete a quick survey we would really appreciate this because this will be considered whether we will come back next year on
[00:34] the stage. Uh so with this let's uh start. So we have really interesting topic today and we will talk about the right model for the right job. My name is Victoria. I'm a principal technical
[00:47] evangelist at datab bricks and today I'm joined by nono who is a director of forward deployed engineering. So we will talk about a shift that we are seeing now in generative AI in
[01:01] particular in architecture. So over the past two years the default for building AI applications has been uh utilizing API to frontier models and build your
[01:16] your stack. But as we start um having more production volume now we have more concerns regarding cost optimization.
[01:28] So as uh business grows with a usage now uh we we talk more about optimization per task uh token optimization token maxing. So today we're going to address
[01:40] how you can build robust evaluation and LM diversification. So in this session we will deliver a framework that will help you to systematically evaluate
[01:52] optim and optimize your models for your specific task. So if we think about the pace of innovation now it's uh truly mind-blowing and this chart it's
[02:04] basically represent how long each particular LLM stays um at the top before it gets replaced and as you can see a couple years ago this span was
[02:17] about 12 to 18 months and it just starts compressing and compressing so now this shelf life of LLM to stay on top of benchmarks is around two months and on
[02:30] top of this the capabilities of those LLMs they are doubling every seven months and what's also interesting happening that opensource ecosystem is
[02:42] growing very quick so those open weight models they now behind proprietary models by about three and a half months and they're achieving sima intelligence
[02:54] so Those statistics are from artificialanalysis.ai really great website where you can find more. I think one of the interesting uh case was recently with uh Corser. So I'm
[03:07] sure many developers tried Corser IDE uh for coding tasks and they originally built their coding assistant by utilizing proprietary models open AAI
[03:19] and entropic and the business models based on subscription right so the more developers started adopting those AI tools the cost per task became
[03:31] unsustainable because now so many tasks are being done right but you're still charging the same subscription fee. So they realized, you know, they need to change something otherwise they're going to be continuing to be underwater. So
[03:46] what they've done, they actually then fine-tuned an openw weight model Kimmy on the specific tasks. So they are for of course coding. So they looked at
[03:58] their prompt response loops and optimized their pro tasks specifically on coding and this helped them to maintain benchmark while at the same time achieving 86% drop on token per
[04:12] task. So what we want today want to talk today is about how your business can also build sustainable model so that as usage grows um it will still deliver uh
[04:26] net profit and result in return on investment. Um so today during our session we will talk we built a few demos to help you to understand how to applies this
[04:38] framework. So we will be utilizing uh example of brickster sport company. So imagine now there is a football season or soccer season uh and you have an application where different fans from
[04:52] different countries are sending messages and interacting. So our job is to moderate this chat. But moderation actually means many different things. Let's say a lot of the images we will
[05:06] just let them go through. Some of them might be a spam, some of them might be abuse and they will be really easy to differentiate by class utilizing a classification models and right away
[05:19] ban. If there are repeated offenders, maybe we can also b those users, maybe we can issue warnings. But uh some messages sometimes might be really hard to differentiate because is this a joke
[05:32] or this is a thread? Is it a sarcasm? And that's where we might need to utilize a different model or even escalate to human reviewer. But escalating to human reviewer, we need to
[05:44] provide context. And context is not just the message itself. It's also what happened during that game. Uh what was said in previous conversation. So we need to provide all of this context whether it's to a advisory LLM or a
[05:59] human reviewer. So uh in this session we will talk about this two specific tasks and we will try to identify for each of those tasks which model we need to use. So now you can try to guess which model
[06:12] is going to win and we will see if your guess will be right at the end. Okay. So uh what we will walk through is a three step framework. So first we'll
[06:25] talk about different models and their trade-offs. How you can utilize evaluation to really understand where you need to make those trade-offs based on your own use case. Then we'll talk about how to govern and also switch
[06:39] models utilizing AI gateway in production. And finally we will talk about when it's time to really think about fine-tuning and build your own intellectual property. and then we will
[06:51] have some resources to help you to get started. Uh so first let's just audit the kind of like ecosystem to understand what we have um in the market. So currently
[07:04] actually over 37% of enterprises are already utilizing over five models in production and 37% of all token volumes
[07:16] already coming from open weight. So the prop uh this ecosystem open ecosystem is really being utilized now and it's growing. So you can also see from this
[07:28] chart the blue line is how quickly uh open weight models are catching up with intelligence of proprietary.
[07:41] Um this was also a recent chart that I took just before our talk to look at what are those different models and you can see how many model providers are there. So there are many model providers
[07:54] from proprietary there are many model providers not only on the US but also in Europe in China. Uh so the Kim model for example was developed in China. So the ecosystem is growing and of course those
[08:07] choices are great but then they also introduce okay so then how do I determine which model I need to go to so the question really shifted if we two
[08:19] years ago if you were here everybody were asking okay how do I get started but now it's more about which model is right for me and as model continue to change so quickly how do you actually
[08:32] keep your application layer decoupled from this model serving layer so that every time the model changes you don't have to rewrite your application.
[08:44] So if we look at the overall spectrum like I I mentioned a couple times we can look at let's say three types of arch archetypes is proprietary open weight
[08:56] and fine-tuned you can utilize all those three archetypes on datab bricks if you go with proprietary models actually data bricks is the only platform that allows you to access any model right there is
[09:09] no vendor locking you can go to entropic you can utilize gemini you utilize open AAI you pay per token. So you basically yes per pay per API and then
[09:22] with open weight models you now have a choice you can pay per token but you can also host this model and then pay for compute. So depending on your use case,
[09:34] depending on your usage, you might select which uh way it will be better for you, whether it's a paper token or uh self-host. And then the third is fine-tuning. So that's where you can
[09:47] fine-tune model. um you basically download the open weight models, you utilize your data on data bricks, you fine-tune and now you have this intellectual property that you register with unity catalog and now it's not
[10:02] you're you're creating not only that the traces right the basically all this data that's being generated it's your own you are creating really your business differentiator
[10:15] uh couple examples uh one of our customer uh Grammarly I think they now called superhuman uh they fine- tuned a small openweight uh llama model and uh on data
[10:28] bricks and of course it resulted in quite large savings. Uh so besides of course uh this three archetypes we can differentiate models by size. Um so if
[10:41] we look at uh clo right they have opposet and haiko uh those three different sizes they also come with their own uh tradeoffs if we are getting
[10:56] more intelligent model as oppus yes it will excel in terms of intelligence and in in reasoning but it also comes at a higher cost and higher latency and um
[11:10] this is a great framework to get started but of course all of this is not enough to actually find out which model will be right for your use case. So now I'm going to invite Nuno who will talk about
[11:24] how to build evaluation suite so that you create your data sets and you can systematically select and test different models and identify what are those different tradeoffs uh based on your use
[11:37] case. Thanks, Victoria. I'll take step one, building the evaluation. Before you can choose between the models, you need to define
[11:50] what good looks like. You need to set your quality bar or your yard stick. And this is something you do once and you can use every time a new model is released.
[12:02] And the way you define the quality bar is with MLflow, the platform of choice for building highquality AI systems and agents and ML models. It's open source,
[12:14] 25 million downloads. It started just as an experiment tracking and grew into the end toend loop that you see on the right. Evaluation, model registry, serving, visualization,
[12:26] all in one. And that's what we're running today. The pipeline is four steps. Step one, you build the evaluation step in stored
[12:38] in Unity catalog. Two, you define your scores. It could be a simple Python function if you're looking to evaluate an exact match, a plain language guideline, an aristic
[12:51] or an LLM judge. It depends on the use case you have. Three, you actually run it. You just execute one call to ML flow and you get one trace per row. And
[13:04] finally, you can analyze the results. One row per model where you can sort by accuracy, speed, and cost. And look at the trade-offs. The winner might surprise you. In our example, we
[13:17] actually run 16 models on the task that Victoria showed, changing one simple line of code every time. We'll now go through these four steps in
[13:29] our Brixster cup moderation demo. In step one, you create a high quality evaluation set. As we see, there's eight
[13:41] labels such as clean, off-topic, harassment, and hate speech. And we also have the examples and easy examples. Step two, you define your scores. In this case is a simple Python function.
[13:56] Step three, you actually execute the MLflow run. And finally, you can analyze the results, which we'll look into a bit more detail later. Jumping into the human escalation
[14:08] reasoning, we do the same four steps with some small changes. First, you create the evaluation set. And you can see here it's a bit more elaborate. We need to give the user history so that
[14:21] the human reviewer can look at it. Step two, you define the scores. In this case, you're actually trying to compare an escalation semantic similarity instead of an exact match. So, you're
[14:34] not going to use an uh one or zero. You're going to use a one to five. And we'll use OPOS 4.8 as our judge for the best quality possible. In step three, you actually execute the
[14:47] flow with your score on your data set. And finally, you can look at the results. Again, I won't spoil them for now. Victoria, you will show them a bit later. The valuation you just run is now
[14:59] available in MLflow experiments tab almost immediately where you can see all your runs. Here are the 16 models we run against each other increasing the metrics from accuracy, latency, cost or
[15:12] speed to to cost per million messages. And finally, you can also look at the traces individually. So trying to understand for each individual message or model understand in more detail what
[15:25] happened, what was the latency, what was the moderation decision and whether it was correct or incorrect.
[15:44] MLFlow runs evaluations in two modes and both are super important. Mode one is automatic. Every time a model changes or a new model is released or you change the prompt to improve your system, you rerun the evaluations programmatically,
[15:57] which means they can be automated. And mode two, not less important, is the human evaluation because the code and the programmatic cannot catch every single nuance. In some case, you
[16:09] actually need a human in the loop to look at the runs and review them. Databicks provides this review app where human experts can review the traces and log their corrections and those
[16:22] corrections sync straight back into your data set. So every correction is continuously upgrading your evol over overall evaluation. Automatic gives you speed and scale and
[16:36] human gives you the ground truth you would apply to the critical tasks for example or where the model is less confident. Together the evaluation keeps get better and better.
[16:51] Back to the first principles, evaluation is a continuous loop. So it shouldn't be something you do just once. You should do it every time you change any part of the system. Could be the models, could be the prompt. You test it in
[17:03] development, you deploy it, and you look at the results. The edge cases are flagged for review and they're joined back into your test set. So when a new model drops next month, and it will, you
[17:17] don't start over. You just rerun the evaluations. Victoria, over to you. Now that we can measure models, let's access them all.
[17:29] now let's talk about how we can govern those models and how we can set a traffic split. So let's say you actually ran this evaluation and you identified maybe a few candidates. You don't need
[17:41] to make a decision. Okay, this is the only models that I'm going to use. You can split your traffic. You can put three models, right? and then maybe also configure some fallback models and then as you deploy them with uh Unity AI
[17:55] gateway you can also see how those models perform in life and therefore because maybe your training data set and your live data set might you might find some differences right so it's not that
[18:08] you uh you did this once and then you forgot about this you can continuously get in those live traces and analyze and then shift traffic accordingly So u this is actually what we are seeing
[18:21] that right now most enterprises they don't default just to one model they are utilizing several ones and that's why uh creating attaching yourself just to one
[18:33] provider is actually becoming a risk because let's say uh there's a new provider that comes back comes with this amazing intelligence but now you cannot use it because you you stuck with one
[18:46] ecosystem. So that's why many customers go to data bricks that because we try to eliminate this vendor lock and really try to help you to utilize the best model uh for for yourself. Uh what also
[18:59] Unity AI gateway provides uh in addition to accessing different models it actually provides you this one place where you can observe all of those
[19:11] traces. You can configure rate limits. you can uh configure uh permission to different models maybe let's say for marketing you'll say you know I think marketing will be fine with a haiku but
[19:24] my engineering let's give them access to oppos right so as an example so you can make a lot of this uh configurations inside unity AI gateway um also uni
[19:37] gateway is not just for models it's really access to your entire AI estate Um, and this can be also coding agents. This can be MCP servers. And those MCP
[19:50] servers are not just data bricks MCP servers. Those are also partner MCP servers. So think about this as your one place where you can open and see how your entire organization is utilizing um
[20:04] AI and how you can make those cost adjustments. So recording is actually a very good example. Um so as developers we don't use just a one coding tool uh anymore
[20:17] right so we might start with IDE like cursor and we might utilize in cursor some some agents then we might switch to terminal and then cl code do some work
[20:29] and then we can go to geni and then troubleshoot and fix things in genie so we we actually jump in quite a lot and uh basically what it creates then there is no a single place to understand how
[20:42] developers are utilizing the the models, right? Where where the cost usage is growing. So you can power your AI coding tools, your AI agents also also through
[20:54] uh Unity AI gateway. So you basically point code code to the endpoint in your Unity AI gateway and now you can utilize code code or whatever tool uh you want and now you will have this one dashboard
[21:07] where you can go and see how different teams are utilizing them. Uh so how do we actually start? So we run let's say this evaluation and we see uh metrics.
[21:20] Um what we see a lot from customers there's a lot of excitement when the new models get dropped on the market and like oh my god there is this formal I need to start utilizing this uh model but what the really the thing right
[21:35] thing to do is that maybe you don't need a PhD degree to solve a very simple problem right so you can serve it with intern so this the same thing with um
[21:48] those models You can start with something simple then run it against your evaluation. If you clear your quality benchmark then why to pay for more? If you if it doesn't clear
[22:01] then you can climb up right and utilize a larger um more intelligent models and as new models drop just run them in parallel. put put the traffic split you
[22:14] know as a new model release maybe put 10% of your traffic and see uh what what will be a difference another thing to consider that maybe for some of you the 5% difference in accuracy
[22:27] some will say oh this is actually not going to make an impact on my net income but for some that because this 5% can actually generate maybe millions of dollars you'll say no I actually want to
[22:40] increase uh a spend by 20 times because I will get this marginal 5% improvement. Uh so now let's uh review actually dashboard. So Nuno explained about how
[22:53] he created this notebook and run MLflow evaluation. So after MLflow evaluation run, we can review dashboard. So here's an example of dashboard. Basically we have the um F1 this was for
[23:06] classification model F1 metrics and accuracy metrics. So as you can see in the red uh those code metrics they are quite high but also there is some models like Gemma 3 and GPT 54 54 also quite
[23:22] scoring quite high and look at the difference in price right like OPUS compared to Gemma we actually don't even see where Gemma is with with a cost because the difference is so high and the same with latency
[23:35] so now we can look at all the metrics together and as Yes, you know, I think it's not surprise. Oppus 4.8 bit in terms of accuracy, but also look at the cost. So 0.94 accuracy with $3600
[23:52] approximately per 1 million tokens. And then with let's say GPT54 nano, we got 0.085, but it's $9. So it's basically a difference of 40 times, right? In terms
[24:07] of cost. So that's where you need to decide well does this justify for me to spend 40 times more to gain this accuracy and depending as I said on your
[24:20] business metrics and where you implement for some will be the answer yes for some the answer will be no. So now let me just do a quick walk through to show how you can configure this in Unity AI
[24:34] gateway. So here I'm in Unity Gateway. So the first thing that you do you just click create an endpoint then you search for you put the name and then you search for different models. So you can see
[24:46] here the list of different models uh a very high level speed and cost. You can add as many different models as you want. You can then configure the traffic
[24:59] split and uh also configure fallback models. Another thing that you can do, you can set up rate limits. So you can set uh rate limits of tokens per minute and
[25:12] query per minute and you can get very granular. You can get it on the group level on the user level. So this um provides you this governance. Uh once
[25:24] you set this up, you will be able then to monitor usage of this one endpoint. So at the end you're going to have one endpoint and then the the traffic will be split. So here uh let's do you can
[25:38] test it very quickly. So I'm just putting a spam message and you can see right away I got the response. Yes, it was spam. And then I can go to Unity Gateway dashboard which is a pre-built dashboard for you. And here you can uh
[25:52] analyze performance usage uh co coding agents um and let's say if you look at this dashboard and like I I have some other questions you can just simply ask
[26:05] junior so this was actually a dashboard of my team and I said you know how what kind of model Victoria is using the most um and it came back that I'm utilizing
[26:18] in over 69% oppus. So maybe I'm not the most
[26:31] including my manager. Um all of this metrics I stored in system tables. So you can review the system table AI gateway. So you can build your own
[26:43] dashboards, your own genius spaces and get this very granular information. Well, now that we understand how to evaluate models, how to put them in
[26:55] production, I think the last question that we need to answer is, okay, but when do I actually need to fine-tune model and create this intellectual property myself? So, that's when I'm
[27:07] going to call Nuno. >> Thanks, Victoria. Let's hope your manager doesn't watch this presentation about all those tokens. So, we've got a cheap model that's fast and often incorrect and a frontier model
[27:19] that's right but expensive. Step three is about the next option, the third option, fine-tuning a small open model. First, let me be clear. Fine-tuning a
[27:33] model should never be the first thing you do. Most of the time, AI gateway and a good prompt are more than enough. You reach for fine-tuning when these three
[27:45] signals align. One, mistakes are repetitive, so the model keeps missing in the same way, the same wrong category, the same broccoli format, the same tone. Two, you have labeled
[28:00] examples to fine-tune your model. A few thousand good input output pairs are often more than enough. And third, you have maxed out on prompting or rag.
[28:12] Fine-tuning will eventually help you reduce the context size and co cost per call, but is a lot more complex to maintain over time because you need to keep retraining the model. But as
[28:24] Victoria said, it's all about the trade-offs between accuracy, speed, and cost. When a small model cannot meet your quality bar, in a frontier model is too expensive or
[28:36] cost prohibitive, fine-tuning becomes an option. One of our customers, Canar, was trying to match received strings to actual product names, and they started
[28:50] with a Frontier model. The model was working great, but because of the volume, it was becoming too expensive. So, they actually decided to fine-tune the model and obtained a 94% accuracy
[29:03] for a fraction of the cost. Historically, setting up a fine-tuning infrastructure was a huge problem on its own. You
[29:15] needed to find the GPUs, stand up the whole infrastructure, and watch and pay for it while it was sitting idle. Once all that was in place, actually
[29:27] training the model was the most straightforward step. Data bricks has solved this problem by introducing AI runtime. is basically serverless GPU compute. So you basically
[29:39] open a notebook, attach a GPU could be A10 or H00 or multiple H00 as a cluster and because it's serverless you pay only
[29:52] what you use for the seconds while it's operating and that includes scaling to zero. This is how it works. You basically pick a small open source model could be a
[30:05] quen or a llama or a kimi and it put us straight your data straight from unity catalog. You then run your parameter efficient fin fine-tuning could be Laura or Qura
[30:17] which basically freezes the base model and optimizes and trains a new set of weights. So it's very fast and cheap. The run is of course also tracked in ML flow registering the model in Unity
[30:31] catalog and deploying it as easily to a serving endpoint. I'm now going to show fine-tuning a quen model for the bricks up moderation task on a single A10 serverless GPU with AI
[30:46] runtime. You basically select the GPU in a few clicks.
[30:58] After a few minutes, you can see it's up and running. You then install all the necessary Python libraries to train your model and also configure where to get the
[31:12] training data set. In this case, we're fine-tuning a Quen 1.7B model. You then download the data set from agging phase and you only need to do
[31:25] this step once and load the training set from disk. You then load your model.
[31:39] And this is a step where you actually set up the lower parameters like the alpha and the dropout. And you can see it loading the weights in real time.
[31:57] And finally, you can run your training script, your finetuning script. As you can see, MLflow is embedded in every part of the way.
[32:09] While it fine-tunes, you can see the training loss going down as the model, the finetune model is converging. After a few minutes, the model is trained and you can see it actually
[32:23] uploads the artifacts as I said. Then if you can go to MLflow immediately, you can see all your training metrics. So you can see things like whether it finish correctly, how much memory or GPU
[32:38] it used in the training process. And if you're the model geek, you can actually go into the model metrics and understand if it's converge, if there was any problem with the training of the model
[32:51] so that you can actually deploy it. So the next step is actually deploying that model to a serving endpoint. With just a few lines of code, you actually train a whole model, fine-tune
[33:03] a whole model, and deployed it to a server endpoint. After a few minutes, your serving endpoint is up and running and you can immediately start throwing
[33:17] requests at your API. You can also look at all the metrics such as memory usage, response rates,
[33:29] and also what kind of compute you have. Now the part that matters since we have a new model, we need to eat our own dog
[33:42] food and follow our framework. Looking at the the dashboard that Victoria showed, we now have a new data
[33:54] point in our dashboard, which is our fine-tuning dashboard. You can see how it does against all the other models. You can look at the cost, the F1 scores and all the other metrics
[34:07] that matter. For example, if you look at the accuracy, we achieved an accuracy of 91%. If you
[34:19] remember, OPUS 4.8 was at 94%. So, it's actually extremely competitive. It's one of the best models. But if you look at the cost is only $8 versus $3,600
[34:32] for the open model. Latency at 255 seconds is also much faster than any other Frontier models. For example, OPUS is about 1 second and all the other
[34:46] models are at least half a second. But here's a surprise. We only solved one job. If you remember, there were
[34:58] two. The second one was reasoning. The escalations only about 1% of the volume. The messages where there's some ambiguity where we escalated for human review. The system has to do some
[35:11] reasoning. So it's not just a classification. And because it worked the first time, we also tried fine-tuning a model for this use case. It worked for the first time. So watch what happens.
[35:25] The fine-tuning specialist showed 1.3 out of five. If you remember, we had set up an LLM judge for this step. The Frontier model goes the other way.
[35:37] So, Sunonnet shows a 4.7 rating and OPUS a 4.5 rating. So, this is super interesting. How can a ous 4.5 show worse accuracy even when OPUS is
[35:52] actually the judge if you remember? So this is a good reminder to not follow any assumptions and actually do a professor a professional evaluation every time. Looking at the cost is
[36:04] clearly that you need to pay for this accuracy right at $68 per million messages versus uh 50 cents. But in this case, it's worth it because you only call it 1% of
[36:18] the time. So let me bring it on. Let's summarize the framework we discussed today. First build evaluation before you pick the model with MLflow is
[36:32] actually the cheapest thing you'll do and the most important inputs, expected outputs, hard cases, traces, scores, and human feedback. Models will keep changing, but every new model should be
[36:45] measured against the same quality bar. Step two, you can access and govern your models through Unity AI gateway. As Victoria said, start with the smallest, cheapest model and climb only when the
[36:58] evaluation requires it. You can easily add new models to AI gateway, split traffic, watch the traces, and feed your human feedback into your evaluation to
[37:10] improve data sets over time. And finally, only when the tasks earns it, you do the finetuning. It's a high volume task. You got labeled data and you have maxed out every other
[37:23] option. That's when data bricks AI runtime open trans turns a open model into a specialist that you can control and that's yours
[37:38] to support your implementation of this framework. We provide you the following links. The first one is AI devkit. AI defit helps you build all of these that we showed with vibe coding. It really
[37:50] understands the datab bricks documentation and products. So it's better than a generic tool. It's going to accelerate the process of moving this into production and you don't have to start from scratch. For MLflow, we have
[38:04] a tutorial on how to get started for Genai. And finally, some example notebooks and how to fine-tune models as well. So decompose your workload into jobs and
[38:19] evaluate the best model for the job. No model will win every time. No model will win every axis. So you need to choose the axis that matter, not the brand. Right model, right job every time. Thank
[38:34] you and enjoy the reminder of Data AI Summit 206.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.