The Right Model for the Job: A Practical Framework for AI

Summary

This video presents a three-step framework — evaluate, govern, and fine-tune — for selecting and optimizing AI models on Databricks, addressing real cost-quality tradeoffs as open-weight models now trail leading proprietary frontier models by only about 3.5 months in capability.
Step 1 uses MLflow to build an evaluation dataset and a human review app for chat moderation, while Step 2 shows how Unity AI Gateway lets teams govern models, configure split traffic, and apply rate limits without changing application code.
Step 3 demonstrates fine-tuning a specialist Qwen model using serverless GPU on Databricks, then compares it against a frontier model to show that a purpose-built smaller model can match or exceed quality at significantly lower inference cost.

The Right Model for the Job: A Practical Framework for AI

In this video, we walk through how to match models to use cases across real enterprise workloads. You'll see when a frontier model earns its cost, when a small open model wins on latency, and when fine-tuning turns a generic model into a specialist that knows your business. We'll put the tradeoffs in action — quality, cost, latency, control — using concrete examples across customer support, document processing, and analytics.
You'll leave with a repeatable framework for choosing, customizing, and deploying the right model on Databricks every time — and the patterns to keep those decisions sharp as new models land.
Chapters:
00:00 - Trends in AI Benchmarks & Shift in LLM Architecture
06:20 - Agenda
07:00 - Understanding Model Archetypes and Trade-offs
11:38 - Step 1: Building an Evaluation Set with MLflow
13:26 - Demo: Chat Moderation Evaluation and Review App
15:40 - GenAI Evaluation Framework
17:29 - Step 2: Governing and Switching Models via Unity AI Gateway
22:30 - Demo: Gateway Configurations and Endpoint Split
27:08 - Step 3 Fine-Tuning a Specialist Model
30:37 - Demo: Serverless GPU Fine-Tuning Script
33:38 - Demo: Evaluation Comparison (Frontier vs. Fine-Tuned Specialist)
36:21 - Key Takeaways, Summary & Resources

Chapters

FAQs

What is the right model selection framework for enterprise AI workloads?

This video introduces a three-step framework: first build an evaluation set to measure model performance on your specific task using MLflow, then govern and switch models centrally via Unity AI Gateway, and finally fine-tune a smaller specialist model on serverless GPU when a frontier model's cost is not justified. The framework provides a repeatable method for balancing cost, latency, quality, and control as the AI model landscape keeps evolving.

How does Unity AI Gateway help with AI model governance?

Unity AI Gateway acts as a centralized control plane that lets teams configure which models power each application, apply rate limits, and route traffic across multiple endpoints — including splitting traffic between models — without any changes to application code. This makes it straightforward to A/B test models or switch providers in response to cost or performance requirements.

When should you fine-tune a model instead of using a frontier model?

Fine-tuning is worth considering when a frontier model is oversized for a focused, well-defined task — such as chat moderation — where a smaller specialist model can achieve comparable or better accuracy at a fraction of the inference cost. As shown in this video, fine-tuning a Qwen model on serverless GPU on the Databricks Data and AI platform and then evaluating it against a frontier model can validate whether the specialist is ready for production.

How fast is the AI model landscape changing?

According to statistics cited in this video from artificial analysis.ai, the shelf life of an LLM at the top of benchmarks has compressed to roughly two months, and model capabilities are doubling approximately every seven months. Open-weight models now trail leading proprietary models by only about 3.5 months, making them a viable lower-cost alternative for many enterprise use cases.

Full transcript

[00:09] Before we begin, just a quick disclaimer that any forward-looking statements we they're not guaranteed of product maps or features. This is based on market insights trends. So,
[00:21] uh please don't take it as a guarantee. Uh and if you could at the end of the talk complete a quick survey, we would really appreciate this because this will be considered whether we will come back
[00:33] next year on the stage. Uh so, with this, let's uh start. So, we have really interesting topic today and we'll talk about the right model for the right job. My name is Victoria. I'm a
[00:46] principal technical evangelist at Databricks and today I'm joined by Lona, who is a director of forward deployed engineering. So, we will talk about a shift that we
[00:58] are seeing now in generative AI in particular in architecture. So, over the past 2 years, the default for building AI applications has been
[01:11] uh utilizing API to frontier models and build your your stack. But, as we start uh having more production volume, now we
[01:23] have more concerns regarding cost optimization. So, as uh business grows with the usage, now uh we we talk more about optimization per task, uh token
[01:36] optimization, token maxing. So, today we're going to address how you can build robust evaluation and LM diversification. So, in this session, we will deliver a framework that will help
[01:49] you to systematically evaluate optim and optimize your models for your specific task. So, if we think about the pace of innovation now,
[02:01] it's truly mind-blowing. And this chart, it's basically represent how long each particular LLM stays at the top before it gets replaced. And
[02:13] as you can see, a couple years ago, the span was about 12 to 18 months, and it just start compressing and compressing. So, now the shelf life of LLM to stay on
[02:25] top of benchmarks is around 2 months. And on top of this, the capabilities of those LLMs, they are doubling every 7 months. And what's also interesting
[02:38] happening that open source ecosystem is growing very quick. So, those open weight models, they now behind proprietary models by about 3 and 1/2 months. And they're
[02:51] achieving similar intelligence. So, those statistics are from artificial analysis.ai, really great website where you can find more. I think one of the interesting
[03:03] case was recently with cursor. So, I'm sure many developers tried cursor idea for coding tasks. And they originally built their coding assistant by
[03:16] utilizing proprietary models, OpenAI and Anthropic. And the business model is based on subscription, right? So, the more developers started adopting those AI
[03:28] tools, the cost per task became unsustainable. Because now so many tasks are being done, right? But you're still charging the same subscription fee. So, they realized, you know, they need to change
[03:41] something, otherwise they're going to be continuing to be underwater. So, what they've done, they actually then fine-tuned an open weight model, Kimia, on the specific tasks. So, they are for,
[03:56] of course, coding. So, they looked at their prompt response loops and optimized their tasks specifically on coding. And this helped them to maintain benchmark while at the same time
[04:09] achieving 86% drop on token per task. So, what we want today is want to talk today is about how your business can also build a sustainable so that as
[04:22] usage grows, um it will still deliver uh net profit and result in return on investment. Um so, today during our session, we will talk we built a few demos to help you to
[04:36] understand how to apply this framework. So, we will be utilizing uh example of Brickster Sport company. So, imagine now there is a football season or soccer season,
[04:48] uh and you have an application where different fans from different countries are sending messages and interacting. So, our job is to moderate this chat. But, moderation actually means many
[05:02] different things. Let's say, a lot of these images we will just let them go through. Some of them might be a spam. Some of them might be abuse. And they will be really easy to
[05:14] differentiate by class utilizing the classification models and right away ban. If there are repeated offenders, maybe we can also ban those users. Maybe we can issue warnings. But,
[05:26] uh some messages sometimes might be really hard to differentiate because is this a joke or this is a threat? Is it a sarcasm? And that's where we might need to utilize a different model or even
[05:39] escalate to human reviewer. But, escalating to human reviewer, we need to provide context. And context is not just a message itself. It's also what happened during that game. What was said in previous conversation.
[05:53] So we need to provide all of this context whether it's to a advisory LLM or a human reviewer. So in this session we will talk about this two specific tasks and we will try to
[06:05] identify for each of those tasks which model we need to use. So now you can try to guess which model is going to win and we will see if you guess will be right at the end.
[06:21] So what we will walk through is a three-step framework. So first we'll talk about different models and their trade-offs, how you can utilize evaluation to really understand where you need to make those trade-offs based
[06:34] on your own use case. Then we'll talk about how to govern and also switch models utilizing AI gateway in production. And finally we'll talk about when it's time to really think about
[06:46] fine-tuning and build your own intellectual property. And then we'll have some resources to help you to get started. So first let's just audit the kind of
[06:58] like ecosystem to understand what we have in the market. So currently actually over 37% of enterprises already utilizing over
[07:11] five models in production and 37% of all token volumes already coming from open weight. So the this ecosystem open ecosystem is really
[07:23] being utilized now and it's growing. So you can also see from this chart the blue line is how quickly open weight models are catching up with
[07:35] intelligence of proprietary. This was also recent chart that I took just before our talk to look at what are those different models and you can see
[07:48] how many model providers are there. So, there are many model providers on proprietary, there are many model providers not only on the US but also in Europe, in China.
[08:00] So, the Kimi model for example was developed in China. So, the ecosystem is growing and of course those choices are great but then they also introduce okay, so then how do I determine which model I
[08:14] need to go to? So, the question really shifted if we 2 years ago if you were here, everybody were asking okay, how do I get started? But now it's more about which model is right for me and as model
[08:27] continue to change so quickly, how do you actually keep your application layer decoupled from this model serving layer so that every time the model changes,
[08:40] you don't have to rewrite your application. So, if we look at the overall spectrum I mentioned couple times, we can look at let's say three types of
[08:52] archetypes is proprietary, open weight and fine-tuned. You can utilize all those three archetypes on Databricks. If you go with proprietary models, actually Databricks
[09:05] is the only platform that allows you to access any model, right? There is no vendor lock-in. You can go to Anthropic, you can utilize Gemini, you can utilize OpenAI. You pay per token. So, you
[09:19] basically yes, per pay per API. And then with open weight models, you now have a choice. You can pay per token but you can also host this model and then pay
[09:31] for compute. So, depending on your use case, depending on your usage, you might select which way it will be better for you, whether it's a paper token or self-host.
[09:43] And then, the third is fine-tuning. So, that's where you can fine-tune model. You basically download the open weight models. You utilize your data on Databricks. You fine-tune. And now you
[09:55] have this intellectual property that you register with Unity Catalog. And now it's not your You're creating not only that traces, right? Basically, all this
[10:07] data that's being generated, it's your own. You are creating really your business differentiator. A couple examples, one of our customer,
[10:19] uh Grammarly, I think they now called Superhuman. They fine-tuned the small open weight llama model and on Databricks. And of course, it resulted in quite large
[10:32] savings. Uh so, besides of course, the three archetypes, we can differentiate models by size. So, if we look at Claude, right, they have Opus, Sonnet, and Haiku. Uh
[10:47] Those three different sizes, they also come with their own trade-offs. If we are getting more intelligent model as Opus, yes, it will
[10:59] excel in terms of intelligence and in in reasoning, but it also comes at a higher cost and higher latency. And um This is a great framework to get
[11:12] started, but of course, all of this is not enough to actually find out which model will be right for your use case. So, now I'm going to invite Nuno, who will talk about how to build evaluation
[11:26] suite so that you create your data sets, and you can systematically select and test different models and identify what are those different trade-offs based on your use case.
[11:38] >> Thanks, Victoria. I'll take step one, building the evaluation. Before you can choose between the models, you need to define
[11:50] what good looks like. You need to set your quality bar or your yardstick. And this is something you do once and you can use every time any model is released.
[12:02] And the way you define the quality bar is with MLflow. The platform of choice for building high-quality AI systems and agents and ML models. It's open source, 25 million downloads.
[12:16] It started just as an experiment tracking and grew into the end-to-end loop that you see on the right. Evaluation, model registry, serving, visualization, all in one.
[12:28] And that's what we're running today. The pipeline is four steps. Step one, you build the evaluation set and store it in Unity Catalog.
[12:40] Two, you define your scores. It could be a simple Python function if you're looking to evaluate an exact match, a plain language guideline, a heuristic, or an LLM judge. It depends on the use
[12:54] case you have. Three, you actually run it. You just execute one call to MLflow and you get one trace per row. And finally, you can analyze the results. One row per model where you can
[13:08] sort by accuracy, speed, and cost and look at the trade-offs. The winner might surprise you. In our example, we actually run 16 models on the task that Victoria showed,
[13:21] changing one simple line of code every time. We'll now go through these four steps in our Brickster Cop moderation demo.
[13:37] In step one, you create a high-quality evaluation set. As we see, there's eight labels such as clean, off-topic, harassment, and hate speech. And we also have hard examples and easy examples.
[13:49] Step two, you define your scorers. In this case, it's a simple Python function. Step three, you actually execute the MLflow run. And finally, you can analyze
[14:01] the results, which we'll look into a bit more detail later. Jumping into the human escalation reasoning, we do the same four steps with some small changes.
[14:14] First, you create the evaluation set, and you can see here it's a bit more elaborate. We need to give the user history so that the human reviewer can look at it. Step two, you define the scorers.
[14:27] In this case, you're actually trying to compare an escalation semantic similarity instead of an exact match. So, you're not going to use an one or zero, you're going to use a one to five. And we'll use Opus 5.8 as our judge for
[14:41] the best quality possible. In step three, you actually execute the flow with your scorer on your data set. And finally, you can look at the results. Again, I won't spoil them for
[14:53] now, but I will show them a bit later. The evaluation you just run is now available in MLflow experiments tab almost immediately, where you can see all your runs. Here are the 16 models we
[15:07] run against each other, increasing the metrics from accuracy, latency, cost per speed to to cost per million messages. And finally, you can also look at the traces individually. So, trying to
[15:21] understand for each individual message or model, understanding more detail what happened. What was the latency? What was the moderation decision? And whether it was correct or incorrect.
[15:43] MLflow runs evaluations in two modes, and both are super important. Mode one is automatic. Every time a model changes or a new model is released, or you change the prompt to improve your system, you rerun the evaluations
[15:55] programmatically, which means they can be automated. And mode two, not less important, is the human evaluation, because the code and the programmatic cannot catch every
[16:07] single nuance. In some case, you actually need a human in the loop to look at the runs and review them. Databricks provides this review app where human experts can review the
[16:19] traces and log their corrections. And those corrections sync straight back into your data set. So, every correction is continuously upgrading your overall evaluation.
[16:32] Automatic gives you speed and scale, and human gives you the ground truth you would apply to the critical tasks, for example, or where the model is less confident. Together, the evaluation
[16:44] keeps getting better and better. Back to the first principles, evaluation is a continuous loop. So, it shouldn't be something you do just once.
[16:56] You should do it every time you change any part of the system. Could be the models, could be the prompt. You test it in development, you deploy it, and you look at the results. The edge cases are flagged for review,
[17:09] and they're joined back into your test set. So, when a new model drops next month, and it will, you don't start You just rerun the evaluations. Victoria, over to you. Now that we can
[17:22] measure models, let's access them all. >> Yeah.
[17:35] actually run this evaluation and you identified maybe a few candidates. You don't need to make a decision, okay, this is the only model that I'm going to use. You can split your traffic. You can put three models, right? And then maybe
[17:49] also configure some fallback models. And then as you deploy them with Unity AI Gateway, you can also see how these models perform in life. And therefore,
[18:01] because maybe your training data set and your live data set might you might find some differences, right? So, it's not that you you did this once and then you forgot about this. You can continuously getting
[18:13] those live traces and analyze and then shift traffic accordingly. So, this is actually what we are seeing that right now most enterprises they don't default just to one model. They are
[18:26] utilizing several ones and that's why creating attaching yourself just to one provider is actually becoming a risk because let's say
[18:38] there's a new provider that comes back comes with this amazing intelligence, but now you cannot use it because you you stuck with one ecosystem. So, that's why many customers go to Databricks that because we try to
[18:51] eliminate this vendor lock and really try to help you to utilize the best model for for yourself. Uh what also Unity AI Gateway provides, in addition to
[19:04] accessing different models, it actually provides you this one place where you can observe all of those traces. You can configure rate limits. You can configure
[19:17] permission to different models. Maybe let's say for marketing you'll say, "You know, I think marketing will be fine with a high cool, but my engineering let give them access to Opus, right?" So, as
[19:29] an example. So, you can make a lot of those configurations inside Unity AI Gateway. Also, Unity AI Gateway is not just for models. It's really access to
[19:41] your entire AI estate. And this can be also coding agents. This can be MCP servers. And those MCP servers are not just Databricks MCP servers. Those are also partner MCP
[19:54] servers. So, think about this as your one place where you can open and see how your entire organization is utilizing AI and how you can make those cost
[20:06] adjustments. So, according is actually a very good example. So, as developers, we don't use just one coding tool anymore, right? So,
[20:18] we might start with IDE like Cursor, and we might utilize in Cursor some some agents. Then we might switch to terminal, and then code code, do some work, and then we can go to Genie, and
[20:31] then troubleshoot and fix things in Genie. So, we we actually jumping quite a lot. And basically what it creates then there is no a single place to understand how developers are utilizing
[20:44] the models, right? Where where the cost usage is growing. So, you can power your AI coding tools, your AI agents also also through Unity AI Gateway. So, you basically
[20:56] point code code to the endpoint in your Unity AI Gateway, and now you can utilize code code or whatever tool you want. And now you will have this one dashboard where you can go and see how
[21:09] different teams are utilizing them. Uh so, how do we actually start? So, we run, let's say, this evaluation and we see uh metrics. Um what we see a lot from customers,
[21:22] there's a lot of excitement when the new models get dropped on the market and like, "Oh my god, there is this formal. I need to start utilizing this uh model." But, what the really the thing
[21:34] right thing to do is that maybe you don't need a PhD degree to solve a very simple problem, right? So, you can solve it is the intern. So, this the same thing with um
[21:48] those models. You can start with something simple, then run it against your evaluation. If you clear your quality benchmark, then why to pay for more? If you If it doesn't
[22:00] clear, then you can climb up, right? And utilize a larger um more intelligent models. And as new models drop, just run them in parallel. Put Put the
[22:12] traffic split in always a new model release, maybe put the 10% of your traffic and see uh what what will be a difference. Another thing to consider that maybe for some of you the 5% difference in
[22:25] accuracy Some will say, "Oh, this is actually not going to make an impact on my net income." But, for some that because this 5% can actually generate maybe millions
[22:37] of dollars, you'll say, "No, I actually want to increase uh a spend by 20 times because I will get this marginal 5% improvement." Uh so, now let's uh review actually
[22:49] dashboard. So, Nuno explained about how he created this notebook and run MLflow evaluation. So, after MLflow evaluation run, we can review dashboard. So, here's
[23:01] an example of dashboard. Basically, we have the uh F1 because this is for classification model, F1 metrics, and accuracy metrics. So, as you can see in the red, the score metrics, they are
[23:14] quite high, but also there is some models like Gemma 3 and GPT-5 54, also quite scoring quite high. And look at the difference in price, right? Like Opus compared to Gemma, we actually
[23:28] don't even see where a Gemma is with the with the cost because the difference is so high. And the same with latency. So, now we can look at all the metrics together. And as yes, you know, I think
[23:41] it's not surprised Opus 4.8 bit in terms of accuracy, but also look at the cost. So, 0.94 accuracy with $3,600 approximately per 1 million tokens. And
[23:54] then with, let's say, GPT-54 Nano, we got 0.085, but it's $9. So, it's basically difference of 40 times, right? In terms of cost. So,
[24:08] that's where you need to decide, well, does this justify for me to spend 40 times more to gain this accuracy? And depending, as I said, on your business metrics and
[24:22] where you implement. For some, it will be the answer yes. For some, the answer will be no. So, now let me just do a quick walk through to show how you can configure this in Unity AI Gateway. So, here I'm
[24:35] in Unity AI Gateway. So, the first thing that you do, you just click create an endpoint. Then you search for you put the name, and then you search for different models. So, you can see here
[24:47] the list of different models a very high-level speed and cost. You can add as many different models as you want. You can then configure the traffic split and
[25:00] also configure fallback models. Another thing that you can do, you can set up rate limits. So, you can set rate limits of tokens per minute and query
[25:13] per minute and you can get a very granular. You can get it on the group level, on the user level. So, this provides you this governance. Once you set this up,
[25:26] you will be able then to monitor usage of this one endpoint. So, at the end you're going to have one endpoint and then the the traffic will be split. So, here let's do you can test it very quickly.
[25:39] So, I'm just putting a spam message and you can see right away I got the response. Yes, it was spam. And then I can go to Unity AI Gateway dashboard, which is a pre-built dashboard for you. And here you can analyze performance,
[25:53] usage, uh coding agents. And let's say if you look at this dashboard and look I I have some other questions, you can just simply ask
[26:05] Genie. So, this was actually a dashboard of my team and I said, you know, how what kind of model Victoria is using the most? Um and it came back that I'm
[26:17] utilizing in over 69% Opus. So, maybe I'm not the most >>
[26:30] visibility, including my manager. Um all of this metrics are stored in system tables. So, you can review the system table AI Gateway, so you can build your
[26:42] own dashboards, your own Genie spaces and get this very granular information. Well, now that we understand how to evaluate models, how to put them in
[26:55] production, I think the last question that we need to answer is okay, but when do I actually need to fine-tune model and create this intellectual property myself? So, that's when I'm going to call Nuno.
[27:08] >> Thanks, Victoria. Let's hope your manager doesn't watch this presentation about all those tokens. So, we've got a cheap model that's fast and often incorrect, and a frontier model that's right, very expensive.
[27:22] Step three is about the next option, the third option, fine-tuning a small open model. First, let me be clear. Fine-tuning a model should never be the
[27:35] first thing you do. Most of the time, AI gateway and a good prompt are more than enough. You reach for fine-tuning when these three signals align. One, mistakes are
[27:49] repetitive. So, the model keeps missing in the same way, the same wrong category, the same broken format, the same tone. Two, you have labeled examples to fine-tune your model. A few thousand
[28:03] good input-output pairs are often more than enough. And third, you have maxed out on prompting or rag. Fine-tuning will eventually help you reduce the context size and cost cost
[28:16] per call, but it's a lot more complex to maintain over time because you need to keep retraining the model. But as Victoria said, it's all about the trade-offs between accuracy, speed, and
[28:28] cost. When a small model cannot meet your quality bar and a frontier model is too expensive or cost-prohibitive, fine-tuning becomes an option.
[28:41] One of our customers, Canter, was trying to match receipt strings to actual product names. And they started with a frontier model. The model was working great, but because
[28:54] of the volume, it was becoming too expensive. So, they actually decided to fine-tune a model and obtained a 94% accuracy for a fraction of the cost.
[29:11] Historically, setting up a fine-tuning infrastructure was a huge problem on its own. You needed to find the GPUs, stand up the whole infrastructure, and watch and pay for it while it was sitting idle.
[29:25] Once all that was in place, actually training the model was the most straightforward step. Databricks has solved this problem by introducing AI runtime. It's basically serverless GPU compute.
[29:38] So, you basically open a notebook, attach a GPU, could be A10 or H100 or multiple H100 as a cluster, and because it's serverless,
[29:51] you pay only what you use for the seconds while it's operating. And that includes scaling to zero. This is how it works. You basically pick a small open-source model, could be a
[30:05] Quan or a Lama or a Kimi, and you put a straight your data straight from Unity Catalog. You then run your parameter efficient fine-tuning, could be a Laura or QLoRA,
[30:17] which basically freezes the base model and optimizes and trains a new set of weights. So, it's very fast and cheap. The run is, of course, also tracked in MLflow.
[30:29] Registering the model in Unity Catalog and deploying it as easily to a serving endpoint. I'm now going to show a fine-tuning a Quan model for the Brick-a-Cup moderation task on a single 810
[30:44] serverless GPU. With AI runtime, you basically select the GPU in a few clicks.
[30:57] After a few minutes, you can see it's up and running. You then install all the necessary Python libraries to train your model.
[31:11] And also configure where to get the training data set. In this case, we're fine-tuning a Qwen 1.7B model.
[31:23] You then download the data set from Hugging Face, and you only need to do this step once. And load the training set from disk. You then load your model.
[31:39] And this is a step where you actually set up the lower parameters like the alpha and the dropout. And you can see it loading the weights in real time.
[31:56] And finally, you can run your training script, your fine-tuning script. As you can see, MLflow is embedded in every part of the way.
[32:09] While it fine-tunes, you can see the training loss going down as the model, the fine-tuned model, is converging. After a few minutes, the model is trained.
[32:21] And you can see it actually uploads the artifacts. As I said, then if you can go to MLflow, immediately you can see all your training metrics. So, you can see things like whether it
[32:34] finished correctly, how much memory or GPU it used in the training process. And if you're a model geek, you can actually go into the model metrics and understand if it's converged, if there
[32:46] was any problem with the training of the model. So that you can actually deploy it. So the next step is actually deploying that model to a serving endpoint.
[32:59] With just a few lines of code, you actually train a whole model, fine-tune a whole model, and deploy it to a serving endpoint. After a few minutes, your serving endpoint is up and running.
[33:17] And you can immediately start throwing requests at your API. You can also look at all the metrics such as memory usage response rates
[33:29] and also what kind of compute you have. Now, the part that matters. Since we have a new model
[33:41] we need to eat our own dog food and follow our framework. Looking at the the dashboard that Victoria showed we now have a new data point in our
[33:54] dashboard, which is our fine-tuning dashboard. You can see how it does against all the other models. You can look at the cost, the F1 score and all the other metrics that matter.
[34:15] For example, if you look at the accuracy we achieved an accuracy of 91%. If you remember Opus 4.8 was at 94%. So it's actually extremely competitive. It's one of the best models.
[34:27] But if you look at the cost, it's only $8 versus $3,600 for the open model. Latency at 255 seconds is also much faster than any other
[34:40] Frontier models. For example, Opus is about 1 second and while the other models are at least half a second.
[34:55] But here's the surprise. We only solved one job. If you remember, there were two. The second one was reasoning, the escalations. Only about 1% of the volume, the messages where there's some ambiguity
[35:07] where we escalated for human review. The system has to do some reasoning, so it's not just a classification. And because it worked the first time, we also tried fine-tuning a model for this use case. It worked for the first time.
[35:22] So, watch what happens. The fine-tuning specialist showed 1.3 out of five. If you remember, we had set up an LLM judge for this step.
[35:37] The Frontier model goes the other way. So, Sonnet shows a 4.7 rating and Opus a 4.5 rating. So, this is so very interesting. How can a Opus 4.5 show worse accuracy
[35:50] even when Opus is actually the judge, if you remember. So, this is a good reminder to not follow any assumptions and actually do a professor professional evaluation every time.
[36:03] Looking at the cost is clearly that you need to pay for this accuracy, right? At $68 per million messages versus uh 50 cents. But in this case, it's worth it because
[36:16] you only call it 1% of the time. So, let me bring it home. Let's summarize the framework we discussed today. First, build the
[36:28] before you pick the model. With MLflow, it's actually the cheapest thing you'll do and the most important. Inputs, expected outputs, hard cases, traces, scores, and
[36:40] human feedback. Models will keep changing, but every new model should be measured against the same quality bar. Step two, you can access and govern your models through Unity AI Gateway.
[36:53] As Victoria said, start with the smallest, cheapest model and climb only when the evaluation requires it. You can easily add new models to AI Gateway, split traffic, watch the
[37:05] traces, and feed your human feedback into your evaluation to improve their assets over time. And finally, only when the task earns it, you do the fine-tuning.
[37:18] It's a high-volume task, you got labeled data, and you have maxed out every other option. That's when Databricks AI Runtime open model turns turns a open model into
[37:30] a specialist that you can control, and that's yours. To support your implementation of this framework, we provided the following links. The first one
[37:42] is AI Dev Kit. AI Dev Kit helps you build all of these that we showed with live coding. It really understands the Databricks documentation and products, so it's better than a generic tool.
[37:56] It's going to accelerate the process of moving this into production, and you don't have to start from scratch. For MLflow, we have a tutorial on how to get started for GenAI.
[38:08] And finally, some example notebooks on how to fine-tune models as well. So, decompose your workload into jobs and evaluate the best model for the job.
[38:23] No model will win every time. No model will win every axis. So, you need to choose the axis that matter, not the brand. Right model, right job, every time. Thank you, and enjoy the remainder of
[38:36] Data and AI Summit 2026. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.