Optimize Cost and User Value Through Model Routing AI Agent

Summary

• Dynamic LLM selection: Model routing optimizes cost and performance by intelligently selecting different LLMs based on latency, modality, API costs, and complexity.
• Enterprise optimization: Organizations balance cost reduction with maximizing use-case value by considering unique strengths of each foundation model.
• Scalable orchestration: Databricks enables enhanced scalability, cost efficiency, and user satisfaction through intelligent LLM routing strategies.

Optimize Cost and User Value Through Model Routing AI Agent

Talk by Aditya Gautam, Machine Learning Lead, Meta
Each LLM has unique strengths and weaknesses, and there is no one-size-fits-all solution. Companies strive to balance cost reduction with maximizing the value of their use cases by considering various factors such as latency, multi-modality, API costs, user need, and prompt complexity. Model routing helps in optimizing performance and cost along with enhanced scalability and user satisfaction. Overview of cost-effective models training using AI gateway logs, user feedback, prompt, and model features to design an intelligent model-routing AI agent. Covers different strategies for model routing, deployment in Mosaic AI, re-training, and evaluation through A/B testing and end-to-end Databricks workflows. Additionally, it will delve into the details of training data collection, feature engineering, prompt formatting, custom loss functions, architectural modifications, addressing cold-start problems, query embedding generation and clustering through VectorDB, and RL policy-based exploration.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and agenda
• 2:36 LLM components and Llama
• 6:43 LLM routing strategies
• 15:06 Technical details of routing
• 28:05 Implementation with Databricks

FAQs

What is model routing for LLMs?

Model routing is a technique for intelligently selecting different LLMs based on factors like latency, multi-modality support, API costs, user needs, and prompt complexity.

Why does each LLM have unique strengths?

Different foundation models excel at different tasks and have varying performance, cost, and capability profiles. No single LLM is optimal for all use cases.

AI gateway with intelligent model routing?

This video demonstrates intelligent model routing in the Databricks Data and AI platform that automatically selects between different LLMs based on cost, latency, and quality requirements. The routing agent evaluates requests and directs them to the most efficient model for that workload. This optimization reduces costs while maintaining performance.

Which AI gateways offer intelligent model routing for cost optimization?

This video showcases the Databricks Data and AI platform AI gateway with intelligent routing that optimizes LLM costs by directing requests to appropriate models. The platform tracks cost, latency, and quality metrics to continuously improve routing decisions. Organizations achieve significant cost reductions compared to running full-scale models.

What is a cold start?

This video explains how intelligent routing reduces cold start latency by directing requests to warm model instances. The platform maintains multiple model replicas and routes intelligently based on current capacity and performance characteristics. This eliminates the latency spike when models scale up from idle state.

Full transcript

0:00 Introduction and agenda

Hello everyone. Uh welcome to this session of optimizing cost and user value through routing agent. I'm from Meta. I'm a machine learning engineer working on uh different uh ranking recommendations and uh uh different problem spaces within meta. uh before I get started I want to get a sense of how many of the people here in their organization or product are using only single LLM API provider just one one not a lot people okay couple of them there uh okay and how many of you think would it would be great if you can have a blackbox which can figure out I don't care about which API I'm using which alm I'm using I just care about the user value the what is the performance coming from that LLM and what is the cost or the billing that is happening to me how many you think that would be great to have in your organization or product okay great awesome uh okay so you are in the right room uh so I hope by the end of this talk we are going to uh talk a lot about technical details but you will have a lot of takeaway with respect to why you would want to diversify your LLM APIs and how you can do that technically and like through data bricks platform.
So uh the high level talk would be on the four sections. First we will learn about what are the different LLM components which is uh related to the cost and how does different LLM providers has a cost variability. We will dive little bit little bit into the llama and the open source system. Uh the second part of the talk would be understanding what are the different goals and user value and categories and technical uh uh high level details of LRM routing.
And in the third part we will take a technical deep dive. We will understand the core components. What are the feature data training loss uh and how you can do online and offline evaluation? How can you measure the cost and savings and different aspect related to the cold start? For example, if there is a new model come in or if there is a new prompt or a query time come in, how do you kind of like deal with those cases?
And in the last section, we will be talking about uh how do we implement these part three about technical details using data brick component. I believe most of the people here are very much familiar with that. So we'll go through how the routing and datab bricks comes together. So the first section is LLM component. When we look at the LLMs, there are different component of the cost associated with it. The providers are charging you a particular dollar in and out for input and output token.

2:36 LLM components and Llama

That will be the premium and the most important cost. The bigger your input length is, the bigger output length is, the more cost it is to provide and the most cost it is to you. Embedding layer is another one where the embedding size and dimension is can change the cost from the provider's perspective. Uh third would be a parame parameter count. The bigger and the more complex and the larger number of transform layers on the model it is the more flops it is needed.
The more flops it is needed the higher the cost would be. That's why we are seeing a huge variability that okay we have a 8B models and 500B models having a huge difference in the cost. Then there are other components like retrieval, augmented generation and other minor cost related to the uh embeddings, loggings and all which is pretty minor. So uh this is a chart which is showing that how drastically the variation happen in the cost.
If on the one side we have some APIs which are take charging all the way to $75 and the other side we have 0. 15. through the open source model and these are the models which are um very cost effective cost effective and uh a little bit of comparison of like the left left most and the rightmost is like 500 times that's huge if even if you're able to divert like 10% of your query to this like the old model like or the smaller model or some x percentage your cost will be multiplied by a fraction of what you are actually being currently paying So we will be talking here the llama ecosystem.
This is an open source from meta. I think we all are aware of this thing and they it has series like three uh and four series. It is a collection of open-source model to provide stability to tailor to a particular AI needs. Uh since it is open source provide higher transparency security and also the greater control to individual organization as compared to a closed source model. And um just because you are using an open-source and a lowc cost model doesn't mean that you do not have a competitive edge.
Uh these are the models here that you can deploy anywhere like on cloud on premise u in your like box device GPUs anywhere you want. It provides a higher data security and privacy control. It creates an ecosystem within your organization because some for example for some use cases you might need just 8B model for other one you need like a high-end reasoning models because it is open source the weights are available out that means you can do a lot of customization over to the model and um change the architectures and like do different kind of experimentation with it and we believe the future of the LLMs are open source and we are heavily investing on this side.
Um this slide shows a like what are the different features and performance of different llama models. We have uh we have a lightweight ultraast model. Then uh the reasoning models the model which can understand uh like text and multimod multimedia in a better way and uh there are also model which are 1b 3b which are ondevice model if you want to use like some ondevice and have the privacy and other aspect uh preserved.
Uh on the rightmost side we have the latest models of Llama 4. 0 which is multimodel. It has a superior text and visual intelligence. It is based on MOE architecture and it supports like 12 uh languages and it has a context window of 10 million which is huge. You can put a book to LLM and ask any questions out of it. So based on like different characteristics you can figure out what is the right accuracy you need what is the latency of the model that you need and what is the cost associated with model and you can make a smart decision.
So this is we are talking only within llama ecosystem like this is the variability you have now let's come to the second section where we are talking about the goal of the routing what are advantages and categories. So the primary goal for routing of your LLM router is that you want to optimize cost but you do not want to compromise on the user quality. You still want to maintain it. You want to make sure that the users are still like being good but the cost is lower.

6:43 LLM routing strategies

Why do we need routing in the first place? First and the most important is it is cost effective. Like we have seen that in the previous slide. It's a fraction of a cost lama 3 as compared to the other closed source model. it can lead to a performance op optimization because there is no one model fits all the cases. There are models which are good for uh coding, reasonings, mathematical operation and those kind of things.
So by intelligently rerouting you are actually finding the best model for the given prompt and a query. There is no vendor lock in because router will take care if the new model come in there is a new company which is open sourcing or like putting out the model. It will automatically kind of like take that into consideration and make sure that anything which is there out there is available there and there is no vendor uh lock in happening in this case. uh enhanced scalability, higher user satisfaction because now you have a diversity on a pool of model to answer from reliability and failover.
If one model fails, there is a policy issue or any kind of things, you are still like good with your u infrastructure stack and you are still rerouting in the right way and then a quick roll out of the models. Coming to the user satisfaction, what do we care about like when when when you are interacting with uh LLMs, when your customers are interacting with LM, LLMs, like what are the factors which consider in the user satisfaction?
One is like the complete and a concise answer. What is the cost associated with it? If you have two kind of uh for example in a B2C, if one is charging $100, other one is charging $20 a month and they're providing the same quality, it's a no-brainer to go with a lowc cost one. accuracy and factually correct, how relevant your answer is to the question asked, how clear it is and whether it is able to safeguard and have guardrails related to toxic and other kind of like issues that happens and u in in wider ecosystem and of course the latency where you want to make sure like the answer is coming quickly user user is not waiting for seconds to get the answer.
Now coming to the categories of LLM routing. We have first one which is the static routing which is the like the low complexity where you are actually manually putting if else and different kind of condition to say that okay it belongs to coding and I know that there is a particular u coding model which is good like let me rerout. So these are the predefined rules. The second is more sophisticated which is a classifier based routing where you are understanding the prompt pretty well and you are classifying that prompt based on some model classification and then there are there's a semantic routing within it which we will talk where we understand the similarity of the prompt and we see that what are the prompts what are the prompts which are similar and what are the responses in the past which has been given uh by this LLM router and we directly take the top voted one and there are LLM based methods like reward model like putting the prompt directly within LLM to find what is the best uh routing model to go with.
So okay static routing. So for example if the question is like what is a weather in C um SF you really don't need to call LLM and do a lot of context processing reasoning and all those things we know that this is a direct function call. So you don't even need to pay for the input output token. You just need to make an API call and you will get the weather and just put it in the format. Similar to that if you have in your organizations a fine-tuning model basically you are understanding uh okay this query belongs to a particular aspect and there are already model that I've tuned within my organization and it's a no-brainer to reroute the query to that's where the fine-tuned model would give the highest performance and it's a very static and a simple routing uh similar to that we can have a keyword based extraction that okay if it is a coding related thing you understand there are certain kind of like code aspect which are there in the query.
That means you know which model to read out it to for the highest and the optimal performance and then you can have different custom based rules like length and finding different attributes within your prompt and say that okay this belongs to reasoning let me find the reasoning model rather than just calling the default model. So in the MLbased classifier routing uh how do we train it? uh we will uh we can take a existing the board based model for classification where the input is u the prompt embedding then output is the model classification.
So if you have five models of pool that you want to like have the rerouter to have intelligence on that would be your output node and input would be prompt embedding and different features. Uh this could be a binary that means a single uh a single class or a multi-class classification that means uh your model can be trained in a way in which your inference uh your uh in-house preference data set can have multiple output which are all good and then you pick up the lowest cost based on the threshold that you have picked in.
For example, if the threshold is point8 and there are three models coming in which are passing that for a input prompt you pick the lowest cost model because everyone everything is above your threshold. So you are not compromising on the user quality. Um on the cementing routing this is where you are actually getting into the embedding space understanding uh like how do we reroute the query based on the embedding uh of the input query and what is already out there.
So in the offline processing you would out have a predefined data set with a human labeling and then what is a corresponding best LLM uh for those queries already out there and then we will generate the embedding of this put it into a vector database uh using some um similarity library and then whenever the new prompt come in we will generate the embedding of this find the nearby prompt and then find the model which is selected for those similar prompt.
So this is more like you are understanding and from the past what is the semantic un routing similarity and based on that um you are updating the uh response. LLM as a router is another one where the LLM is trained to predict a particular scalar score let's say between zero and one which indicates the quality of a response and this is trained on a human preference data set or LLM as a judge based on certain rubrics like helpfulness clarity uh harmlessness and in this uh framework a cheap model can be first uh given to your input query lm reward will give a reward and if it is lower than threshold that means only then you need to give it to a costly model otherwise we you are good on this part and this could be deployed in a hybrid uh environment that means you can have a semantic or a rulebased query along with this to merge and make a custom output uh another way would be that like you can directly uh put a prompt to an LLM it may not be the optimal but still it's uh good to know that this is another way to do it uh where all the information related to LLMs for example what is the benchmark performance what is the cost input output API cost and other things uh out there what are the speciality of those LLM uh providers you put it in a database and then like given this standard prompt that okay this is the information extracted from the rag or your external databases and then appending it the user query to that prompt so that means you are directly asking LLM that okay please provide me what is the best LM to uh read out this too.
Uh and in this uh we can add like few short examples of something that has been done in the past or some of the example through which uh LLM can improve your performance by in context learning and uh of course like there is a a way to do the output formatting if you want to have a JSON object or some other way to see like whatever is the specific use case you have for this. Um going to the technical details the core component of the routers would be the features.
These are the features related to the prompt whether the prompt like what category of the prompt it is whether it's a complex prompt raising prompt coding prompt we don't know user what is the expertise of a user uh for example if the user there is a user who is a PhD in ML and there is a high school student who want to learn about machine learning same query would have a different response based on the user uh expertise right so you understanding the user what they have done in the past like using that history would be helpful model is like complexity of the model performance on the benchmark cost input output.

15:06 Technical details of routing

Yeah, on the model pool basically all the candidates of the model that you want to consider for routing and a router which is a intelligence blackbox system which can figure out this is the prompt and this is the best uh response configured this is the best LLM to pick from uh everything and then the training data which is like how do we train this model what we talked about with respect to the uh birdbased classifier or semantic routing or anything uh category which we talked earlier uh The training data would include the query, the response, the LLM cost quality and uh inhuman or some way to give a quality measure either through LLM as a charge and of course the once you deploy these kind of system you want to understand the explicit and implicit user feedback and make sure that you are not degrading in the quality just by root by putting a router in the middle as compared to using only one L&M API.
Uh so some of the features for routing from the prompt side we would have a raw text prompt length embeddings entit uh or different keyword that you can think of this is um this is something which is custom but like these are some of the basic features from the user. Um you would would want to know what exactly is the expertise of the users. What are the historical interaction that the user had uh what are the section details whatever you can take uh from the user as per your policies and other thing and of course like the explicit implicit feedback from the user on the model feature size.
We are talking about the input output tokus token cost which is the most important feature that we are optimizing for capability of the model whether it's a reasoning u rag or like chain of thoughts or a simple good for coding or something performance on the standard benchmark to understand like which like if the query is similar to some benchmark then which is a lowc cost model you can go with latency of the model because this is also related to the user satisfaction and any prior knowledge.
Okay. So now to build your training or creating your custom router the first thing you would need to collect the data and the most uh best way to collect data for your specific use case would be to have in-house or take a third party data where you are given a prompt and for that prompt you are given the response from all the LLM pools and there is a human which will say that okay from 0 to one on different rubrics like what is the score for all of those prompts and that's the training data that you would want to have for your router and then there's the synthetic data uh because it's not possible to humanly create a lot of data set and label all of them.
Um so there is this concept of synthetic data generation which is happening a lot in uh different companies and different organizations where you generate through LLM given the seed data please generate similar prompts and outputs and understanding the uh historical performance and getting the users uh implicit and explicit signals from the user to understand like how well it is performing is an important aspect of this.
Okay. So we talked about like the training data. This is more deep dive into a loss function because you are create creating your own classifier. Uh the loss function would be a cross entropy for a single or a multiclass thing. For example, if we are talking about the board based classifier to give a score to each model given a prom, then we would be basically picking up a dynamic threshold or whatever we think is good for our cost and during the inference time we will pick the lowest card model which are above threshold uh of that and okay so here we talked only about the quality that's how we are training this model but if you want to make the model output not just the quality but as some function of the cost then you would be changing your loss function and putting a cost variable in there and this by this lambda you can you can just like change the dimension from the very high preference to the cost and like little to quality to high quality and low cost efficiency kind of thing.
So this is this will give a little bit of like flexibility for user to uh when they're training their own router to give more. Yeah. So how do we train the model? We have our data collections. We have all the features what we talked about. We have a pool of LLM. We have a responses from these LLM from in-house uh uh trainers. We know the co uh score and we already have the cost which is available statically uh from uh different providers and the quality score could be human and editor inhouse.
It could be LLM as a judge or uh whatever way you can have confidence on what you mean by quality on that side. Then based on this data set, this is all you need to train a model. You can do a supervised learning and training um with loss function that you pick for. Um yeah and then uh also there is a way to fine-tune a smaller model on this data set which can be used as a router. uh going into a little bit more advanced thing uh which we would not go um too much into detail but like uh there could be a RL based agent which can learn about these interactions with the environment and it would have three all the states that we need for RL that is this uh state which is a prompt feature what is the action action is like choosing which LLM and the reward is a function of cost and quality that you can uh dynamically define in this one okay so once we have trained did we have our data we have loss function figured out and we know how to do everything on the training side how do we do the offline evaluation so on the data set that you have already created you will have a hold out which are containing which contains a ground truth based on your quality assessment your inhuman uh uh or however you want to design that quality aspect that's uh very subjective and then u based on the router like there is a routing decision bas from a base model to like this pool of model depending on the prompt and other feature and um likewise in the offline evaluation uh we can understand what's the router accuracy what is the quality score because we have the ground truth and then once it is good enough then we will go to the second stage which is online AB testing so in the online AB testing we would have first set up two different buckets basically we will split our traffic into two bucket A and B uh un knowing And basically just doing our homework that uh there is no underlying distribution change or variation in that there are there is nothing statically significant.
For example, if you put everyone into expert all the expert people who are doing like very reasoning query into one bucket and all the naive people who are doing very basic mathematical or like the simple query into other bucket that would create a significance problem in our experimentation. So no sample ratio mismatch these and u yeah and then we can deploy different kind of routing strategies to different bucket to see which one is doing well.
So by well we mean we are going to collect different metrices for example API cost incurred in each of the buckets. What is the user satisfaction based on either implicit or explicit feedback. What is the rate of completion like how many of the queries and the prompt or response are good? Engagement metrics of the user which is uh how how often the users are coming to your platform. They really like this is a proxy matrix whether they really like your chatbot or however you have deployed your LLM application.
What is the observed latency and the error rates that you are getting and the qualitative feedback from the users either through survey or some other way. Once we have set up the matrices use standard statical methods to understand and analyze basically how we are doing on this aspect and uh iteratively improve on this using different routing logic adding more feature maybe having more uh models and deploy new experimentation and quantitatively understand like how we can improve once the router is already launched on that side.
How do we measure the cost saving? A very simple this is a very simple way of doing it. there are more complicated ways also. So in the first bucket uh you already have like a predefined uh API or the model that you're using where the cost for all the LLM calls is same. Now in this one you are actually doing a model rerouting based on the uh LLM router and this will diversify the uh API call on different uh LLMs. That means your cost is now diversified among like the prompt and other complexity and then you can just do a standard uh cost saving on this aspect uh to understand like what is the percentage uh we are saving.
So we talked about the costsaving but this is uh if if you just want to talk about costsaving you can just pick the lowest cost model and read out all the queries to there but that is not going to do justice because it would may or may not compromise with the user quality. So the other aspect is to make sure that user response performance is still like maintained and it's pretty good. So how do we measure that? One is to have like a quality metrics using a automated blue rogue or LLM as a judge uh which is used a lot right now because these are subjective evaluation as compared to like a discriminative evaluations earlier using prelimm error and then having human loop like uh understand like on different rubrics accuracy relevance coherence like how well we are actually doing whether you are still maintaining those uh rubrics in after deploying LLM router and earlier using only one LLM API uh latency end to end measurement of the latency understand like what is the average latency P99 P55 uh P95 and other things uh all the systematic aspect on that uh third one would be the factuality understanding how often the model is producing incorrect or fabricated information and having some way to do the histation index u this is again you would want to have like a human loop to make sure this is done correctly. uh toxic and I would say like uh all the policy wise whether the model is doing really well it is not producing any toxic output and it's like doing a very graceful handling of these edge cases and having um like where the very tricky and tricky questions are asked to ls how do we make sure that that aspect is still maintained so uh by measuring the response performance or the user quality like these are some dimensions uh we can look into and probably have our evaluation based on these.
Okay. So, uh when we are deploying it, there would be a possibility that for example uh in in in your system uh there is no coding query and suddenly there is a coding query comes in and or some other reasoning query comes in which has not been there in your data set before like on which you uh created the preference data set. So how do we make sure like to solve this? So the uh default way would be to put it to the router which is more robust and a general purpose because you do not want to compromise the quality of the user response.
Active learning is where you have a new query patterns. You generate the labels for them using agent evaluation or your inhuman in in in-house human labeling team and then create it into into data set understand and reroute uh retrain your model and put it back again. There could be a possibility of a new model. So new model comes into the market and you already have the router deployed with like let's say five models.
There is a sixth model come in which is pretty promising as per the reports and everything and you want to add it. So if you want to add a new model uh first thing would be to understand what is the outside benchmarking performance of those models I believe uh if one want to add into the L&M router that is already wetted. So um now the represented data set that you have in inside for the for your specific use case understand what is the performance coming out like on this on on that data set and add uh the inference or the labels on that data set using this new model.
Um there is another online way where you can uh have a small percentage of this query is read out to this new model and gather the performance data loop back in retrain your model and then deploy it again. Okay. Um, LLM routings through datab bricks component. I believe a lot of you are already familiar with this u the different components. So we will go through like okay we talked about a lot of like aspects how to build it how to collect the data train the model and have the cold start problems but how do we do this with data bricks component.

28:05 Implementation with Databricks

So on data bricks we have mosaic AI which is the singular most important point for placing your router because all your queries would be coming in through Mosaic uh API gateway and that means that's where you need to have your intelligence where the LLM router is sitting in and understanding what is the prompt complexity and other features and um rerouting it uh appropriately to the right LLM. Um this is also where you can do the traffic splitting like AB testing online experimentation evaluation and all the logging related to latencies prompt what is going in which model is picked up all the training data would come up from here uh unity unity catalog we talked a lot about like the features like what are the different features for the prompt for user for model that can be put into the model and bake a really intelligent uh LLM router so that's where the unity catalog will come into picture and also for semantic routing we would want to have a prompt embedding and a vector databases which is um provided uh by datab bricks so we can use those component uh when we are deploying the uh llm router or when we are doing any kind of experimentation that's where the ML flow will come into picture which understands all the all the model related experimentation evaluations different configuration artifacts and uh traces of the model and model registry where we have we might uh want to deploy like multiple different routers into the production different on different use cases.
So that's where you can have all the model like registered there and used for inference uh mosaic AI evaluation framework and this is the there I think they like I just attended the talk in the morning where with MLflow 3. 0 They are enhancing a lot on the AI uh agent evaluation which would be very useful in that this particular case and they are also having uh the LLM as a judge which is kind of like a black box sitting we like you just provide the input problem configuration and it can basically just produce a LM as a judge for your specific evaluation data set and say that okay based on these characteristics please evaluate like this response and provide me some uh understanding and rubrics based on Okay.
So, uh this is a little bit like a lot of text there, but let me go through and this is important because that's where like the end to end process of the router flow would happen. The first would be that uh the request comes in API gateway received the prompt from the user and it forwards that prompt or uh to the routing agent. So, this is the AI agent that is deployed within your Mosaic AI. Uh once the router received a prompt it fetches all the features from the catalog and other like feature restoring vector databases out there uh and then it will put it into the format of how the model is trained and give it to the model.
Model will give a response. Okay based on uh these features that you have provided I think uh the model which is good would be like llama uh 8b u the lowest cost and it would do pretty well on the uh prompt that is provided. So now once that classification has been done Islama 3B model that you would want to use then it will be uh putting it back to the API gateway API gateway knows what exactly are the endpoint for each of the models which are deployed in the L&M router um and then it will call that particular model API to get a response back.
So now you see like how just like the okay let's talk about the architecture if you do not have a routing agent it goes to API gateway API gateway will call the default LLM and you will be inferring all the cost of that LLM and whatever is the response quality but now you are intelligently rerouting your uh LLM APIs and which one to use how to optimize the cost and still maintain the user performance. So this is something where a lot of your cost reduction would happen and you can smartly understand and say that okay which LLM to use for which for which particular purpose.
Okay. Uh and then this is how like the end to end flow would happen at a high level. Uh so the training data which we talked about earlier like all the payload logging the API gateway would contain a lot of information what is going into the system what is going out back to the response to the user everything is logged in through the API gateway. So you can have the user ID, what is a prompt, what is a response, which model was selected through the API router and the latency and other related things.
And this is something which is useful for your uh training. And then all the usage based matrices are what is the cost of input output which is usually static but it keeps on changing. So those catalogs information would be pre present in the unity catalog and user feedback positive negative things which is logged back into the uh data brick system and then you can use the spark ML to join all the things because here you have all the feature the input output model latency and uh other static features like what is the input output cost for that model and uh explicit and implicit feedback.
So this is all the data set like can be converted into a training format normalized scaled to make sure that uh you can retrain your model and router again based on that. Uh for the feature engineering uh we talked like there are different features from the prompt users and models we can use. Uh this is just more like what are the different database component you can use gateway logs park uh the client API for example from for the prompt embedding uh there are you can use uh open source closed source data bricks inhouse and different kind of prompt like embedding models and you don't know which one is good and which one is bad for you.
So there are a lot of experimentation that can be done on this part and uh databicks does provide the client API so directly callable from there. Okay. And on the model feature side uh all most of the features would be present in the pre-processing using a spark and then the catalog. Um so yeah on the semantic classification uh this what we talked about is that like you can have the uh open source or any kind of models deployed on the model serving to convert the incoming queries into the vector embedding and this is something you can use for semantic classification and finding what are the similar queries that has been served and which is the uh chosen LLM router model for that.
Um uh vector search we already talked about this. So yeah clustering uh offline clustering using the logs and spark ML uh and in online um semantic routing uh whenever new query comes in generates an embedding based on the model which we already using u have the vector search databases to find similar queries and find the best lm for that. So for evaluation like traffic and splitting can be done by API gateway log collection and analyzing different metrices using different ML component and using ML flow to track different experimentation down there. uh how do we measure the cost uh and performance like uh we talked about it like in a a technical manner but here you can build dashboards and other things over to API gateway logs and cost table that is there in the unity catalog and get as much insights uh have your custom dashboards and other out there uh from the quality scores point of view like LLM is adjust using agent evaluation in the mosaic and u based on that you can get like different quality scores and what is the ROI bas based on your custom or use case specific matrices.
Uh we are also there in booth 121. Uh so uh feel free to drop by at a meta booth and uh yeah so this is it. Thank you very much.

Learn more about the Databricks Data and AI platform.