Beyond AI Accuracy: Building Trustworthy and Responsible AI Application Through Mosaic AI Framework
Summary
- Custom AI Metrics: Generic LLM metrics are insufficient; this video shows how to build bespoke custom metrics tailored to your organization's specific GenAI application needs.
- Responsible AI Practices: Discover evaluation strategies, autonomous AI evaluation suites for multi-agent systems and ground truth generation at scale with Mosaic AI Framework.
- Production Monitoring: Learn how to implement robust AI evaluation frameworks that ensure quality outputs are not only powerful but also relevant to your organization's unique requirements.
Beyond AI Accuracy: Building Trustworthy and Responsible AI Application Through Mosaic AI Framework

Talk by Ananya Roy, Specialist Solution Architect, Databricks
Generic LLM metrics are useless until it meets your business needs.In this video we will dive deep into creating bespoke custom state-of-the-art AI metrics that matters to you. Discuss best practices on LLM evaluation strategies, when to use LLM judge vs. statistical metrics and many more. Through a live demo using Mosaic AI Framework, we will showcase: How you can build your own custom AI metric tailored to your needs for your GenAI application Implement autonomous AI evaluation suite for complex, multi-agent systems Generate ground truth data at scale and production monitoring strategies Drawing from extensive experience on working with customers on real-world use cases, we will share actionable insights on building a robust AI evaluation framework By the end of this video, you'll be equipped to create AI solutions that are not only powerful but also relevant to your organizations needs. Join us to transform your AI strategy and make a tangible impact on your business!
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
Why are generic LLM metrics insufficient?
Generic metrics don't capture the specific business needs and requirements of your organization, so custom metrics are necessary to ensure GenAI applications deliver real value.
What does Mosaic AI Framework provide?
Mosaic AI Framework enables building custom AI metrics, autonomous evaluation suites for complex multi-agent systems and ground truth generation at scale for production monitoring.
How do you build an automated evaluation suite?
This video demonstrates how to create an autonomous AI evaluation suite for complex, multi-agent systems using Mosaic AI Framework capabilities.
What are key practices for responsible AI?
Key practices include accurate evaluation of system performance, observability at the internal level of agents and automated monitoring for production deployment.
Full transcript
Cool. Uh hello everyone. Thank you all for joining today. Uh. I am a specialist solution architect here at databorks. And what I'm going to talk about is a pretty important and relevant topic when we are working with agents. and that is becoming more and more predominant nowadays. So it's about how we build these type of systems responsibly. So couple of key takeaways I want you to get out from this session. Number one is there is a fundamental shift in our thinking about how we have been utilizing or implementing responsible AI practice and how it change when we are talking about agents.
So that's something which we are going to talk about today. I'm also going to talk about why the way we have been creating different matrices those are not sufficient when we are specifically talking about agent tech systems and what is the need for custom matrix. Um I'm going to show some of those capabilities through mosaic AI evaluation suite. If you have been in the session today or yesterday, you might have seen there are lot of announcements on this product.
There are a couple of product sessions as well. Just a disclaimer that this is not a product deep dive. There has been other sessions by the product team. So if you are interested um on understanding more about it, highly encourage to watch those sessions. Um and lastly I will basically conclude my talk with some of the best practice which I have been seeing across different customers and what we should be thinking about when we are thinking about an automated evaluation suite um for your agentic application that is trustworthy.
Now um just to understand about what is responsible AI right so from the lens of agents there are a couple of different pillars and I'm not going to cover all of them but the major ones I'll cover today so number one it's a given accuracy and evaluation so it is important when you are building this type of production application to assess how accurate they are performing and there are different measures and which we are going to cover today.
Observability and transparency. So that's one of the pillar which goes kind of hand in hand which basically says that I'm not only going to look at the output quality of my agent, I'm also going to understand what is happening at the internal level of his agent. So that is actually giving me quite a good understanding of how my agent is working and I see foresee a future where we are not only talking about agent in its itself we are more talking about a software application where agent genai LLM is basically one part of the equation so that becomes important second is safety and guardrail.
So if you are building a state-of-the-art application and you do not have any safety and guardrail measure, you might as well not go into production. So that is really important when we are talking about implementing a responsible AI um for an agentic application. Um human centric I think we have been talking a lot that no matter whatever state-of-the-art application you are building, you need to have human in loop. You need to have that final control of yourme or your human or whoever you want to have that final say whether this application is trustworthy to go into production or if things go wrong what should I do and how should I do number four and five basically yes four and five are basically about production when we talk about productionizing these applications application monitoring um and security that are these two matrix are also pretty crucial.
Um that's basically saying say you have evaluated your application it's working as you expect but there is no monitoring system in production then how would you ensure that these applications are performing as you expect it. So monitoring and security goes hand in hand. And last but not the least governance that actually goes basically for all of them whether you are building whether you are implementing whether you are monitoring um governance should be the basic principle which should be present in all and every aspect of um your agent system for the purpose because this is very huge topic.
I'm not going to cover all of them today. What we are going to cover today is these three um and I'm going to go a little bit more detail on these ones in the subsequent sessions. So why evaluation is important? Um it's kind of like self-given of um I am building some application be it a data application AI application I need to ensure that is working as my expectation. Um but I just want to put some real world examples in this case to make things more relatable.
Several um was one company um who has build one conversational agent for their user and one of the user they later bragged about it um is they were able to manipulate the prompt and got a $76,000 car in $1. and because it was done through the official channel um there has been legal proceedings which has happened afterwards. So it's a pretty big risk factor. Similarly for Air Canada, very similar situation happened where their conversational agent was interacting with a customer and provided due to some misleading behavior provided a refund more than the said amount and that went on to subsequent court cases what not.
So the idea what I'm saying is that you will get more and more applications like this but if you do not have the right evaluation practice or a trustworthy application in place these situations are bound to happen. Before going into the detail of how would he do what is um what are the different ways we can do it there is some fundamental things which we I want to clear. Number one is evaluating a language model is not equal to evaluating your AI agent.
And let me unpack that a little bit more. So when we are talking about LLM evaluation, we are basically talking about the capability of a language model to produce an output. What does it mean by that? I have an input and then I am evaluating how good the language model is producing the output. be it a summarization, sentiment analysis, categorization, so on and so forth. So your your environment is quite constrained and you already know what is expected at least in some sort of form.
But whereas agent is different. Agent is not only generating answer, it is giving you an output basically doing some action. So things becomes completely different when you are expecting to do certain things for an application. Similarly for language model as I mentioned your environment is uh static you are using different types of um user parameters or user question but for example in a summarization use case no matter what type of user input you are giving you expect a certain summary so that's the end of it but for agent it's very different because what you are doing is output are changing based on your in information or based on the user input.
So if one user is asking a question which is um taking 20 different tool executions for the model then it's a complete different ballgame altogether. And similarly when we are thinking about agent majority of the time if you have already built it in your organization they are purpose-built. So you should not be um validating a code generated agent with a conversational agent. The metrics will be completely different.
So and last but not the least cost and I think that is something which is not spoken quite widely but I think that is really relevant reason because for a language model at least if you know the input and the output token length you can estimate the cost maybe not 100% but you can estimate but cost is unbounded when we are talking about agent and the reason because as I mentioned you can get For one individual user question, you can get one tool execution and then you get the answer.
But for a separate user question, you might think of there are multiple sub aents which are calling each other. They are going in loop. They are getting informations from any number of tools. So you cannot predict that beforehand. So cost can quite easily go off ceiling if you are not kind of like considerate about cost when you are evaluating these type of applications. So let's understand evaluation a little bit more of what we are doing today and why I am saying that might not be sufficient.
So let's imagine you are creating one AI application and then one of the most common use cases which we are seeing across customers are maybe some sort of a batch use case summarization use case classical sentiment analysis whatn not you have certain guardrails and certain type of evaluation practice in place example statistical matrix human and loop lm judges all those things are fine um again sometimes you're using rag type of agent then you're using the different metrics like context relevance groundedness so on and so forth but if you just take one step back and think that all of these are on an on a high level looking at the output quality you're still thinking of your application as a black box maybe not so much for RAM but overall so it is looking at how your output quality is but it's not going under your inside your application or inside your agent to understand how it has derived to this situation and that becomes really important when we are talking agent um because there are multiple tools multiple things are coming together to give what the user want or what you want.
So it's kind of quite analogous to say that you are not judging a chef by the dish he produces. You're basically looking at what ingredients was used, how he has come into this conclusion or how he has created the dish, not basically burning down the entire kitchen to give you the dish. Right? So that's the idea of we should have an understanding of each and every step of your agentic workflow. Now custom matrix, what is the reason these standard matrix are not enough?
I'm going to give three different examples just to put things into perspective. So we start with a simple summarization then one of the most common one which is rag and then we are talking about agentic apps. So for example summarization majority of the statistical matrix would be sufficient today if you are working with a summarization problem they can say very well how it has summarized so on and so forth if it's coherent what not what if I want to understand more I want to understand if this summary has captured my company's risk policy or not has it captured the relevant information I need for which is relevant for my company.
Has it done that? No. I can I can I assess this if I go with the normal standard metrics? No. For rank, if I want to ensure that all of the retrieval context are coming that are always the latest and there are now duplicate context which is returned. Can I do that with current standard matrix? No, we can't. And things becomes wild when we are talking about agent because for agent some of the basic question we should be asking is whether the right tool was called or not whether the routing was correct or not whether the tools trajectory was right or not.
We won't be able to answer these if we go with the standard matrix or standard way of evaluating. We need more and that's where custom matrix comes in which is basically saying create your matrix so that it is aligning with your business goal and that matters to you and just not some random generic metric. I'm going to explain that a little bit with a hypothetical scenario of a financial portfolio analyst and that is something which I'm going to demo as well.
Um so let's imagine I have built one agent which is a portfolio analyst and there are multiple tools involved and this is giving um suggestions to customer on their portfolio what they need to do um understanding fraud so on and so forth. So with that say for this agent user comes in and they ask a question about I want to rebalance my portfolio from stock X to stock Y by 10%. Can you tell me what would be my future situation looks like whether I would be in profit or loss?
Explain it to me and give me a suggestion. That's a loaded question because uh the agent need to do any number of things to give you a correct answer. So there is a bit of planning involved. the model will understand this um so the brain or the model which is kind of like the center part of an agent if we are talking about single agent they'll understand what are the tools I have what are the tools I need to execute so for example first it needs to understand what is the current portfolio status for this customer then it might need to go and find out the relevant stocks this customer has perform certain sort of simulation and then give user the Result fraud selection detection.
This is one of the tool present. Would this be important for this question? No, might not be. So the agent need to assess what are the tools it needs to execute and then execute it and give the result back to the user. So what you're seeing here is that it's not only one single person, single component which is making the decision. It's a combination of all of these. Now how do I ensure of such complicated system and this is not complicated I'm sure that for your organization your things must be much more complicated um when we are talking about real use case so how do I ensure that this is actually I I build some sort of a responsible system which is trustworthy um I can trust so that my customer also can trust number one we need to assess the overall quality that's a given that's non-negotiable you need to kind of like assess how it is performing.
That's what we are doing today. No problem in that. Um we also kind of like crack open this agent and understand the inner workings so that I know that based on certain user question the routing has been correct. It has gone to the right tool, it has gone to the right sub agent if need be. The trajectory that's quite interesting um and I'm going to show you in a moment. Um once you have understood that okay maybe five different tools need to be executed in order to answer a particular query sometimes the execution order is also important.
So for example in this case if the stock analysis tool has been executed first before the portfolio analysis basically it is null and void because what we are extracting what we are looking for if we don't even know what is the status of the customer. So sometimes this trajectory is also important not all the time. um chain of thought reasoning. If we are using some state-of-the-art models, majority of the time that is the case, we need to understand if this decisioning is logically consistent or not.
And it's very similar to planning. So the planning has been correct, whether the execution has been correct or not. And I can go on and on but these are some of the things which comes top in my mind in terms of evaluating an agent. So what I have done is I have talked a lot about different ways of evaluating what not. So what I've done is I've created three different categories. That is something kind of like a ballpark figure you can think of when you are building this type of application.
Number one, I'm calling it as a macromeatrix which we are already doing today which is basically the standard matrix like statistical matrix relevance correctness then safe toxicity bias these matrix we should be doing regardless we should be validating these matrix regardless and see how it is performing so that's that and then I'm going to also validate something called micrometric which is basically the inner workings of your agent, the entire execution flow of your agent and understand how the agent has come to a conclusion and what are the reasons.
So it's giving you that why and how answer some of the matrix would be as I mentioned routing and tool calling tool path trajectory um individual tool output. So a same tool with different parameters would be giving you completely different result. So whether the input and output was also relevant or not that's something you can also check. Basically, you can go as deep as you want if you have the right information, chunk relevance, so on and so forth and I can go on and on as I mentioned and then so these are all micro and then we cannot forget about the performance matrix because as I mentioned cost can really go up to the ceiling if we are not considerate about it.
Um and think about the situation that you have two agents. Both of them been asked the same question. Both of them has given you the correct result. One has executed two tools. Another one has 20 tools. Which one should you choose? Because every tool execution is associated with a cost. So cost becomes important when we are thinking about production scale applications. We are scaling with multiple customers so on and so forth.
So how can you make kind of like a generalized architecture of a robust evaluation framework? You have your AI application, you have the macro metrics, you have the micro and then you have your per performance. All these three three things together can go into your business decisioning process. And with all of these matrix together, you can make a informed decision of what should we do? Should we ship it into production?
Should we do some sort of tweaking so on and so forth? Now I have talked enough about matrix evaluation. Let's talk a little bit about observability. Everything I have talked about in terms of tool trajectory so on and so forth. How can I achieve that if I don't have the right information? And that's where observability comes in. that is basically saying I have all the information I need to create those metrics that is relevant to my business need.
So I am not considering my application as a black box. I am fully aware of how any conclusion was made. um I know the why and then the way I can do that is if there is any mechanism through which I can log every execution of my workflow and that inevitably will be increasing trust and uh reliability on my application. If you ask on data bricks how we can do that um I'm sure you have heard about MLflow trace. So MLflow trace is our capability when you are building agent inside the platform.
It has the capability of kind of like take a snapshot of every execution and then hold all the information about that trace um in a format that is kind of like widely expect accepted which is open telemetry format. So in this picture what you are seeing is one single execution. You have your agent you got the final answer. You can see at every span what was what has happened and then what was the tool that was executed and it can grow more trust me trace can go more because your execution can be complex.
So that's observability safety and guard rail. Um I think I have talked a lot about this but this is just kind of like a guiding architecture. I know for real use cases for it it can be quite fluid but the idea here what I'm trying to say is if you are building your agentic application if you have evaluated you need to have some have some sort of guardrail. So whether those are dev guardrails like safety, toxicity, bias, PII detection what not or whether these are production guardrail where it is saying I have the correct rate limit.
I have the usage tracking so that I know if something go wrong I have all the mechanisms to prevent it or if it happens I know how to backtrack it. And it's completely up to you whether you have you kind of like combine all of these guardrails into one or split it out. But the idea is there has to be some sort of guardrail mechanism which should be present when you are building and planning this type of application. Now let's understand a little bit about how we can do this in databicks.
Now I will show some of the capabilities of Mosaic AI evaluation suite and then from the lens of a financial portfolio advisor agent which I have already mentioned. So okay this is what I have built. So this agent what it is doing is it is um as I mentioned a financial portfolio analyzer agent. I have created a bunch of different tools here. Some of them are extracting and authenticating customer. Some of them are extracting portfolio details.
Some of them are looking at the internal knowledge base portfolio rebalancing so on and so forth. I have stored all of them into unity catalog and then I have created an agent and served it via model serving endpoint. That is my agent which I'm kind of currently using everywhere. Now let's understand how I can evaluate the performance. So that's my goal. I need to understand how um my application is performing. So I go here.
What I have done and what I'm showing you here today is I have run my application across different type of questions and those questions I have kind of like uh deliberately created some edge cases question because I want to ensure that my application is performing well because to be honest let's be honest we can't trust humans. they can ask any questions in any format because we are intelligent species and um that's why to best of our capability we can create um as much as possible test cases to assess how my application is performing.
Um so these are some of this is the UI how it is looking like. Now let me go through with some of these metrics. What you are seeing at the right hand side they are all custom metrics which is I have created this. Some of them are called as for example it's saying conflict of interest scorer which is basically saying the output which I am producing is there any conflicting um information that can kind of like have a negative impact on my reputation. correct routing which I mentioned earlier whether the routing is correct based on the user question how can I do that and remember all of these I have created with the help of MLflow trace the inputs and the outputs that's it that's only the information which you need safety tool trajectory total number of tool count and that becomes quite relevant for example if I am trying to assess multiple different agents or multiple different versions if the same questions is answered with a smaller number of tool call.
Then obviously I will be choosing that agent that becomes kind of like higher in my performance bar. Now let's unpack some of these um custom matrix a little bit more. Let's see some of the faults. What I can see is that routing is 83% correct, 17% incorrect, which is not what I want. Maybe let's double down on this question a little bit more. So the question which was asked is with post code blah I want to rebalance my portfolio from Netflix to Amazon by 10%.
I want to understand what would be my future portfolio looks like provide me some suggestions blah blah blah. Now this is our latest kind of like UI and I'm going to show you the code also how you can achieve this. So what you are seeing here a summary version of your um agentic execution. So you have got your chat then it is executing multiple different tools one after the other. First it has extracted customer details then portfolio details simulation stock inquiry.
Now I'm not sure about stock inquiry because I don't think I need to get a realtime stock inquiry. That is quite suspicious. I need to look onto this one a little bit more. Then internal knowledge search but yeah I don't need this but why it has come up. So this is my debugging process. If I want to understand a little bit more, I can go into the detail and you can see the detail execution flow, the inputs and outputs of every execution.
How MLflow traces and what were the input, what were the outputs, so on and so forth. Um so this gives you a detailed idea. Now let's look at the scorers. Scoders are the matrix which I have been talking about. Um let's look at the one which has been showing as false. So false it's saying so false is based on my ground truth. I already have created my ground truth data saying that if the question is on this route you should be now there are a number of ways which you can create your ground truth you can take help of LM judge or you can create your own bespoke ground truth data as well.
Um so there's no problem in that for this use case I have said that if the question is about portfolio rebalancing then the expected one is about you would authenticate your customer then subsequently extract your portfolio de then you would be performing a simulation of rebalancing and whatn not but ideally the model has gone actually gone uh too far and it has um it has it is actually more intelligent than I feel I believe.
So it has gone and then um done all those three things and also extracted those realtime stock of Amazon and Netflix currently as of today. Then went into my internal knowledge search find out relevant information and then provided a suggestion and that might be correct in some situations. But if I'm in a highly regulated industry and I have strict guidelines of what to follow and what not to and maybe I do not maybe my company does not want real-time stock information but it has gone regardless.
So then it can create a problem. So that's how tool trajectory is one of the matrix that is giving me a little bit open idea about what's happening under the hood. Similarly, if I look at the other matrix, so total tool count number six, which is quite big in answering this question, but it can be expected if the question is complicated in itself. So, if you have right ground truth data, you can evaluate your model's execution based on your ground truth.
And if you don't have ground truth, then also there are ways you can generate ground truth through your theme evaluations and whatnot. Or you can actually um use other matrixes which are not reliant on ground truth. But if we are talking about understanding the tool calling and flow then ground truth is important. These are not related. The reason because this correctness and relevance to query are specific because if we are using internal search and whatnot.
Um safety is fine correct routing. Um so for example I always want to ensure that user is authenticated before answering any questions. So that's why extracting customer details are important. So that has been done here. So that's why it's a pass. Now it is not bounded by this boolean true or false value. You can also create you can also create this number. So boolean numbers decimals whatnot. you can create this kind of like a quantitative um matrix that you can use in your automated evaluation workflow.
So no one is stopping there. Um so that's a pretty powerful UI as well as you can get data frame to make um robust evaluation pipeline. So that's that. Now if I come back on how have I achieved this. So I'm not going to go through all the bits of the code but what I want to say here is that it's pretty easy to build this applications in itself and custom matrix. So for example I have provided this prompt saying that okay for conflict of interest this is my prompt.
Um I have mentioned this decorator. This is how MLflow understands of how um it is a metric. So with the scorer function in MLflow 3 and the only thing which I need to be worrying about MLflow geni evaluate that's the part which is doing all the magic which I haven't been talking about. Um and then you pass your data and the metric and rest of it is self-explanatory. So with that said um what you have seen here is that it is pretty easy to build custom matrix and basically the world is your oyster here.
As long as you have the right logs and the traces um you can build custom matrix that matters to you and build a trustworthy application. Now with that said I want to show you what are the some of the best practices you can take when you are building um a robust evaluation pipeline. So when you evaluate what are kind of like some of the best practice you can take. Number one, say you have obviously the first part is building that.
Once you have built it, you create a good test case planning which would be um ideally it should have some sort of simulation of good test cases, bad test cases, adversarial scenario, whatnot, then you generate ground data. If you don't have if you have then well well and good you don't need to worry about it. But if you don't have ground truth data then generate it. We have our evaluation generate eval um data frame API that's pretty easy to generate these synthetic data and then you perform the evaluation and then you iterate upon it.
So there is a way in MLflow trace and I'm more than happy to show you afterwards if you are interested to basically compare different runs. Say for example you have changed your prompt you again run it your application performance has gone up by 5%. And you want to see the difference. So iteration is important. There is no single answer when we are talking about genai or ML in general. So you iterate upon it. Once you're happy with the application or the output then you go into production and production monitoring should go hand in hand. do do not kind of like think about I can do it later because this becomes more and more important even more than evaluation when things are in production.
Um and that loop which I mentioned earlier should be going there should be some sort of feedback loop which is going back to your agent development life cycle and whether you are validating through humans or through any alert mechanisms through machines that's up to you. Um that's the overall flow I would say. Now how we can help I have shown a part of that um application. How we can help is in this application we have everything which is needed for you to enable yourself to build this application.
We have Mosaic AI development suite which is helping you to build that agent. Then we have got our evaluation framework which is having everything as I mentioned in terms of building a robust pipeline and then tracing it's kind of like closely integrated with all the services which I mentioned and lakehouse monitoring um there were some uh talks about lakehouse monitoring I think in the keynote and also in the session that is actually talking about how we can create even more robust production monitoring system when we're talking about genai Um so all of them together are helping you to build a trustworthy application.
Now I want to conclude my talk by saying that we have understood a lot of different things. Um the first thing is we have gone through the different pillars of responsible AI. We understood what is important and how we need to think a little bit differently when we are building and evaluating agentic applications. why custom matrix are important and how we can build it. Um visibility, observability and trace that's prime if we want to implement this in a proper manner.
And then guardrail implementation is also kind of like goes quite hand in hand. Um I would conclude my talk in a sense that this place is moving at lightning fast speed. The thing which I'm talking today might not be relevant 6 months or one year down the time. So we need to think about when we are building this system a little bit futurep proof. So that's why we need to build a system in such a way that these can adapt to the newer changes and make it modular as much as possible.
So with that said, thank you everyone for your time and have a good rest of your day. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.