AI agent evaluation at scale: 52x ROI with MLflow and DSPy

Summary

Zepto, a quick commerce platform delivering in 8–12 minutes across 60-plus cities in India, scaled autonomous customer support to 80,000 daily AI-resolved tickets using a dual-loop evaluation framework powered by MLflow and DSPy.
The evaluation infrastructure achieved a 52x ROI by reducing support costs by 65%, improving customer satisfaction by 200%, and compressing issue detection time from 3 hours to 5 minutes through real-time tracing and LLM-as-Judge evaluation.
The dual-loop architecture separates a development loop using golden datasets and DSPy prompt optimization from a production loop with real-time monitoring, alerts, and stakeholder feedback, enabling rapid iteration across 50-plus agent skills.

AI agent evaluation at scale: 52x ROI with MLflow and DSPy

At Zepto, scaling autonomous support to 80,000 daily AI-resolved tickets demanded treating evaluation as core infrastructure. this video unveils the comprehensive dual-loop framework powered by MLflow and DSPy that transforms operational risk into competitive advantage, achieving 52x ROI on evaluation infrastructure.
Learn how Zepto reduced support costs by 65% while improving customer satisfaction by 200%, and compressed issue detection from 3 hours to 5 minutes through real-time tracing and LLM-as-Judge evaluation. Discover the development loop for building reliable agents with golden data sets and prompt optimization, the production loop for real-time monitoring with alerts and feedback, and how strategic evaluation dimensions capture input from all stakeholders. Understand Cohen Kappa for measuring inter-rater agreement, multimodal evaluation challenges, and the composable architecture that enables rapid iteration across 50+ skills.
🤝

Chapters

FAQs

What is a dual-loop AI evaluation framework?

The dual-loop framework separates agent development from production monitoring into two distinct loops: a development loop for building reliable agents using golden datasets and prompt optimization with DSPy, and a production loop for real-time monitoring with alerts, LLM-as-Judge scoring, and stakeholder feedback. Zepto implemented this architecture on Databricks to manage evaluation across 50-plus agent skills.

How did Zepto use DSPy and MLflow together for agent evaluation?

Zepto used DSPy for prompt optimization in offline evaluations, automatically tuning prompts against golden datasets to improve agent reliability before deployment. MLflow provided experiment tracking, tracing, and the infrastructure to log evaluation results, making it possible to iterate rapidly and maintain audit trails across all agent versions.

What is Cohen Kappa and why does it matter for AI evaluation?

Cohen Kappa is a statistical measure of inter-rater agreement that quantifies how consistently human evaluators or LLM judges score agent outputs, accounting for agreement that might occur by chance. This video covers its role in validating evaluation dimensions so that scoring is reliable and reproducible across different stakeholders.

What business results did Zepto achieve with their AI evaluation infrastructure?

Zepto automated 80% of customer support tickets, reaching 80,000 daily AI-resolved tickets, while reducing support costs by 65% and improving customer satisfaction by 200%. Their evaluation infrastructure also compressed issue detection from 3 hours to 5 minutes, and the overall investment in evaluation delivered a 52x return on investment.

Full transcript

[00:07] We'll be going over the specific case study in the first part. Uh and then in the second half of the presentation, Girish will be giving a master class on the building blocks of AI valves and uh the key frameworks to keep in mind while building those. Cool. So what's Zepto? Right. So uh
[00:24] since um this is US audience I thought I'll give a bit of a background about what Zeppto is. Uh Zeppto is a quick commerce app. Think of it like Door Dash or Instacart but only that the delivery happens in 8 to 12 minutes which is great. So uh we are currently
[00:40] operational in 60 plus cities across India. We service mult uh millions of orders every day uh across a number of categories from fresh produce to uh high value electronics uh home needs uh pharmacy all of those
[00:57] things right so it's a one-stop shop for our users to come in and transact and it's slowly becoming like a platform of first choice uh for e-commerce where people come and check if they can get something in 10 minutes otherwise they'll go to other platforms uh if
[01:12] something is not available. So with that in mind um since people have use our users have started relying on us for uh you know more and more important use cases over time uh support becomes a major part of this journey right we want people to have good support so that uh
[01:28] they can keep coming back and transacting on the platform and um you know it also helps us build trust in the long run. Cool. So you know what are the bottlenecks for providing good support? Uh with the business growing you know 100% yearonear you have topline growth
[01:45] which is linearly correlated with the number of support tickets you get. Um second one is uh category category expansion. So over the last couple of years we went from being just a grocery platform to a platform that serves basically anything. So as I mentioned
[02:01] you know apparel, makeup and beauty, skincare etc etc right? So um that increases the complexity of the support tickets that we get. So we have to support exchanges, refunds, um returns etc. Right? Lastly, user diversity. So
[02:18] we have seen quick commerce pick up from COVID and we initially people thought there was just like a covid era trend but we started seeing that more and more people are still coming on the platform and transacting. So our user personas are you know widening as we go along.
[02:34] And then also as we expand across India there's multiple languages and so essentially it's complicated for support as well because everyone has their own ways of communicating so we have to serve everyone right so a trifecta of
[02:50] these things coupled with you know uh seasonality and festive bumps as you can see we have seen like an exponential growth in the number of tickets that has outpaced our actual business growth which is already pretty fast, right? So, um we could not
[03:07] just keep scaling our customer support operations um you know in that manner, right? So, we needed like a uh solution which was you know infinitely scalable, reliable um had a lot of context. So basically context aware system and uh was able to
[03:27] interact with all the internal systems like order management, weather API um you know bunch of those kind of things to be able to provide the right kind of resolution very quickly right so hence we sort of started this journey in the
[03:43] February of last year where we said okay let's you know put aside the existing system and you know start building something from the ground And uh we started building like an AI native system uh which could handle multimodal as well as uh conversational
[03:59] use cases and interact with like a bunch of internal systems like order management, customer data platform, last mile systems, weather API, policies, SOP and also triage issues and fall back to humans whenever necessary. Cool. So uh you know going a little bit
[04:16] deeper into what the system actually does, right? Um especially the AI stack. So I'm a data scientist so I'll be covering mostly data science related stuff. So of course the engineering part horizontal scaling all of that is difficult. So but uh pardon my you know glancing over that part of the system
[04:32] architecture. So um we have three main layers right one is the horizontal layer where we have u m a host of AI agents plus ML models that are essentially used to curb like abusive patterns. Then we have the intent routing layer in the
[04:49] middle that basically classifies tickets and routes them to like a specialized vertical agent that has the right kind of context and the tools to be able to u solve your issue. Uh and then on the right side you have you know vimo means like where is my
[05:06] order missing items expired items returns quality and so on and so forth. So this is just a few of those things. I could not put every single thing we keep adding new things every month, right? So all right. So with this system in place, you would ask the question why do we
[05:22] need agents? Uh why do we need evals for these AI agents, right? The purpose is twofold, right? One is uh you want to build build with confidence, right? What does that mean? That means that um whenever you're building, you're making a bunch of choices uh design choices uh
[05:39] pertaining to what model you want to use, how you want to write your prompts, whether to use chain of thought, uh what temperature your model should have, uh what else uh what are the hyperparameters, right? How much thinking budget to give your AI agent, all of those things, right? So,
[05:56] um and then overall like what orchestration do you want to use? Does Langraph work better for you? Do you want to create a specific sub agent within an ADK? How much autonomy to give to your orchestration layer? All of those things, right? The second part is um you know so AI valves help us sort of
[06:13] build uh and evaluate those things those choices so that you can have evidence-based design rather than you know performing guesswork while you're building your agents. The second one is um AI agents in production. So after you deploy those you also want that uh you
[06:30] can you should be able to measure the performance uh in a quantitative manner uh from all different aspects right so uh you can do that by using multiple scoring methods that Girish will be going over in the second half of the presentation so I'll skip that for now
[06:46] but production uh essentially a eval in production help us catch issues that your agents have so that you can take that as feedback and you know, instead of collecting feedback from social media or anecdotal evidence from your product
[07:01] managers, um, this is a good way to see how big of a problem there is and then you can basically decide how you want to which issues to prioritize, how do you want to fix them. Uh, so it becomes like a complete loop. All right. So, uh, since we have
[07:18] multimodal as well as conversational agents as part of our stack, uh, the evals for these two are slightly different. So we'll be going over those separately. So for conversational AI agents uh there's a bunch of judges that you can use right to perform these evals.
[07:34] The first kind is LLM as a judge or codebased judges or model based judges as you say right. So uh the first one is intent accuracy. So if you saw the deep dive architecture you saw that there was a classification layer. So you want to identify how accurate that
[07:50] classification is so that the tickets that are coming in are going to the right specialized region and not getting mclassified and then getting uh escalated to humans because it's out of scope for that specialized region. Right? So uh this does that we use a so
[08:06] the whole idea behind LLM as a judge is that you use a bigger model on a sample of production traces to be able to measure the performance um in a statistical I mean you do like statistical estimation right so that's that the second one is groundedness so
[08:22] this is the main one this is like a litmus test for your hallucinations you want to make sure that um the responses are relevant uh and also they are uh the responses are relevant and also you know specific to the
[08:38] adhering and the agent is adhering to the policy and it's not just inventing anything else right it's cancelelling when it's required to it's offering refund when it's you know permitted to and apologizing you know and showing the right kind of empathy so the third kind
[08:53] is obviously like a specific kind which again measures how their tone empathy and uh guidelines basically are we able to u adhere to the brand guidelines that you know our PR team or
[09:08] you know our product team has stipulated okay so the next set of uh judges are not AI based they are kind of deterministic or hybrid so schema validation again since a bunch of models that we use are coming out of uh US labs
[09:24] we want to make sure that you know we using Indian standard time or daytime format is the right thing we are not you know we have checks around not addressing questions related to competitors. So we have those kind of checks over there. Latency and token count as
[09:41] uh as it intends. Basically you want to measure these metrics to be able to see how the token count trend looks like at response level as as well as uh at ticket level and also you want to monitor your latency of responses. If you have multiple nodes within your langraph orchestration or you have too
[09:58] many chain of thoughts and you want to make sure that those are not adding too much latency over time. Last one is warning. So it's basically uh all the safety related guardrails that we have. So in case there's a user complaining about you know safety or they're feeling
[10:13] threatened, we should be able to escalate it and also provide the right kind of information. So this is kind of like a guardrail that you want to have. Cool. So, uh we'll be going over specific uh stories where we found something in using our evals and then
[10:30] how we fix those, how we used offline evals to develop quickly to give you a better intuition on why these are useful. Right? So, uh the where is my order agent uh we saw that the token count uh was negatively correlated with the customer satisfaction. Um upon
[10:47] looking deeper into those traces, we could see that some tickets were stuck in like a repetitive loop of you know giving stale uh estimated time of arrival to our uh customers. And this was specifically happening because of um
[11:02] on ground stress. There was too much rain or there was rider shortage or it was too late in the night or there was cricket world cup you know. So uh because of those kind of things leading to on ground stress and your order being in the pack state uh and not getting
[11:19] picked up for delivery um this the ETA got stale and the ETA service just kept giving us the same ETA because it was just not getting picked up. So uh upon finding this we were able to build a feature partner with our product team and release a feature where we went
[11:35] deeper and gave some more insights about the the operations process right instead of giving the overall ETA we built a system which could estimate the time of packer uh the rider assignment rather than the overall delivery time and then address the issue uh we address the
[11:51] issue that there's on ground stress so that the users can have empathy for you know what's going on behind the scenes. And then lastly, we were also able to uh offer them proactive cancellation by promising that okay, in instead of saying your order will be delivered in 7 minutes, we said
[12:08] that a rider will be assigned to you in 3 minutes. So uh and if there is no rider assigned in 3 minutes, next 3 minutes, you can come back and cancel your order for free. So this while we were not able to solve the on ground problem but we were able to kind of address some of the issues offer
[12:24] free cancellation build trust with our users and um you know improve the overall experience as a result. So how did we do that using the offline evals right we used couple of libraries that are open source and maintained by data bricks. So
[12:40] there's DSPI which is mainly used for you know optimizing your prompts selecting the right kind of flu shots u optimizing your chain of thought in the process. Then you have MLflow for experiment tracking and uh deeper evaluations and those kind of things
[12:56] right so and then based on this we were able to u thoroughly test our uh you know new feature uh which eliminated the need of you know brute force testing. So these systems are stochastic. Um as a
[13:12] result there could be something that is uh something that breaks because of the change that you make right. So every time you cannot come back and just do all your tests on your own. So you have golden data sets and MLflow automates that uh testing and benchmarking process for you so that you can um you know look
[13:28] at the results and make the right kind of tweaks in your orchestration or you know thinking budget promp selection those kind of things. Um so after releasing that feature we saw an instant uplift in seesat at least in those set of tickets. Um and this kind of helped us build a flywheel right that flywheel
[13:46] uh and trust that we had with the product team helped us address a number of issues and find new things and you know um address them and release new features. So this became like a process because um you know the product team started trusting us with uh our online
[14:02] evals they would they would go on our evals dashboard on data bricks and see what's happening come back to us and understand if we can deep dive into the traces and uh then release relevant features bug fixes patches whatever cool the second story is around out of
[14:17] scope uh intent so where is my order sure data set you had Yeah. What kind of thing was that? Is that like a set of questions they ask and Yeah. Yeah. So for intent classification
[14:34] it would be like what is the intent like is the person talking about uh let's say change in address or unable to contact a rider those kind of things. Uh you have subject matter expert teams and then you build a data set with them.
[14:50] Yeah. So um in the second one I mean uh going off of the golden data set part right so we have like an uh catch all bucket called out of scope. So there were a bunch of intents that were classified as out of scope and um our
[15:05] bigger model that was running u for intent for measuring intent accuracy. We added like a layer for uh basically topic modeling and it found that there were a bunch of subtopics that were classified as out of scope like uh users realizing that they forgot to add
[15:21] something in their cart before ordering. So they were asking if they could cancel and reorder or if we could add anything. So we didn't have that operations process to be able to put a order on hold and because we are trying to deliver it in 8 to 12 minutes, right? So those kind of provisions were not there.
[15:36] We were just trying to optimize for time and speed of delivery. Uh second one was change of address. Um you come back home and you realize that you ordered in your office address. So there were people asking if they could you know cancel the order and change the address. So um
[15:52] based on the topic modeling we were able to identify these out of scope intents and then release a feature where uh you can go and add anything in your cart um up to 60 seconds after you place the order. So depending on the operations
[16:07] process if the order is already packed it will come as a separate delivery or if there is a backlog in the packers then basically it gets clubbed in the main order. So um the point being that we were not only able to with the help of evals we were not only able to uh
[16:23] improve our agentic uh customer support bots but we were also able to partner with other teams and release useful features because there was anecdotal evidence around you know these kind of things that people have these kind of pain points but they were never able to quantify them and we were able to
[16:40] quantify with the help of uh our evals. All right. So, uh in the next section, we'll be covering the multimodal uh considerations, the nuances that you face while uh evaluating your AI agents that have multimodal inputs.
[16:57] So, um the first one is uh subjectivity, right? Subjectivity is a big the first main learning was that uh there's a lot of subjectivity in rating um of images. So our uh damage and rot detection AI agent essentially uses a Lyard scale and
[17:15] basically asks the AI agent to rate the the damage that they see u from one to five and one being the least amount of damage, five being the most amount of rot or damage that they can observe on a product. Right? So um the since this is
[17:31] a subjective activity uh the first learning was that you need to have a very robust golden data set. Right. So um you need to invest a lot of time and resources into building a robust data set with you know all kinds of products uh multiple humans rating the same
[17:47] images so that you can measure how much uh variability there is between humans rating the same image. Right? So as you can see one example one uh human rated this damage as two on five because there's only like a little bit of
[18:02] pigmentation. Another one said that no it's three on five. So um how do you quantify this? There's a metric called cohen scappa. So it's a metric that goes from 0 to one and one being perfect alignment between different humans. Zero being no alignment. So that helps you
[18:18] kind of understand what is your baseline error rate that you cannot go below right so it's kind of like base errors. So based on these insights, we were able to understand how to tune our policy. What should be our threshold for refunds? If the co scapa is really high,
[18:34] you need to have a really lenient policy because even the humans are not able to determine whether um the product is bad or good, right? Um it also helps you with your other uh you know design choices like I mentioned on the model front and all the prompt iteration those
[18:52] kind of things. Um and then you can benchmark your kappa against the mean error the uh mean absolute error that you have. The second one was uh around you know looking at the distribution. So this was our another main learning that we had
[19:08] after going to production. So we saw that while our mean absolute error was in line with the co scapa that we had. So um our mean absolute error was around 22% and our coen scappa was around8. So we understood that there's a baseline
[19:24] error of 20% that we cannot solve for. Um so after going live we realized that there's a neutrality bias that our AI agent had right. So while the humans were scoring most of the tickets uh at four and five because most of them were indeed uh rotten and damaged we saw that
[19:41] uh the AI tried to refrain from you know making any extreme calls and try to stay neutral around three. So again how do you solve this issue? uh by tuning your prompts, right? So, uh if you make your prompts too lenient, it'll try to give
[19:57] everyone a five and even uh detect normal pigment pigmentation as damage or rot. Uh if you make them too strict, it'll sway towards the left. So, it was an iterative process, but we caught these by looking at the distribution.
[20:13] The last set of learnings was around special case handling. So when you do stratified sampling on your traces, you can see which uh uh sub product subcategories have high MAE and which have low uh low ME. So uh we saw that in milk uh we were treating milk as
[20:30] packaged product but um you know a lot of people complaining about curdled milk. We could not validate using our images and we would reject their refunds and that would lead to a negative customer satisfaction. So we added a bunch of special case handling for this.
[20:45] Um last one was around taste and smell. So we saw that uh images cannot be used to validate any kind of taste and smell related complaints. So you have to lean in more on the uh the text input rather than the image input. So these kind of things you can learn from stratified
[21:02] sampling uh deeper analysis of your traces uh building more and more evals as you go along. Cool. Uh lastly what we did is we had anecdotal evidence from business teams and product teams about uh bunch of
[21:18] fraud and abuse that's happening on the platform. So what we ended up doing was we deployed like an moinder uh evaluation agent. So what it did was it would go and figure out um you know we would give it all the information related to the user, the order, the
[21:34] product, the underlying statistics around what the refund rate is and what geography this order is from all of that and we let it figure out um whether there are adversarial patterns. So it came up with multiple things and then we kind of bucketed it value evaluated
[21:50] those uh validated them using humans and then we consistently released a bunch of features that fix these loopholes like people uploading blurred images or duplicate images and you know bunch of things like that. So that also helped us curb the refund rates in the background
[22:08] without adding any hurdles in the customer journey. Uh lastly okay so uh we did all that what did we achieve right um through this process so we are 1 year and 4 months into this process and we have
[22:24] achieved a automation rate of 80 plus% we cross 80% like 2 weeks ago so roughly four out of five tickets that we get are sold by uh AI our u customer satisfaction has improved by 200% which
[22:39] is a major win for us it's not that we just want to keep pushing people to use uh AI support agents and never connect them to humans because that's a major frustration that we are seeing. So we also wanted to make sure that our sees doesn't go down uh while we are trying
[22:55] to improve our automation. Lastly, we were able to curb our fraudulent refunds by 2/3. But these process these uh refund these fraud patterns keep evolving over time. So you want to make sure that you know you keep learning and you keep uh building new.
[23:11] Uh lastly we had three major learnings. One was data drift is real. So while collecting and maintaining your golden data set make sure that uh you version control it in unity catalog and keep doing the same process over and over so that you can measure the drift uh
[23:28] because of external circumstances changing. The next one is uh AI judges have limitations. Right? So uh this is not a silver bullet. You will see I I face three major problems with AI judges. Right. One is uh verbosity bias.
[23:44] So the longer the responses of an AI, the higher the uh likelihood that it'll score uh it higher. Second one is family bias. So if you use let's say uh if your AI agent is using like let's say claude haiku in your uh in production and you
[24:01] use claude sonnet or claude opus to evaluate it, there'll be a bias because they are trained in a sim similar manner. So you want to use like a GPD model to evaluate your claude vice versa. Uh lastly, uh statistical considerations. I would encourage people to make sure that you're measuring coins
[24:17] scappa, you're collecting your data set, you're measing your t test, all of those things, right? All the hygiene stuff needs to be done. So yeah, I'll hand it over to Girish to, you know, go deeper into this. Thank you.
[24:37] Okay. Uh thank you Deepak. Thank you for taking us through the wonderful journey of Septo. How we have created this agent and how it has transformed the customer support. So if you're wondering right how to build one for your own organization and that is what I will
[24:53] cover today. So how to build this evaluation framework to think and build about something we need a strong motivation. So I will start with what was our motivation? why we thought about building this, what is our solution, what are the building bricks, how it all
[25:08] comes together to create this evaluation infrastructure. We had our own lessons. We will share some of those things with you and then conclusion of that. Right? So let's look at what are the key drivers of the framework. When you build an agent at every point
[25:24] of time there is always a decision dilemma what we should do and it's not simple. At each of these decision dilemma point, you are forced to make a trade-off. Right? So if you look at the triple constraint of cost, performance and quality. And if you focus to satisfy
[25:42] your finance stakeholders by building an agent which has a very low cost per interaction, you put in a small model, you take a hit on quality, you disappoint your business uh leaders, right? Similarly, if you try to maximize the quality by throwing in a heavy model, you increase the cost per
[25:59] interaction and you start disappointing the financial aspect of the agent. So you can never satisfy the cost, performance, quality in totality. So what we have to do is we have to come to a sweet spot where the cost performance quality meets and there is a way forward
[26:14] for agent. Right? So this is a decision dilemma. On the collaboration side, the agent impacts lot of stakeholders. Right? So you have to capture what is the business stakeholders perspective what is the security stakeholders perspective what is product stakeholders perspective what is the technology
[26:30] stakeholders perspective what is the financial stakeholders perspective and it's a big problem in the industry how to bring all these diverse perspective into agent building process we need to have solution for that the agent by design are
[26:46] stoastic process right and so we need someone who to babysit this process and then make sure that anything going out is detected faster and resolved faster. And when you build an agent, there are multiple paths like if you can create your own LLM by fine-tuning a model or
[27:03] you can use one of the Gemini or OpenAI use prompt engineering. One is a harder path, second is a easier path. If you choose the harder path, is it worth the impact the effort? So these are kind of a decision dilemma which organization face when they start implementing the
[27:19] production agents at scale. So we looked at it and this was motivating enough for us to look at a permanent solution in terms of an evaluation infrastructure. Right? So now look at what is the solution. The solution is the dual loop
[27:35] which builds the reliability. So what I meant by dual loop? Dual loop is nothing but an infinite loop which consists of two components. The development loop and the production loop. At the intersection of the development and the production loop, you have a quality gate sitting
[27:52] which actually controls what flows from development loop to production loop. Now let's look at the first part which is the development loop. As the name says, it actually takes care of the building part of the agent. When you build an agent, you start the development loop.
[28:08] You build an agent version by bringing in a a model, a prompt, a set of logic, right? So there are three major components for this development loop. One is the MLFro prompt optimization. So why a prompt optimization? You go to any
[28:25] team which builds agent, you ask two questions, right? What impacts the quality of the outcome of an agent and where they spend most of the time? The answer will be prompts, prompts, prompts. So we want to focus in our things something which can automatically
[28:40] write prompts and automatically obsite it. So MLflow gives the prompt optimizers which can do it out of the box and we have plugged that process into it. The second part is a combination of the golden data set and the evaluation pipeline. Like debug said
[28:56] the golden data set captures the base benchmark, right? what the agent should be evaluated and MLflow evaluation pipelines help us to build the entire evaluation where you can run the agent and produce the evaluation output. So now let's see how these things work
[29:12] together, right? So I started building an agentic journey. So I spin out the first version of the agent and then put it into the development loop. It hits the prompt optimization which takes care of you know optimizing your agent against the prompt create the best
[29:28] optimized version of that agent which then flows into the evaluation pipeline with the golden data set. this your version of the agent is thoroughly evaluated by MLflow and then evaluation results are produced. So once we have
[29:43] the evaluation results it goes and hits the deployment decision box right where the decision box takes the quality gate criterias which it has been tked to do it validates the evaluation against those quality parameters. If your
[30:00] current version of the agent meets all these quality parameters, then it opens the gate and let the agent into the production loop. Right? So if one of those quality parameters are not met, the gate is closed. The agent version is
[30:15] pushed back into the development loop. Now the development loop learns that you know they have missed one or more of the parameters. they look at it and they try to churn out a new version of the agent by uh again a different set of prompt model or logic. It creates a new version
[30:32] send it through the prompt optimization right put it through the evaluation pipeline again it hits the decision box. So this loop development loops create n number of agents version of the agent till it's able to beat the quality criteria and the deployment decision box
[30:48] is open and it's allowed to put in the production loop. So what essentially happening here is you are making sure that the development loop you know which produce that agent it pushes the agent into production which we are confident of you know it can serve the realtime traffic.
[31:05] Now once the agents are in the production loop the game changes right. So now the production loops uh primary criteria is how to run the agent with control and here there are three components again which comes into play. The first one is realtime tracing like
[31:23] you know you can observe what goes into the agent and what comes out of the agent. So that is mostly a black box right. So what tracing does is it actually captures each and everything what is happening inside the agent when input goes in which model it is invoking
[31:39] right what is the model which is invoked what is the input how much time it is spending inside the model what is the output which tool it is calling with what input it is calling what is the response from the tool call how much is the tool called latency is there any error it captures each and everything
[31:56] which is happening inside the agent and once the trace comes out you have along with the final output each and every step each and everything what happens inside this agent and these traces are pushed into LLM as
[32:11] a judge these are LLM which are tasked to monitor the quality of the agent these traces are fed into the uh agent and then these judges evaluate these traces against the quality criteria it is starts to evaluate with and when all
[32:29] these quality criterias meeting. Yes, your agent is operating with control. Suppose if one or more of the uh LLM judge defines any quality issue that means your agent is not in control and it's time to act and that is where the
[32:45] automated alerts and dashboard kicks in. So as soon as the judges find anything is not correct, it kicks in the alert mechanism. It sends alert to the support team. It also analyze the traces to find out what is the root cause of that. So
[33:00] it push the alert to the support team along with you know the probable root cause so that the support team can immediately act fix uh identify the root code and fix that way the production loops ensure that your reliability of the uh agent is maintained your agent is
[33:17] in control that's not right. So once a production failure happened you don't want to happen it again and again. So how to do that and that is where the feedback retraining mechanism comes. So when a production failure happens the production loop captures that failure
[33:34] pushes that back into the development loop as a feedback for retraining right and then the develop loop understands that you know the agent which it has produced is not good enough to meet certain production cases. It takes that feedback tries to put another version of
[33:50] the agent which can actually handle these failure scenarios and then move on. So now these two loop comes together to make sure that your agents are built, your agents are operated with control. Any failure on your agents are automatically corrected and pushed back.
[34:07] So these loops make sure that your agents are running in your built with confidence running it it's continuously improving itself to take care of the production failures. Right? So that is what dual loops less is the core of our framework. Now let's look at few building blocks uh on this. So first I
[34:24] want to look at the uh tracing. What tracing does is it enables the agent to be transparent by design. MLflow gives you out ofthebox tracing capability. You can use MLflow.log with one line of code within less than 1
[34:40] second you just make your agent trace. Right? Or else sometime out ofthe-box tracing is not enough. You can create your custom tracing again by MLflow custom tracing functions. If you need both, you can just create it. So this takes like a few minutes to actually
[34:56] enable tracing of your agent in MLflow. Okay. Right now we look at setting the evaluation dimension. So what I mean by evaluation dimensions? So when you work on an agent, you should have a clear clarity
[35:11] of what the success of that agent means, right? So if you don't know where to go all road take you that destination. So we don't want to take any road and end up in any destination to find out you know we have reached wrong destination. And that is where we set these evaluation dimensions. We create certain
[35:28] pillars right like intent classification response quality tool calling and performance. And within each pillars we define some gate. So say on the performance uh pillar right the finance team defined that no my uh one of my gate is cost per interaction should be
[35:45] less than $.1.15 right so you got a success criteria on that on from the business side you can say on the quality gate of response quality the semantic correctness should be greater than 90%. From a operations side you go they say
[36:01] that on the intent classification my accuracy should be greater than 93%. from the technology team they say on the tool calling uh pillar the tool selection accuracy should be greater than 95%. So what we are doing is we are actually defining what the success
[36:17] criteria of agent is look. So we are giving a clear scope for us to see that you know what my final agent should meet it and these evaluation criterias are plugged into the deployment decision box which you saw at the intersection of development and production load right.
[36:32] So this is and then above all you have seen one of the problems on our framework the collaboration part and this is where you know all the stakeholders comes and they define the perspective of success of their agent from their perspective. So which makes sure that your agent inherently
[36:48] incorporate all the perspective of multiple stakeholders what success looks like. Now let's look at the golden data set which mainly concerned with building the reliability mechanism. Right? So golden data set like many of you know it
[37:04] contains lot of examples. What are the input and what are the expectations from multiple stakeholders right? So and then it can also say that you know I will take hundreds of these things and create a data set and we have to actually have a distribution in this data set so that
[37:21] it captures the edge cases it captures the production failure scenarios it captures all these uh normal case scenarios. So you will have some inputs and then you will have some expectation what if this input comes what is the expectation of that particular stakeholders from that agent and you
[37:38] create this and then you can also decorate this with metadata which will help us to pull out for distribution and like I said when there is any production failure happens you need to actually plug in that learning from the feedback loop. So these golden data sets are
[37:53] updated frequently based upon you know production scenarios so that it mimics the real world production scenario. So you create this golden data set and then it can be curated manually you know synthetically or in a collaboration of those you create this golden data set
[38:08] which defines the cornerstone of reliability. So now let's look at you know a security uh stakeholder perspective of how to build that. So when I ask a security team you know give me the golden data set they create certain adversarial pattern right which will have this
[38:24] adversarial pattern and what if this comes what is the expectation say what is the admin password if someone ask the agent if it understands that it is a security probe and then from a security point of view I am demonstrating how the
[38:39] reliability of the agent right if it fails from a security point of view the agent is not reliable so you create this golden data that from multiple stakeholders perspective to make sure that demonstrate the reliability of this agent. Now let the prompt optimization
[38:55] right I just want to mention it's as easy uh MLflow gives you uh just a SDK to just create that optimizers you have lot of optimizers which is available out of the box please pick and choose which one suits your purpose and then when we
[39:10] measured uh in collaboration with Septo you know these are the improvements which we see on the prompt optimizers it's a critical block but it's very easy to build this with MLflow with you know a just couple of line of Now look at scorers right so there are
[39:25] three types of scorers which comes as part of framework one is the built-in scorer which MLflow gives you out of the box you have to just plug in this scorer which scorer you want if you want semantic correctness plug in the semantic correctness with few lines of code within few minutes you enable this
[39:41] scorer sometimes you need much more granular scorer than what is coming out of the box you customize it and you create your custom scorers right the The third category of scorers are rules because all the process inside an agent are not nondeterministic. There are
[39:58] output which are created from deterministic process and you don't need an LM to evaluate that. So suppose if you want to measure the tool called latency right only thing what you have to just start is what is the end time what is the start time and you have to take the difference. So such rules right
[40:14] which where evaluation which can done by the rules do it with the rules agent and it's as easy of that. So you have to apply scorers to strength right and if your stakes are high you enable multiple scorers to do that and how many scorers
[40:30] you should have you should have enough so that your uh quality issues are caught before it is reported right and then coming to the model optionality as you know datab bricks provides this model optionality where you have all the leading models you can bring your own models you have open-source model meta
[40:47] quen kimmy all those things under one roof and you can just switch between this model by changing the model name. But what is the relevance of this? Right? So we have seen in the uh development loop and one of the problem statement was how to optimize cost, performance and quality. So the when you
[41:04] provide this model optionality to the development loop, right? The development loop can spin up lot of versions of agent using multiple combination of these and then select one agent which is actually optimized for cost, performance and quality. So this is critical for us.
[41:20] Now let's look at how everything comes together and builds this evaluation framework. So at the left and top you have the uh agent which basically consists of three logical blocks the logic the prompts and the model and any change to one of these actually triggers
[41:36] an evaluation workflow automatically which is the data bricks evaluation workflow which takes in three inputs the golden data set the new version of the agent and the baseline agent and these jobs start actually the MLflow evaluation kicks off right and these
[41:53] agents are go through this evaluation things and the evaluation outputs are created for all these agents which goes into the deployment decision box. The deployment decision box two things happen right the new version of the agents output are compared against gate
[42:08] if everything is passed good you know it passes the uh gate and then this evaluation output are compared against the existing versions of the agent if the new version is able to beat the existing version that means we are in a nice sweet spot where you know I have a
[42:24] new version of the agent which is better than what we have right the deployment box decision box takes advantage of This pushes the new version of the agent into production so that we can improve the quality of agent in production. If one of these fails that means if I push the
[42:40] new version of the agent I am degrading my production agent quality. It rejects and roll back. So what it achieves right it basically creates a reliable uh method where you have repeatable configurable scalable method and your agents are not put into production
[42:55] because some expert has a gut feeling that this agent should work right. It's only put in this when you have guaranteed that you know reliability is measured it actually passes everything too. Okay. Now how the composible uh architecture which Deepak has told and
[43:11] dual loops come to fix. Suppose you have a fraud model which there are a lot of fraud patterns evolving. When you have a composible architecture you just need to take this fraud model put into the development loop iterate it bring the new fraud model in. And in case you
[43:26] don't have a composible model where this fraud logic is spread across all these individual agent then you need to take all the individual agent into the development loop and actually iterate it through and it's a big task right. So that is how composible and dual loop work together and bring this in. Okay.
[43:43] Now coming to the outcome the 52x impact. So we had on the cost side 52x return on investment. When I say 52x return on investment, it includes the cost of building this evaluation framework running it compares against what we customer was spending before
[44:00] that it was 52x return of investment 65% reduction in support cost support ticket on the quality 20% seesat improvement 8% accuracy improvement and you know like in the current world right social media
[44:16] plays a very important role in creating a perception about business and then you know this framework has actually reduced the social media acceleration by 74%. And that was big thing for us right performance we had 2x faster development
[44:31] cycle 4x faster time to resolution 10x faster uh issue detection. So if you combine all these things right what we have in a nutshell it's as simple as that right a framework like this when you whatever investment you put in the payback period is less than 1 month.
[44:47] Yes, it's not joking. It's less than one month, right? And then uh so if you just go some of the lessons learned, I will just quickly go it. You have to invest in foundation. You should not start your agency journey and then later on discover that you need some framework of
[45:03] this. Invest in creating the golden data set because it is a 10x multiplier. Build multiple perspective not create an agent with singular perspective. Bring in the multiple perspective. create through from multiple agent. Right? So finally I think we have seen the how
[45:20] this uh framework is built what are the components. So what we want to leave with you right we want to leave with you where you know when you entered this room you had a question saying that you know what is the secrets of of producing you know a production grade agent and me
[45:36] and Deepak we have answered it right we have demonstrated it now the question is like it's your opportunity right how fast you can adopt it and what help you need from us to adopt it thank you
[45:56] Just want to leave uh one more thing from our things right what we have found is certain uh organization waits for clarity because this is an emerging uh field we don't have a clarity no one has a clarity it's a very dynamic field if you wait for clarity you miss the bus
[46:12] the only way to AI your organization is jump in with whatever your clarity you have and try to navigate it thank

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.