Taming Rogue AI Agents with Observability-Driven Evaluation

Summary

• Metric-driven agent reliability: A repeatable framework detects and diagnoses agent drift by instrumenting tool selection, error rates, and action progression with fine-grained observability signals.
• Stock-trading system example: This video demonstrates fixing brittle retrieval and faulty business logic in a stock-trading agent through prompt refactoring and tracing verification.
• Evaluation layer integration: Sending telemetry to an evaluation layer enables a virtuous cycle of continuous improvement through metric-validated optimization.

Taming Rogue AI Agents with Observability-Driven Evaluation

Talk by Atindriyo Sanyal, Co-founder, Galileo (Sponsor Speaker)
LLM agents often drift into failure when prompts, retrieval, external data, and policies interact in unpredictable ways. This technical session introduces a repeatable, metric-driven framework for detecting, diagnosing, and correcting these undesirable behaviors in agentic systems at production scale. We demonstrate how to instrument the agent loop with fine-grained signals—tool-selection quality, error rates, action progression, latency, and domain-specific metrics—and send them into an evaluation layer (e.g. Galileo). This telemetry enables a virtuous cycle of system improvement. We present a practical example of a stock-trading system and show how brittle retrieval and faulty business logic cause undesirable behavior. We refactor prompts, adjust the retrieval pipeline—verifying recovery through improved metrics. Attendees will learn how to: add observability with minimal code change, pinpoint root causes via tracing, and drive continuous, metric-validated improvement.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why do LLM agents drift into failure?

Agent drift occurs when prompts, retrieval systems, external data, and policies interact in unpredictable ways, causing the agent to behave undesirably—this is addressed through continuous observability and metric monitoring.

What metrics are essential for agent observability?

Key metrics include tool-selection quality, error rates, action progression, latency, and domain-specific metrics that signal when an agent is deviating from desired behavior.

How does root-cause analysis work in agentic systems?

Fine-grained tracing instruments the agent loop to identify whether failures stem from prompt issues, retrieval problems, business logic errors, or external data quality—enabling targeted fixes.

Can evaluation frameworks improve production agents?

Yes, by instrumenting agents with observability signals and sending telemetry to an evaluation layer, teams can establish metric-validated improvement cycles that continuously enhance agent performance.

Full transcript

All right, I think uh I'll just get started. Uh so firstly, thanks for joining us. Uh hey uh so we are part of uh Galileo and I have my uh teammate with me Roi here and if you've not heard of Galileo we are uh an a genai evaluations and observability company and we've been around for about four and a half years squarely in language modeling and been dealing with LLMs and LLM based systems for pretty much since day zero of our existence.
Uh over the years we we are uh we've grown quite a bit. We're series B at this point and uh we've had the privilege of working with a lot of um uh large some of the largest banks in the US uh some of the largest telos in the world and they're all using us to essentially ship reliable Genai applications and really build a system of robust um evaluations and observability for kind of their Genai ecosystem because it's not about just one app or um you know just a single thing that you build.
It's more like how do you set up yourself for scale uh and a lot of it is about preventing flying blind uh answering questions like hey if it did something wrong can I root cause it and can I fix it so that it doesn't make the same mistake. Uh so really it's a it's a platform that allows you to bring in that level of diligence and uh responsibility in your workflows as you're building right from day zero as you start building all the way to you have an application in production and how do you guard rail it how do you safeguard it to prevent bad outcomes.
Um, so this talk was originally titled um it was something else. It was um taming rogue agents with eval driven observability and it was a bit of a mouthful. So I thought about what I really wanted to leave you guys with and uh it was really about a new reliability playbook kind of a a playbook which takes the best practices from the previous era and bakes it into the new world of agents and orchestration systems and uh LLMs really like how do you build a robust system of eval uh in the new era.
So with that um I I want to start with kind of an ironic uh uh graph and uh it basically states that as the couple of years have gone by since chat GPD happened the intelligence of LMS has gone up but if you talk to any leader in the enterprise and I speak to leaders every other day uh just the level of confidence in putting something out there is either stagnant and in most cases has come And it's a bit ironic given, you know, models can do a lot more than they could 2 years ago and at a fraction of the price.
But there's a reason for this. And the reason is that we've kind of gone from, you know, basic elementary systems and workflows to a lot more complicated multi-agentic systems which are interacting with each other. So, we figured out a formula to build more sophisticated things with LLMs that are not just question and answering systems, but they can actually do things. But the question still remains, how do you evaluate them?
And how do you know whether they're doing the right thing and can you quantify it, automate it, and do this at scale? In fact, even the rag system that we all know, the the simple architecture that you see at the top with the vector store and the prompt, uh that has evolved into something a lot more sophisticated. In fact, uh we work with customers who are very large uh companies across financial services and consumer goods and telco who are building internal rag systems which have memory and context management. they have a citation platform which is uh often using data lookups to you know various systems uh you using tool calls and using APIs.
So this is the new rag of 2025. Uh and that this is just to say that um a lot of the newer systems we're building especially with agents in the mix are a lot more nuanced and have a lot more moving parts which is why the confidence to release them out in the world without any harness and evaluations is it that's the question in leaders minds that you know hey I run a large company and yeah I want to do something in Genai.
I'm sure a lot of you guys also are strategizing your you know genai strategy right now. Uh but the question remains yeah what if something goes wrong and it can harm the business it can harm the brand and uh the probability of that happening is a lot more despite all the advancements in the last couple of years. Uh before I kind of go into the meat of the presentation, I wanted to take a step back and kind of just align on what an agent is.
So just grounding us on the same definition of agent. Agents really simply put are essentially a system that can plan and take actions based on data and instructions. If you simplify it down, that's essentially what they're doing. They can be constructed in simple or complex ways, but at the end of it, you're essentially supplying the right information to a set of functions or tools to take an action. There's different constituents of an agent, but primarily there's three big parts to an agent.
Uh first is kind of the reasoning or the brain of an agent, which really takes it's the difference between software 1. 0 and 2. 0 in that software now can think. So that's the reasoning of the brain of the agent. Then there's memory. Uh you can build an entire system of memory management which submits the right context to the reasoning bit so that you know the next interaction has more you know uh uh data available from past interactions.
And finally there's the tools which are these you can think of tools as item potent functions which do very deterministic things but the quality of the execution can vary drastically if you submit the right or the wrong information. Uh so these are the constituents of an agent. A lot can go wrong with agentic systems. I'll give you an example of one undesirable agentic behavior. And this is something that we've seen a with some of our experience with uh 100 companies.
Uh an agentic bot asks how are you how can I help you? The user says do you have corporate cards? And it asks you about like good interest rates. The agent will gather that and ask it for more information. The user supplies the information and it goes ahead and does something completely different. And this might be shocking here on a slide, but you'll be surprised to know how often this happens. And these are kind of the issues.
These are the reliability problems of agents. Um there's three reasons why these kind of things happen. I put them as like three obstacles to reliable agents. first is just the inherent unpredictability in these language models, right? You're that's kind of a a restriction that you're dealing with. Um that can lead to, you know, tooling mistakes, memory errors, incorrect tool selection, but that's basically because of the inherent uh stochasticness of LLMs.
The second impediment or obstacle is prompt sensitivity. So if you talk to anyone who's built a fairly sophisticated agent, they would tell you that a single word or a single phrase, one sentence can make night and day difference between the output, you know, good and bad output of the agent. Uh that's where the importance of good prompt engineering comes in where you want to supply the very specific instructions but it's almost an art uh on how to select the right set of instructions and it it depends on the context it depends on the request.
That's why prompt engineering kind of becomes an art when you're building a good agent. And the third is the whole garbage in garbage out problem. Right? If you're building as an example an agentic rag system, the quality of your retrieval will basically affect everything right the kind of data you retrieved. Uh I kind of call it the compounding effect of tiny errors where uh a small token here and there in your prompt or uh a document whether it's present in your retrieval or not will make all the difference between yeah actually executing the right thing and going totally rogue.
So the question becomes how do you tame these agents given you know you you know these impediments at Galileo through a lot of our experience with many of these companies uh not just financial companies or telos uh we've even worked with companies of massive scale like Airbnb and Reddit uh who use Galileo to safeguard their content systems the question was how do we tame rogue agents uh we came up with the principles of what we call integrated observability.
And this is a bit of a mouthful. There's a lot going on here, but I'm going to go step by step, which is the meat of this presentation on how do you take this eightstep playbook of integrated observability and bake it into your Genai workflows and together is what will get you a Genai system that you can trust. A lot of people come and ask me, oh, what is that magic metric that will solve all my problems? Or, you know, how how do you detect hallucinations?
It's less about a single metric or, you know, some kind of magic pill that will solve everything for you. It's really a system of integrated observability that you need to build. And what we realized is that it is a continuous system. It never ends. Uh that's because your application will always meet new data and there'll be issues like drift or new clusters or new topics of data that your application may or may not perform well on.
But this playbook will help you not only uh detect it beforehand or pre you know pre uh but also prevent it uh from happening uh and in case it happens it lets you root cause effectively and prevent it from happening in the future. So with that I'll get into the eightstep playbook for agentic reliability. Step one is creating the right evals uh for your agents or the right metrics for your agents. Now I look at the play one and it kind of reminds me of test-driven development if you've heard of it in the traditional world where you need to think about your failure cases first and uh kind of design your testing ecosystem around them and then you start building.
Similar to that here the first step is to really think about what you want to measure and the question is or the challenge is that what you want to measure differs from application to application. That's where what we do as a platform is we say there's three kind of general holy grails of measurement that you want to do for agentic quality and that includes you want to measure your tools error rate tool selection quality and action advancement whether your agent is actually moving towards the goal that you set out for.
But agentic architectures are custom architectures. If you've built agents before you can you know that they can be built in a million different ways. In fact, one of my favorite um blogs uh that I often quote developers is uh a blog that was written by Antropic over six months ago, building effective agents. And I like that blog because number one, it's very simple to understand, but also it touches upon this idea that there's about 12 to 15 design patterns around agents which should cover uh you know a vast majority of sort of actions that you want to take.
But the broader meaning behind that article is really that uh it's like a jigsaw puzzle like building an a agent is like a jigsaw puzzle and you can construct it in a million different ways and custom architectures need custom evals. So we've had many u customers who've used the Gallio platform to create all kinds of metrics for their agents and you want to apply different evals at different parts of your system. So that's what agentic sort of uh reliability or building agentic metrics is all about.
You you think of the kind of the three uh three things high level that you want to measure uh but then you know you kind of circumvent that or rather you augment that with uh with custom metrics that you want to think about before you even start building your application. Step two is once you have the eval kind of built out and set in your mind, you want to set up your experimental infrastructure. And I'll give you an example of what this means.
Uh it's a four-step process to fixing an issue that you detect in agents in production. Firstly, you spot it. Usually, it's a data issue. So, you want to isolate the segment of data that you find anomalous. Um once you isolate the anomalous data, you want to do an experiment. Now what the purpose of experiments usually is is to kind of isolate the data that was misbehaving or that caused the agent to misbehave and then you experiment with different prompts, different LLMs or different sets of documents which means if you're if you have a vector store or some kind of database where you're fetching data from, you want to tweak the parameters to see whether you're fetching the right data or not.
But setting up your infrastructure to be able to do that at speed kind of becomes important because what your eventual aim is is to once you detect an issue, you want to quickly know what to what to change to not make that issue happen again. Uh and it's kind of like, you know, I found a dirty sock and I want to change, you know, the machinery around it to get to a clean sock and I compare the clean and the dirty sock like here. and then I know what you know constitutes my clean sock and then I quickly reconfigure my app.
Play number three is kind of I think one of the most important plays. It's all about the prompts and this is primarily because a lot of agentic engineering is prompt engineering and uh uh a lot of like sophisticated developers will say that man like I've been tweaking strings for a living at this point. Uh that's where I think an organized system around prompts becomes super critical. Just two days ago at the data bricks summit uh someone introduced me to the concept of prompt folding which is essentially um sort of dynamically adjusting or minimizing certain parts of the prompt or folding it to adjust to the context length issues etc.
So, you know, being able to manage prompts dynamically, version control them because you're constantly editing them, tweaking them, giving you the lineage of the prompts and essentially giving you a way to build, test, and compare prompts and doing experiments on them becomes very critical. So, building that discipline and prompt engineering is one of the most critical plays in this agentic observability playbook. Number four is um kind of something that's pretty brass tax, which is any observability platform that you've used should offer you the ability to debug traces.
But what we've realized is each request is a trace. And if you get the ability to look at what happened at each step of the trace, that's necessary but not sufficient because uh there's three levels of debugging that you need to do. Firstly at the trace level then at the you can call it a session level or an interaction level which is kind of like a set of traces that is uh trying to accomplish a subtask and then at the application level which is the top level.
Uh so you want to set up analytics at all these three levels to be able to have that 360 view of the health of your application. Number five goes into the importance of kind of offline guard railing which is CI/CD and how do you bake in agentic tests and agentic evals into CI/CD. Uh and a lot of this is traditional uh you know software engineering discipline that we've all you know seen and worked on before. The only difference is now you're kind of testing uh and validating different components right uh you have prompts you have uh you know parts of your agentic app which are LLM driven so how do you mock those you have uh you know machine learning this is a machine learning system right so you have data sets sometimes you may have ground truth and may not have ground truth so how do you bring all these eval assets into your CI/CD I think that becomes super important uh but the overall purpose of baking in these tests in CI/CD is the same that your agentic app is kind of like any other app except that it's a smarter app and it goes through the same life cycle of being developed on local host put into your dev clusters and all the way to Canary and finally in prod and you probably have some kind of an automated system that's set up for your traditional software you need these components for your genai app as well and uh you want to set up testing for that and I'll I'll get into a little bit more detail on how you set up aic tests in one of the future plays.
Uh but before that I want to talk about the most important thing once you productionize applications now what and this is one area where Galileo truly shines. I think we've done a lot of innovation on how to do eval at scale. Um some of the models that we've built which are you know instruction tuned and done few short learning on on on some of these lighterw weight eval models they've we our internal experiments have benchmarked them against a lot of the industry standard um open-source tooling as well as cloud provided solutions.
We are able to um give you eval at very high scale. If you use LLM as judges um they work great in offline environments. Uh once you start hitting a certain level of throughput, you're at the behest of the LLM provider whether it's context limit issues or uh rate limits and uh I don't think I need to explain that. So uh but we have had the experience of seeing folks trying to productionize LMS judge metrics and it's a pain.
Uh so that's where if you have beyond a certain level of throughput, you need super accurate and fast evals that work at scale. And this is one area of research that we've put in a lot of resources on. And this is where we truly shine. we uh not only get you to production through our platform but in production we are able to um give you these these very highcale SLM small language model driven metrics and uh to be more specific we've we've done a lot of um uh experiments with the different flavors of uh the newer llama models uh kind of 2 to 8 billion parameter and we've achieved significant success in in being able to give you LMS judge level accuracies but at you know a fraction of the latency and cost.
The seventh play and we're we're almost there to the final one which is number eight but seventh one is super important. It ties to uh play number six uh play number five which was the CI/CD uh play. Here the question becomes how do you how do you create test sets for agents? Like how do you test agents right? Agents are essentially functions. you test them um kind of using what we call like agentic test sets. The question is how what is an agentic test set, right?
Uh if you look at traditional data sets, say you you're doing a simple call to open AAI or you you've fine-tuned a model, right? How do you test a model, you have a query, you have a ground truth, you have a model response, and then you measure some kind of a measurement between the ground truth and the response, and then you say that yes, it's a pass or a fail. It's very similar with agents except that the intermediate outputs of agentic systems are usually you know JSON serializable or some kind of an intermediate representation.
Uh so you want to test those but you don't want to in a in a test environment you don't want to do uh real LLM calls. Maybe you want to mock them. That's where uh we there's some specific constructs that we've baked in as part of our of the Galileo SDKs which allow you to author these agentic test sets and put in these little um sort of measure measurement classifiers uh which which you'd be used to in the traditional sort of testing world whether you you know for example is null is a construct uh there's a whole host of those which you can bake into tests and then now this data set is usable ible in your CI/CD workflow.
Uh, and we've had uh success taking a few of our customers uh CI/CD uh workflows and baking these kind of agentic tests in them. The last play is all about metrics and the principle of the final play is that there's no one metric which will work for you all the time. Even if you say that I know what my definition of hallucination is, you can author a metric, LMS judge or whatever you're using behind the scenes, the truth is that as you productionize your app, the the definition of your metric will evolve and you'll start yielding false positives.
So how do you safeguard and guardrail against that? We created this concept of evaluation agents which are essentially taking these metrics and these evaluating them to your data. Um this is kind of the workflow where you have a baseline metric you it yields a false positive and now you need to do something to improve it over time. Uh what we've seen is that there's a feedback framework that you need to build which is low touch human feedback. uh and using that feedback you make the metric not make that mistake later.
Uh and the the how do you achieve this is by firstly you want to centralize metric as an artifact in your platform like a metric is a reusable entity which is stored in a database and others can use it. So that you know if user one detects an anomaly or a false positive they submit an instruction to the metric telling it that oh this is the reason it was wrong when it's used in the future by another user the same metric shouldn't made make the mistake.
So all this machinery together is part of a metrics platform. But going deeper into the effect of different kinds of feedback. Uh this is a lot of this is kind of a research area that we've delved deep into with our AI researchers. There are three types of critical feedback that we explore and its effect on the accuracy of a metric over time. There's a binary preference signal which is basically a thumbs up thumbs down uh kind of feedback very low touch.
Uh there's the critique of explanation or COE. What that is is it's essentially a verbal feedback that is a critique of why a value was or the explanation behind a value was wrong. In Galileo, we augment each of our metrics or eval with an explanation of why the metric is a certain way and it uses a technique called chain pull which is uh a paper we wrote. It's an augmentation to LLMS as judge but it enables each metric to have this deep explanability.
Uh but our feedback system explored the effect of critiquing that explanation. And the third kind of feedback is critique them critiquing the value itself. Say you have a hallucination metric 0 to one you know if it's one that means it's it's you know good zero means whatever hallucinatory. uh if there's a false positive, if you critique the value itself, there's a different effect that it has on the metric over time.
And here's some of the results. I'll show you some very interesting results of feedback on different metrics. So, we took a baseline metric, which is the black bar you see, and we gave it different forms of feedback over time, and we realized that actually short feedback over time is actually better than long feedback. uh if if we can go deep into why and we explored you know potential reasons where the loss in the middle phenomenon happens with LLMs etc.
But the short of it is that if you detect a false positive you give multiple pieces of shorter feedback over time you actually get a better longerlasting metric and secondly um high quality descriptions of your feedback as well as of your metric. If you author a c custom metric even if it's LMS judge always give it a very high quality description because over here you can see the performance the black bar is a high quality metric using a phenomenon in Galileo called autogen which is the ability for you to create a metric on the platform.
So you create a high quality metric versus the same metric which is we took these two metrics instruction adherence and PII which are kind of the first one is a hallucination metric and the other is a um of course you know private sensitive data metric but we created two different versions of it and you can see the baseline performance is astronomically different but the more surprising thing is we took the lowquality feedback and gave it the three kind of critiques and look at how better it got over time.
So that's the magic of low touch human feedback and the reason this is the play eight is because you need this kind of a human feedback mechanism for your eval baked in to your uh integrated observability uh playbook. So with those eight I'll bring it back to the uh the the spinning wheel or whatever you want to call it like the infinite loop here. Um h so the question still remains how do you actually how does integrated observability actually accelerate your workflows and this is where I do want to show h how Galileo as a platform is completely about about this phenomenon and this is how on during the development stage we allow you to gather uh ingest millions of signals from your genai app analyze them give you deep analytics and then eventually fix them root root cause them efficiently.
But also in production, this is where we've done a ton of innovation uh using our suite of SLMs, which we call the Luna suite of models, and they're able to achieve step function better accuracy. Not only that, but it's able to achieve end-to-end latencies, which are pretty unheard of. And that enables you to guardrail your chat bots or guardrail any kind of genai app which is in production working at scale. And that's it.
And with that, we'll do a little demo of the Gallio platform and then we'll do questions after. Make it a little bit bigger. Okay. So, what I'm going to show you here is like a very very um uh low high level view of what the Galler framework looks like a little bit when we're using it with an agentic application. So, this is a small aentic application that I create created that does some very basic things. uh it sort of simulates what it would be like to have sort of a an an agent a a banker agent with a simulated bank account and basically it will allow you to move money between your accounts and go do some uh uh buying on the stock market.
So you could tell it, hey go um I want to buy a certain amount of stock of stocks in some sector and it will go ahead and do that job for you. And that specifically um is um done by first looking at your accounts, calculating some risk uh profile and then going to look based on your risk profile at the potential stocks that you want to uh buy, actually looking for those on the web, getting their ticker prices from Polygon.
So like going out to some API and then coming back with actual prices and executing that um action against a a sort of real bank account, right? Like so I have I can actually uh remove funds from that account and move them into uh my stocks. So what you'll see here is like several sessions that this that this particular user has had with uh with the system. Um first they checked for the balance, then they um uh said they wanted to buy some tech stocks. the system kind of allocated this this amount um even though they asked for to spend more they the system knew to allocate less um and actually did what it was supposed to which was to buy some stocks and then the user deposited more money um into their account and went through through this process again um and this time the system actually failed to do everything that it was supposed to right so there was a step in there where there was a behavior that sort Looks like the agent completed.
It does say that it it failed with paying a specific bill, but it's unclear why this happened, right? So, like I have this again, and this happens a lot when you're developing agentic applications, when you're developing complex applications um with a lot of LLM dependency. Things can go kind of haywire in that process and it's very hard to track again what exactly went wrong and how how did it go wrong, right? So, I'll flip into Galileo and show you my uh trace for this particular um session.
Um so, what you can see here is first I get like an overview of um a set of traces. I'm going to flip back and forth between them so you could get a a sense of what I'm talking about. Um and you'll see here that you know here I'm I'm I'm seeing each node of the activity that the that the agent took, right? So, for example, here it's doing the initial tool selection. So it's getting some input um asking for balance going through that and producing um a particular name of a tool that it needs to call in order to produce that balance and then it goes you know through the motion gets the gets the the information that the user needs and outputs it.
And in this case I can tell that my action advancement is 100% but for some reason my um uh uh uh agent here thinks that it's not exactly doing what it's supposed to. So that already immediately gives me an idea that hey maybe something didn't work quite right. Um as I move along I can see here that my first action worked as expected. So everything happened like the entire transfer happened as expected and then when I do it again I can see that something went wrong.
So like the system already identified that I had a goal, right? The user said something in this case that they wanted to to buy a certain amount of stocks and that there was some failure that didn't satisfy what my user wanted, right? And this is the an example of our um agent specific uh metrics which again look at the entire action that an agent is performing with all of its constitu constituent parts and then can assess whether or not things worked as you intended or not.
And as mentioned, right, like these could be like out of the box not exactly what I'm looking for. And I can on the fly kind of sort of introduce my own logic and say, "Hey, this metric should look more like this or more like that." Thus introducing my domain knowledge into the process itself. One last thing I want to show you here is my log insights. So our log insights um is is a relatively new feature and what basically it does is it looks at the entire performance of my application overall right so beyond each trace right it looks at the overall performance of the application and it highlights things to me that it thinks might be problematic so for example here it's telling me that I'm giving back inconsistent research perform inconsistent re research and it explains very deeply right like and I'll I make you read all of this, but it basically tells me, hey, you're giving for the same request that a user may ask.
Sometimes you're giving 25 stock recommendations and sometimes you're giving three stocks recommendations. What's up with that? Right? Is that what you intended to do or not? And some of these um metrics uh some sorry, some of these insights that we're getting are things that you wouldn't have caught, right, had you not tried many many many times to work with your own application. And even then you wouldn't land on all of the uh uh more niche use uh uh uh issues that might might crop up and you wouldn't even uh notice um in in addition to that right they can also gives me give me like some good indications right so like what you know like it understand that I have a multilm architecture it understands that I have a complex investment flow and sort of highlight again where it thinks that that complexity arises so that's another way for us as developers to kind of gain a more burst level view understanding of what exactly is happening with our applications as we develop them.
Um, that's basically it. I'll give one more example of an error that was detected in the logstream insight section with one of our customers where uh an an unauthorized transaction was executed, a financial transaction that was not as so that showed up as a red saying unauthorized transaction and it was able to point to the specific trace where the agent made that mistake. So a tool like this you can see that you know you spot the anomaly we give you the shortcuts high level and then you can dive super deep at the trace level to see specifically what happened and maybe one more thing to add here is that right here in this particular example you're seeing um just the uh agentic relevant metrics uh enabled of course you can think of all of the things that we've seen in the last year or two um kind of fold into this or anything that's retrieval related right like if you have a rag tool that is a just yet another tool in the chain, right?
All of these retrieval um uh spans are still interesting and you could understand again more about how your application um operates just by looking at those even beyond the agentic uh specific stuffs. And of course custommetrics um uh go into that as well. Totally. And what you saw is like probably one or two% of Galileo, there's a lot of other stuff baked in and you can use our SDKs and APIs to basically fetch all of these insights in your own app and guardrail them.
And people have used Galla in many innovative ways to safeguard their LLM systems. Uh so just wanted to call out that this is just to show you one or two highle workflows but uh happy to take questions. So, I have a QR code for you if you want to point your screen at it. It takes you to galileo. ai. Of course, you can just go there directly. Um, but my email is right here. Uh, we've been doing this for a while and we have a lot of battle scars of taking uh very serious Genai practitioners into production and we're also learning as we go.
But uh certainly uh have the wisdom of for whatever it's worth uh this is a new area. Uh but we've we've seen a few early applications especially on the agentic front being productionized. So we understand this this area really well. If this is interesting please email me or you know we can chat right after as well. Um always happy to chat with folks who are building cool genai stuff.

Learn more about the Databricks Data and AI platform.