BI for AI: Transforming OTel Agent Telemetry into Enterprise Analytics

Summary

Storing MLflow and OpenTelemetry agent traces in Unity Catalog as Delta tables transforms them from passive observability logs into enterprise data assets that can be joined with business data sources — such as Zendesk tickets and Salesforce records — to answer whether an AI agent is delivering measurable business value.
SQL-driven agent analytics on Databricks can reveal systemic tool failures, latency bottlenecks, infinite loops, cost per answer across model tiers, and model performance comparisons across thousands of agent runs — insights that span-level trace data makes possible at a scale no manual log review could achieve.
Production trace logs can feed directly into data-driven evaluation loops: joining trace tables with business outcomes identifies the exact failure cases worth targeting for LLM evaluation datasets and fine-tuning, turning surveillance into active learning.

BI for AI: Transforming OTel Agent Telemetry into Enterprise Analytics

You’ve instrumented your GenAI application, captured your MLflow traces using OpenTelemetry standards and stored them in Unity Catalog. Now what? Too often, observability stops at ingestion, treating traces like passive, isolated logs instead of valuable corporate data. this video looks past the initial setup to show the analytical power unlocked when agent telemetry lives inside a governed lakehouse. By merging your trace tables with existing enterprise data, you can move past micro-debugging and start running true business intelligence on your artificial intelligence. Key takeaways: Cross-domain data ingestion (join MLflow trace tables with Zendesk/Salesforce data to correlate agent performance with ticket volume/CSAT) SQL-driven agent analytics (isolate systemic tool failures, latency bottlenecks, infinite loops across thousands of runs) Data-driven evaluation loops (harvest production logs for targeted LLM eval and fine-tuning)
Talk By: Oleksandra Bovkun, Sr. Developer Advocate, Databricks ;

Chapters

FAQs

Why is collecting agent traces not enough on its own?

Collecting traces produces millions of lines of telemetry, but if the only use is spotting individual bugs, those traces are doing surveillance — capturing what happened — rather than organizational learning about whether AI is working. This video argues that the more important question is whether an agent is actually delivering business value, and that answering it requires moving beyond per-trace debugging to cross-domain analytics that join trace data with business outcomes. Storing traces in Unity Catalog as Delta tables on the Databricks Data and AI platform makes this cross-domain analysis possible with standard SQL.

How do you join agent traces with Zendesk or Salesforce data in Databricks?

This video demonstrates joining MLflow trace tables — stored in Unity Catalog as Delta tables — with Zendesk support ticket records and Salesforce data using standard SQL on the Databricks Data and AI platform. The join key is typically a session or conversation identifier that exists in both the trace record and the downstream business system. This allows analysts to correlate agent failures with ticket volume spikes, measure whether high-latency agent responses are associated with customer churn signals, or attribute cost per resolved ticket across different model configurations.

What kinds of analytics can I run on agent trace data in Databricks?

Once trace data is in Unity Catalog, this video demonstrates several analytical approaches: cost per answer broken down by model tier to understand the economics of different model configurations, latency analysis across span types to find bottlenecks in the agent pipeline, detection of systemic tool failure patterns across thousands of runs, and identification of infinite loops where the agent cycles without making progress. These SQL-driven analyses operate at a scale that makes it possible to discover systemic issues that would never surface from reviewing individual traces manually.

How can production traces be used to improve agent quality over time?

Production traces can be mined to build evaluation datasets and fine-tuning examples by identifying the specific cases where agents failed, were slow, or produced unexpected outputs — giving teams a data-driven basis for knowing what to improve rather than guessing. This video describes using an LLM judge against production trace logs to score response quality at scale, then harvesting the lowest-scoring examples as targeted evaluation cases for the next iteration. This closes the loop from observability to model improvement and turns the surveillance that trace collection provides into an active learning asset.

Full transcript

[00:09] Hello. Good afternoon. I hope you hear me. If you don't hear, put a headset on. So, my name is Alexandra, and today I'm going to talk with you about BI for AI. Essentially, what we're going to talk about is how you can
[00:23] extract more from a normal traces that you collect for your AI agent. Before we start, a little bit of uh housekeeping. You will see that slide many, many times. So, don't don't take a picture of it. It's just like a default
[00:35] one. And please, please, please submit survey after that. It will help me. It will help Data and AI Summit to be better all the time.
[00:47] Cool. So, let's start with the following. A little bit like a survey. Um how many of you are actually using MLflow or any other tool to trace uh to to collect traces and um information
[01:01] about your agents? Okay, quite some. How many of you are building agents to start with? Okay, that's good. So, um the question that was really bothering me is that um
[01:14] we do collect a lot of traces about every single agent. So, ultimately, if you're using Databricks or if you're using MLflow open source, you can collect those traces almost out of the box. But, what you actually doing with this
[01:26] information? You collect these millions of lines of traces. And what do you do? So, to me, the most important question that I would like to answer is if my agent
[01:39] does my agent actually brings business value. And what I figure out to answer this question is not that easy out of the box. So, what I'm going to walk you through today is how um I try to figure
[01:54] out what you can do, how you can approach collecting traces, and what else you can do with it. Please don't take this talk as a like a the only possible way of doing it. It's just like basically my exploration of
[02:07] how you can approach the problem. So, the the question is the following, does my agent actually brings business value? And this is what I would like to answer today together with you.
[02:20] So, one thing that I've noticed is that when we are collecting traces for a AI agent, we are actually doing surveillance, not actual learning. So, you do collect traces, you put a lot of
[02:32] you you capture like a millions of lines. Potentially, if you see some kind of a bug, you try to understand where this bug is coming from, you find that exact trace that tells you more information. You go go over it, and um
[02:46] yeah, basically and and you're done. That's it. You your traces are just sitting there doing nothing for the rest of the time. So, um I wanted to go a little bit deeper and try to understand what else
[03:00] I'm going to do with that. So, right now there is a very handy feature from from Databricks that you can store your MLflow traces for GenAI application in uh Unity Catalog as a
[03:12] Delta table. So, this feature just hit private public preview, so you can use it. There is a documentation I will share the link at the end. And essentially, what I try to do, I try to go beyond of just viewing my
[03:26] traces as I was doing previously, to storing them as a table, and then joining it with a business data that I already have in my lakehouse. And this is exactly what we going to
[03:40] look at. So, first of all, a little bit of a setup. So, technically, you have your application, you have your GenAI application, so your agent is running somewhere, you are using MLflow to your instrument, those applications. MLflow
[03:55] has uh autolog, which is essentially one-line statements that allows you to collect almost all interesting information about performance of your agent. Then, you store those traces as a Delta
[04:08] table in a OTL format. So, OTL OpenTelemetry is a de facto standard of how all possible traces are stored right now. It has two main advantages. First of all, it's very common format. So, um
[04:22] it's not a proprietary format of a one specific organization. It's a common format open source shared
[04:34] second advantage that it gives you, it gives you an ability to connect this data to a lot of data coming from other systems. Imagine, for instance, you have the rest of your ecosystem instrumented um
[04:48] and that information like a traces metrics are coming to something like Grafana or like a DataDog. Because OTL OpenTelemetry is a shared format, you can easily connect or combine or send
[05:01] traces from MLflow to one of those systems like, for instance, Grafana. So, when I started my investigation, the first thing I've done, I tried to understand how I can actually figure out
[05:14] intent and group my traces by intent. If you would try to do it in any system which collects your logs, including MLflow, as of now, it would be quite challenging.
[05:26] If you have this data as a Delta table, you can apply AI function, which is like, for instance, in this particular example, I'm using AI classify, to classify all the answers that are all basically all the questions that I got
[05:39] from my agent and understand to which group they belong. So, I was focusing on a essentially I figured out that in my traces, in my particular example, I had four categories. And the ones that was
[05:52] interesting for me the most was essentially the category of other. So, essentially the questions that I thought my agent is not intended to to answer. So, in this particular example, what the
[06:04] agent was doing is basically like a chatbot that helps you to answer like a product question. And what I figured out that a lot of users of this agent are actually asking something like what is the 25 * 17 or how do you say hi
[06:18] in French? Or something like that. If you don't have that aggregated view of all your traces that you can easily query, it's really hard to answer this type of question. So, that was my my first initial SQL
[06:32] query. Obviously, here is just like a main function. It's not like the whole query statement, but just to give you an idea of how you can use it very easily. So, I took a step further and I tried to
[06:44] understand how much the answer of particular question with a certain intent cost. So, my agent in this particular example is doing like answers about like a product, so
[06:57] basically like support agent. So, what I wanted to achieve and in this particular example, it's kind of matches to what I want to have is that recommendations
[07:09] are much more expensive in terms of cost comparing to for instance product questions. This information can be used to for instance to steer a development of of my agent. Maybe I want to kind of have more
[07:23] questions or more information or I need to improve the documentation that I supply to this agent to basically make it like make cheaper or make it better.
[07:35] After that, I went even one step further and I tried to understand for each intent that I have, how much actual calls to you at tools my agent does?
[07:48] So, for different intent, like for instance, for real product queries, it does two, three calls, while for instance, for a off pattern queries, it does one call. Which is kind of okay, because you do want you
[08:01] You kind of can imagine that for instance, for um um more specific questions about the product, it can call like a different tools to answer it. But if you are like for instance, in your particular example, you have more
[08:13] intents or more like deviations, you can see very interesting patterns. But, that was my first step of my investigation. And despite me doing all
[08:25] this analytics, I still couldn't answer question if my agent is bringing me value. So, I have it running, it cost me money, it cost me like support and I don't know, my time, but does it
[08:38] actually solve the problem? So, for that, you need to take one step further. And essentially, what you need to do, you need to be able to combine this data with the rest of the data in your
[08:51] lakehouse. In this particular example, on the left side, you see essentially the function just like a function to call to to ask questions to my agent. Instead of just storing a query in my
[09:05] logs, I also added a customer ID that I probably have from my uh from from my application. Because of that, it allows me to combine that customer ID
[09:18] to um like extract it when these traces are stored in um Delta table. What it allows me to do next, it allows me to combine this data
[09:30] for um for instance, um understand how different customers, different types of customers behave and interact with my agent. So, in my particular example, again,
[09:42] it's a customer support agent. I can group my customers by, for instance, enterprise, digital natives, and startups. And I can see, for instance, that enterprises, in my particular example,
[09:54] obviously not universal to every agent on the world, but in my particular example, enterprises are asking more complex question, and the cost of those questions are higher. Comparing, for
[10:06] instance, to startups, which ask in more off-pattern question. What this information gives me, it allows me to steer a development of my agent in certain area, right? Like maybe tomorrow I want to have two separate
[10:19] agents. One agent focusing on enterprises, one agent focusing on on digital natives, or for instance, startups. Okay. I thought it's kind of great, it's kind of cool. So, what um what else what
[10:33] kind of other information I can extract? Because costs and other parameters are natively stored in a natively collected when you instrument your trace your
[10:45] application with ML flow, cost information is already associated with every single answer that I have that my agent gives. So, basically, every single trace already has information about cost.
[10:58] Which allows me to say, like for instance, okay, digital natives in this example cost less than, for instance, enterprises. Maybe if I'm developing this agent and
[11:10] putting it in production, it have it can have different pricing model depends on who is my customer. Again, not saying that like it's applicable to every single agent, but I would like to trigger you to think how you can use it in your particular
[11:22] example. So, even further, and this is basically the question that I wanted to answer all along. So, like the question that actually bothers me if
[11:35] is my agent bringing business value? In this particular example, does my customer customer support agent actually helps me to reduce the number of tickets
[11:47] that's um the customers create in my system. So, what I did, I essentially classified as previously showed to you, I classified my
[11:59] um traces by intent using AI function. I hope you all know what AI functions do. If you don't, quick reminder, so they allow you to write a SQL function to
[12:12] apply to certain certain model or certain LLM to classify Oh, sorry, to um to apply that LLM to information in your table. So, one of them is AI classify. There is
[12:24] also uh um AI extract document. So, it extracts document, so it extracts text from a document, and so on. So, first of all, I applied it to figure
[12:36] out what is my intent for a particular query or for particular question. And then, I joined in a let's say, I have an information extracted from, for instance, uh ticket and system, like uh ServiceNow or like a
[12:50] Zendesk or something like that. So, I joined it together. And in this particular example, I can see that um questions that agent handles, so the questions that agent can answer, actually don't
[13:04] lead to customer support ticket. While the questions that land in the other group, actually lead to customer support um escalation or customer support ticket.
[13:16] What I wanted to show you in this particular example is not necessarily how to Like, you don't have to copy-paste those queries and apply them tomorrow. What I wanted to show you that having this data in one place allow you
[13:30] to ask very interesting questions to your traces. And if you're storing your traces from MLflow in Unity Catalog, you have all means by default to actually start
[13:43] asking those questions. So, thank you for going through this journey, but if you like if you followed along, you can see now that I can actually answer a question of um
[13:56] basically um what is the values that my agent brings. So, essentially, to summarize what I've done, I had my traces stored in uh um in uh MLflow
[14:09] uh so collected by MLflow, stored in a Delta table. I also used a cost of LLM to understand how much certain questions cost. I also used the customers to to to segment my
[14:23] answers. And I also used information about in my particular example about support tickets to be able to see what is a what is the influence? What is the value of my agent?
[14:35] Going through that journey, you can actually say that, "Okay, this particular agent makes sense to run it in production. Maybe other agent doesn't doesn't really bring value, just sit there and it's like a
[14:47] cool gimmick that you have in your system." So, going through this journey, I can now totally answer a question if it brings business value. Yes, but not only I can
[15:00] answer that question, I can actually prove it with data that I have that it does bring value and what that value is. So, I can even quantify it if I want to to understand like what it what it is if
[15:12] for instance I need to I don't know, invest more in development of that agent. So, um because it's a the talk, I'm not going into too much details, but I wanted to leave you with a few
[15:25] action points because, hey, you need to do something as well, not only me preparing all this talk. So, uh give it a try. First of all, uh scan QR code, you will go to documentation.
[15:37] So, uh you can enable traces in Unity Catalog and uh store them in uh so, enable traces to store them in uh Unity Catalog as complementary to storing them in
[15:50] MLflow. You can use AI classify or any other function. You don't have to You are not bounded only to AI classify. You can actually build your own uh logic how you want to understand intent from a
[16:02] particular query. Find your other bucket in your particular agent and see what is your agent doing outside of what you imagine it was doing or what you thought it was
[16:15] doing. And probably the most powerful one, if you already have your data in the lakehouse, so you can combine um information from your traces together with the data in your lakehouse so that
[16:29] you can actually uh start understanding more about your particular agent. I'm Alexandra. I'm senior developer advocate at Databricks. Feel free to connect with me on LinkedIn
[16:42] if you want to discuss this topic or any other data-related topics. I also created a sample repository on GitHub to to showcase this example. So, if you would like to know more, please let me
[16:55] know. I will share that repository with you as well. That's it. I will stay here if you have any questions. Since it's a lightning talk, I didn't want to take too much of your time. I hope you learned something interesting,
[17:08] and I hope at least I could spark a little bit of interest in what else you can do with those traces. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.