Skip to main content

What Is Your Agent Doing? Making Sense of Agent Traces

Summary

  • Agent traces are the telemetry records — spans, token counts, tool calls, latency, and outcomes — generated by every AI agent action, and ingesting them into a Databricks warehouse transforms them from passive observability logs into queryable enterprise data for cost, ROI, and governance analysis.
  • Five stakeholders ask 'what is your agent doing?' with entirely different meanings: the CEO wants competitive ROI, the CFO wants cost attribution by team, engineering wants to know if shipped code is durable, product wants user experience signals, and sales wants to see account ramp-up or churn from agentic features.
  • dltHub provides tooling to ingest agent telemetry from OpenTelemetry platforms — including Datadog and Grafana — as well as from MLflow and TypeScript agents via the Vercel AI SDK, normalizing trace data into SQL-queryable tables on Databricks for unified multi-stakeholder governance.

What Is Your Agent Doing? Making Sense of Agent Traces

Watch: What Is Your Agent Doing? Making Sense of Agent Traces
Agents are sprawling; your product includes a chatbot, Cursor helps write code, experimentations are done within an LLM Observability platform, Gemini transcribes sales calls, and Claude Cowork automates admin tasks. The CFO wants to know how much agents cost, the CEO wants to know the ROI of your 50+ agents and engineering leadership wants to know how to improve them. This talk teaches how to monitor agents, ingest agent telemetry into your warehouse and transform it into analytics and unified governance on Databricks.
Talk By: Thierry Jean, Lead AI Engineer, dltHub ;

Chapters

FAQs

What are agent traces and why do they matter for enterprise AI governance?

Agent traces are structured telemetry records that capture every step an AI agent takes — tool calls, LLM invocations, token counts, latency, intermediate outputs, and final results — providing a complete audit trail of what the agent did and how much it cost. This video explains that traces are increasingly important because organizations deploying 50 or more agents face the same question from multiple stakeholders — CFOs want cost attribution, CEOs want ROI evidence, and engineering wants to know if agent-generated code is durable — and only structured trace data can answer these questions at scale. Without ingesting traces into a governed warehouse, observability stops at monitoring individual bugs and never becomes organizational intelligence.

How do I get agent traces from tools like Datadog or Grafana into Databricks?

dltHub provides connectors that pull agent telemetry from OpenTelemetry-compatible observability platforms — including Datadog and Grafana — and load it into Databricks warehouse tables in a normalized, SQL-queryable format. For agents built on Databricks, this video explains that a single MLflow flag enables trace export that writes directly to Unity Catalog. TypeScript agents using the Vercel AI SDK can also be normalized and ingested through the same pipeline, giving a unified trace store across heterogeneous agent architectures.

What can I analyze once agent traces are in a Databricks warehouse?

Once traces are in Databricks as SQL-queryable tables, this video demonstrates several analytical patterns: isolating token spend by team or use case to give CFOs cost attribution, identifying systemic tool failures or latency bottlenecks across thousands of agent runs, detecting infinite loops or unproductive code churn where agents write and then remove the same lines, and correlating agent feature usage with product engagement signals. The same trace tables can also be used to harvest production examples for targeted LLM evaluation and fine-tuning, closing the loop from observability to model improvement.

Why did Uber exhaust its 2026 cloud budget early and what does that mean for AI governance?

According to this video, Uber's cloud budget was consumed far ahead of schedule because the organization initially planned for developer-facing code completion tools but then rolled out autonomous agents across teams, causing token costs to explode beyond any forecast as new use cases proliferated. The CTO was quoted saying the budget they thought they would need was already blown away. This example illustrates why agent telemetry governance — tracking what agents are doing, how much they cost, and whether they are delivering outcomes — is now a critical enterprise capability rather than an optional observability add-on.

Full transcript

[00:10] All right, everyone. It's already 1 2:00 p.m. Sorry, a bit jet lag. Everyone's getting comfortable, headphones on. Let's go. So, the title of this presentation is what is your agent doing? Question that probably a lot of you are asking or
[00:23] getting asked. And we'll try to make sense of this using agent traces, a type of data that's becoming more and more important. I'm Siri. I'm lead AI engineer at DLT Hub, and I'll share a bit about what I do on this field.
[00:37] So, everyone ask, what is your agent doing? And you probably see a lot of headlines in Forbes, The Economist that covered, for example, Uber or other big tech companies where Uber exhausted its 2026 cloud budget by April.
[00:51] Why? If you read a bit more into their story, they Oh, great. Okay, it's back. So, yes, Uber was planning to roll out cloud more broadly. Initially, they had developers
[01:04] using, for example, code completion or some ID features, but when they rolled out agents, they seen a lot of new use case pop up across teams, even autonomous agents, and then token costs
[01:18] and expenses exploded. And we have a quote from the CTO saying, "I'm back to the drawing board because the budget I thought I would need is blown away already."
[01:33] One question, what is our agent doing? Five stakeholders, five teams, five different meanings behind that question. For the CEO, it probably means, are we getting considerable competitive advantage from this investment, and is
[01:45] it lasting? Is our competitor getting the same advantage for the same amount of money spent on tokens? For the CFO, it's probably dividing these costs by team, customer, and seeing where results and
[01:59] outcomes happen for these investments. Engineering has a totally different perspective on that question. It's asking, when we're shipping code with agents, is it lasting and durable or is it reverted? Because if you're spending
[02:11] tokens writing new lines of code and then spending tokens to remove them, there's no gains being made. And for product, it's probably about getting a good user experience. Are people happy about the outcomes or are
[02:23] they churning from these agent features? Because if you're spending money on tokens and it leads to a worse product experience, there's again questions to investigate. And for sales, it's a good way to see is your account
[02:35] ramping up or churning based on these agentic features. So, DLT Up, the company, is the cloud-native data engineering platform. It's also the maintainer of the very popular open source Python
[02:48] library called DLT, which is used for ingestion and ETL. It's one of the fastest way to move agent traces to Databricks, and I'll explain that in detail during the presentation. The thesis for us is you can't act on
[03:01] messy, multi-vendor agent traces, which is the default today, and this data is not only messy and hard to reason about, but it's also detached from your company data. It often lives outside the data warehouse, and in
[03:14] in that case it's not usable and it's restricted to only the developer and engineering use cases. So, how did we start investigating investigating this? Like I said, DLT is
[03:26] a popular open source library. It allows people to build custom connectors, and what you see on the graph is the number of of connectors that we see every month in our anonymized telemetry, and what
[03:38] exploded really is the number of custom connectors. Why? Because cloud code, OpenAI, GPT, and Codex model got very good at writing Python code, and so they started
[03:50] writing DLT pipelines. And in this telemetry, one thing that we noticed is a new category of pipelines. Pipelines to ingest agent traces. So, agents building pipelines to ingest agent
[04:02] traces. That's where we're at today. And what we discovered is that there's no single pipeline that you could build to ingest agent traces. You often have a plethora of them because this ecosystem is fragmented currently. And DLT Hub is
[04:16] trying to provide standardized solution to that problem. So, to ground this discussion, I'll start with three cases that we've seen from DLT Hub customers. When leaders are able to decide using agent traces joined
[04:30] with their company data in the data warehouse. The cases have been anonymized obviously, but you'll get the point in what we've been trying to achieve with our customers. So, the first one. We have leaderboard on or a dashboard on token
[04:44] spend. You've probably seen also headlines from Amazon and like the token spend that's going crazy. And you have the CFO and engineering wondering what are we shipping with these tokens? And it's not just how much we're
[04:57] spending, but truly like what is the outcome? What you see on the leaderboard essentially, you can think of it as your typical engineering like KPIs on productivity metrics. So, for example, number of merge PRs, number of reviews,
[05:10] and how these lines of codes persist over time. So, they're like durable outcomes, but also AI spend, which is something that you can reconcile with the the productivity and engineering
[05:23] data. What you see in this case and something that we've noticed frequently is that the highest spenders in your companies are probably not the people that are most efficient at using AI. And that's a great opportunity also for teams to
[05:35] notice where they can upskill and like bring standardized practice on how to produce good outcome per token spent. The main thing also I want you to take away is that you cannot get this full
[05:47] story with a single data source. If you wait for your OpenAI or Anthropic bill at the end of the month, you you have no clue how productive you were with these tokens or not. You need to join them with other sources.
[05:59] Another common case, in this case it was a customer support bot for a bank. And there were discussions between product and engineering and they couldn't agree on how performant their solution was because each team was
[06:12] relying on their own end-road script to process this this these agent traces. And so, if you've been working in analytics or in data science for a while, you know that shared definitions is a table stake.
[06:24] And we've been helping them build observability over, for example, the number of turns that it takes to complete and tie these specific accounts to like the outcome of if the
[06:36] ticket is resolved or not for a given request. And what it allowed teams to do is to decide when to get a human involved because past a certain threshold around like 15 turns, you
[06:49] usually get customers in an endless loop where you spend tokens and you provide a bad experience. And so, having these analytics allow you to make data-driven decisions. And so, the last case I'll share with you is
[07:03] where finance and sales were discussing how they were spending tokens on their agentic features. You've probably seen different products that offer summarization, transcription, even agentic search in their product and
[07:16] often this is heavily subsidized. So, for the finance and sales, it's important to be able to attribute these AI spends and the outcomes per customer because you probably want to budget a
[07:28] high volume of free usage for your customers that are ramping up or for customers that are big accounts, but you want to be also able to downsize or limit people
[07:40] based on their plans. And if again you wait only on your bill at the end of the month from your LLM providers, you'll be missing where agent features are used and where they're useful. And again,
[07:53] this is a story where you need more than one data source to be able to answer any of these questions. So, the second part of this discussion will be a bit more technical and we'll
[08:05] share how we build a foundation for any of these solutions. There are really two problems that you need to solve at first, ingestion of agent traces and then transformation. And this will be obviously on Databricks
[08:17] using DLT. So, what is an agent trace? I've been saying that several times already. You have a screenshot of MLflow. If you're a Databricks user, you've probably seen this or used this before.
[08:29] And an agent trace is essentially a full record of the messages, the file edits, the web searches, the command executions for a single session. There are many related terms or synonyms in that space. You will hear conversations, sessions.
[08:43] AI labs will often call them rollups or span, AI telemetry. But what it really means is you need to have a full record of what the agent did in a single human interaction or session with a
[08:56] human. One interesting thing here is that these tools, broadly called like LLM observability tools, were initially data engineer or engineer products where
[09:09] engineers live and like have different features, for example, like annotation, evaluation. But now that agents are used more broadly across the companies, they're not necessarily the best interface for business stakeholders or
[09:22] different stakeholders to interact with agent traces. So, this is also where we started reflecting on how we could get this data more broadly available across the company. So, the first thing is recording these
[09:35] agent traces. If you've been using Cloud Code, the CLI will just produce big JSON file JSONL files on your disk, so you can actually go on your own computer and see
[09:47] what it looks like. But in when you're using agents inside your product, what it will typically look like is the agent This is a very simple diagram. We'll add complexity in the following slides. The agent is your
[10:00] Cloud Code CLI. It can also be code-based. So, for example, if you define an agent using the agent SDK by Anthropic, the Codex SDK by OpenAI, the Pydantic AI SDK,
[10:12] you need to add instrumentation because the agent alone will not produce any record of what it's doing. And typically, you will send these traces to an observability platform. You have your more general
[10:24] observability platform in the Open Telemetry ecosystem like Datadog or Grafana. But you also have a set of specialized tools like Arize, LangFuse, LangSmith, Pydantic Logfire.
[10:38] And there are challenges when you keep a simple architectures like this. The agent traces live outside your data warehouse. So, like I said, they're not accessible to many of your users. They're not accessible to your chat BI with Genie. And they can't be joined
[10:51] with other sources like your KPIs and your different analytics. Also, storage is very expensive and in these custom-built platform or specialized platform, and it leads to engineering decisions which are typically to sample
[11:04] traffic or discard traffic after 7, 14, 30 days. This is unfortunate because I'm telling you your agent traces are invaluable data. Each trace captures unique human-agent interaction and
[11:16] they're lasting assets for your company. You should store them all of them and like keep them archived for for a while. Um but to do this, you need to be able to move it to a place where storage is less expensive. And Databricks is a good
[11:29] place for that. So, ingestion. Exporting the agent traces, DLT, again, Python library, very flexible. It allows you to build custom connectors, and we are providing off-the-shelf connectors.
[11:41] It can do just that, move your data from your purpose-built observability platform and move it to Databricks. Uh this is not competing with your observability platform. It's making your investment in this tool more valuable because now you're using your data in
[11:54] more places. And you keep your engineers happy with the the software and the workflow that they built on these platforms. But again, there are challenges. These traces contain sensitive data, and PI data. I'm telling you, if you think
[12:06] there are no PI data, you should look again. Uh questions like just the history with the timestamp, the user IDs and everything. And there are two common patterns that we see with DLT. You can uh
[12:19] process and anonymize and make the data compliant on the fly, but you can also do it after the fact. So, what's uh we see some customer do is move the data to a raw layer inside Databricks where they have very strict uh role-based access control, and this uh data will be only
[12:34] uh available to certain account or permission, then make this data compliant within Databricks, and then uh delete this original data that's very sensitive, and I've just the like raw compliant data landing in
[12:48] Databricks. So, there are two patterns there that you can do with DLT. And the other thing that like is a pain point for your engineering team is that agent traces are deeply nested and irregular. They require normalization to
[13:00] fit in a uh a SQL table. You can keep everything in a blob column, but it will make your scalability and your joins difficult. And DLT can do that automatically for you. So, it facilitates joins and
[13:12] storage um over time. Then you you have data in Databricks, but you need to do transformation because uh if you've been tackling this problem, you know that the formats are very uh
[13:24] um eclectic, let's say, and there are many permutations in the wild. You will have your LLM provider times your instrumentation tool times the version of each times the LLM observability
[13:36] platform that you're using, and there's a growing standard that's currently like being more and more adopted. It's a open telemetry gen AI semantic convention. I encourage you to Google it after this talk.
[13:48] And if you're using MLflow, you're lucky because there's a flag to enable this format. You can just like set a boolean like true, store it in the common format that's being adopted everywhere. But we can help you transform it
[14:00] to that format. We support like many of these permutations out of the box. And there are other conventions or standards that you might be interested in. To name a few, the OpenAI response
[14:12] format is very common for everyone like training LLMs or working in that space. And the Hugging Face conversations format because the Hugging Face ecosystem has many capabilities for fine-tuning that are based on that. So,
[14:24] as long as you pick one format and you're able to convert all of your traces to that format, you're good. And I I I highlighted it here. Like it's a problem you will face the moment you have more than one tool, and I'm guessing everyone has already five
[14:36] different types of agents running. So, you you probably have this problem already. And this requires maintenance because I I mentioned there are many permutations, but all of these tools evolve independently of your platform. So, there will be updates and breaking
[14:49] changes, and you need to handle all of that. And DLT Hub can help you on that front. So, cool. Now you have common traces, and you want to do analytics.
[15:01] This is the easy part. Everyone loves to do like nice dashboards and metrics, but if you have a common trace format, it's really not that bad, and you're able to define your metrics only once,
[15:13] and it will work across tool. And this is where we really see like a wow moment for our customers. It's when they're able to see data from like from Claude, from uh uh for example, the
[15:25] Vercel AI SDK in the TypeScript ecosystem. And I've all of these metrics consistently applied to data from these different tooling ecosystem. And this is when like governance is enabled. Like this is what everyone
[15:39] tries to achieve, but you really need to go through all of these uh painful steps. And what's great is uh when you have these analytics, you can serve it in many different ways. This is my recap slide uh of what DLT
[15:52] Labs is building. Uh there are three buckets. I shared with you uh the bucket number one and bucket number two. We built standardized ingestion for all of your platforms, uh different permutations. And then we built
[16:04] specialized transformations for this data. And the last bucket is integration to the end product. And this is where I'm I'm excited. Like I work with AI Labs. I work with these uh design partners. And
[16:16] there are a lot of patterns that they want. Uh we have, for example, uh customers that want to provide a dashboard to the their end users, their customers. So, they can just vendor what we build. In that case, like you have a
[16:28] very expensive uh um AI Lab uh budget for your team. They don't want to be spending it on data engineering problems. It's seen as like uh task that's not for them. And it's also like maybe outside of their
[16:40] skill set. And so, that's something that we can offer. For other uh cases, you just have uh analytics for uh higher like business level or C-level management. And uh another case that I'm
[16:53] working on is uh with a leading AI Lab that's working on post-training and distillation. And with them, it's more so having a reverse ETL pattern where we can provide them uh fresh data sets for them to do uh
[17:05] post-training. So, the key takeaways. Uh again, this is to make everything digestible. I've been speaking fast. I am looking forward to continue the discussion with you.
[17:17] Number one, agents are everywhere. Every team is asking, "What is our agent doing?" If you have this question in your business, you should start by wondering who's asking, what do they mean by that, and start from there
[17:30] because it can mean very different things depending on who's asking. Second, answers require joining agent traces with existing KPIs that team already own.
[17:42] I don't know the specific case you're in, but it's a good way to introduce more agentic and good practices into your company if you're able to tie using agents with existing KPIs. Don't want to
[17:54] invent reinvent the wheel. You want to be able to show that your token spend and your engineering efforts are worthwhile. And so, having these agent traces in your data warehouse and adding them to your
[18:07] different models that you have for governance is a like straightforward way to do so. And then, finally, like the point three, ingesting, normalizing, and joining
[18:19] agent traces is table stakes. And I would say that it's currently unsolved. I believe it will be solved in like two or three years probably. Maybe there will be just a solution or maybe everyone agreed on a convention and everybody's respecting it. But what I
[18:31] encourage you to do is to have a plan now and not wait for someone to solve it for you. And if you want to work with someone, I think I'm a good partner. I think the LTM is a good partner. We're actively working on this. We're working with
[18:44] customers and we're also working with technology partners. There's a lot of different presentation on traces and a lot of people we've been talking to here at the Data Bricks conference and you should hear more about DLT in that space in the following weeks and months.
[18:56] So, the QR code is actually my calendar. If you want to talk and book a meeting with me, or otherwise, I will be right here to have a discussions and questions. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.