Deploying AI Agents: Planning, Context, Testing, and Observability

Summary

• Aparna from Arize shares four lessons learned building their AI engineering agent Alyx in production: keeping agents on task with plan tools and finish gates, managing context overflow with handles and structure-preserving compression, testing non-deterministic behavior using golden production traces and LLM judges, and debugging production failures by chaining traces across observability platforms.
• A key architectural insight is separating planning state from conversation history, which prevents agents from losing multi-step task instructions mid-workflow even as the conversation context grows.
• This video also covers a paradigm shift in observability — moving from human-readable dashboards to agent-accessible structured telemetry — enabling agents to autonomously diagnose and fix issues across an entire infrastructure stack.

Deploying AI Agents: Planning, Context, Testing, and Observability

Shipping production-grade agents like Arize's Alyx reveals challenges that lab testing misses: agents forgetting multi-step requests, getting overwhelmed by tool output, and becoming invisible in production. Aparna from Arize shares four fixes that held up at scale: enforcing rules through plan tools and finish gates to keep agents on task, managing infinite context with handles and structure-preserving data compression, testing with golden production traces and LLM judges, and debugging failures by chaining traces across observability platforms.
Learn how to separate planning state from conversation history, use hard output limits to prevent context overflow, turn production sessions into ground truth, and run CI validation using live APIs. Includes the paradigm shift from human-readable dashboards to agent-accessible structured telemetry, and how to enable agents to autonomously diagnose and fix issues across your entire infrastructure.
🤝

Chapters

• 00:00 Introduction and Four Lessons from Production
• 02:12 Lesson 1: Staying on Task with Plan Tools and Finish Gates
• 08:03 Lesson 2: Context Management and Large Data Handling
• 16:13 Lesson 3: Testing Non-Deterministic Agents
• 22:24 Lesson 4: Production Debugging and Observability
• 24:21 The Observability Paradigm Shift for Agents
• 26:31 Giving Agents Access to Your Telemetry Stack

FAQs

How do you keep an AI agent on task when it handles multi-step requests?

Arize solved this by introducing plan tools and finish gates, which enforce that agents complete all requested steps before ending a task. Separating planning state from conversation history ensures the agent retains its full task list even as the conversation context grows over a long session.

How did Arize handle context overflow in their production AI agent?

Arize used context handles and structure-preserving data compression to prevent agents from being overwhelmed by large tool outputs. Hard output limits ensure that individual tool responses do not flood the context window, keeping agents responsive even when processing large data sets.

How do you test an AI agent that behaves non-deterministically?

Arize built a testing approach using golden production traces — real sessions captured from production — as ground truth for evaluating agent behavior. They then run CI validation against live APIs using LLM judges to assess whether the agent's responses and action sequences meet quality criteria.

What is the observability paradigm shift for AI agents?

Traditional observability dashboards are designed for humans, but AI agents need structured, machine-readable telemetry they can query and reason over directly. Arize built an observability layer that exposes traces and metrics in formats agents can access, enabling agents to autonomously diagnose and fix issues across the infrastructure stack.

Full transcript

[00:03] So, hey everyone. My name is Aparna. I'm one of the founders at Arize. I lead up the product team here and this is a actually going to be a really fun presentation. It's actually about an agent that our team has built ourselves. And instead of sharing kind of just some of the best practices around evals and
[00:19] debugging, I actually just wanted to share some of the lessons we learned along the way. Some of it was expected, some of it really wasn't and hopefully all of this is help useful to everyone in this room who's kind of building building an agent. Our entire product has actually changed over the last year.
[00:35] You used to have to go in, go figure out a UI. We now are completely agent-first. And this agent that you're looking at on the screen here is called Alex. User can come in here and do all sorts of things like say, "Hey, help me go trace my agent. Help me go, you know,
[00:53] um I can go create a new chat and say, "Hey, help me go try the playground, build an evaluator." These are all workflows that our agent actually handles today. And we learned a lot in actually building building this agent. So, I'm going to walk you through a bit of those lessons today. So, we're going
[01:08] to talk about Alex and I'm going to share a bit of the lessons that we learned while we were building Alex. I'm going to share some of the four lessons we learned. One of them is about staying on task, the other one's about context management, next is about crystallizing good behavior, and then the last is about actually debugging a really good
[01:23] agent. I showed you guys kind of in the product what Alex was, which is our AI engineering agent harness. This actually has context about all of your traces. It has context about your evals. It understands your sessions. And it also
[01:39] it has a whole harness underneath it that does planning, has access to tool calls and skills, and then memory that's used across the sessions that people do with Alex. We built tools to basically help people build AI systems and we thought, you know, we should probably build an AI system and in the process of
[01:55] doing that, we actually learned a lot. Some of it was expected, some of it wasn't expected, and hopefully you guys get some lessons out of this that give you a bit of a head start. The first big lesson that we learned was actually about staying on task. And this is something every agent builder
[02:12] actually hits. You ask the agent to do three things, it does the first one, and then it wanders off. And in our case, we actually had a tool called finish, but the agent would always call that tool call before it was actually finished. Like, you would give
[02:27] it three different things to do, it would finish one of them, get really, really deep, and then it would just kind of, you know, quit. Um this is a real issue. And you know, I can't really blame Alex because this is what junior engineers do, too. Um but it's just not what we actually
[02:43] wanted. And this actually happened in our very first version of building Alex. In one of the sessions, there's this person that kicked off essentially 27 different tool calls with their question. And it just did all these 27
[02:59] different tool calls, but it didn't know to come back and actually answer the user's question. So, it went around in circles. And you might ask, like, why is this happening? It's not a hallucination problem, it's not a capability problem, it's actually an attention problem.
[03:15] The model's context window um gets filled up with all of the information in that scenario, like with 27 different tool calls, it got filled up, and it got filled up with all the intermediate data that was collected in order to answer the question. So, the initial original kind of user question just got lost
[03:31] under pages of output. By the time the agent actually had enough information to be able to figure out what to do next, it forgot what next was. And so, the solution to this is actually planning. Um the fix is to actually make the agent
[03:47] write a plan before it does anything. And this is not a polite suggestion. It's not something in the prompt that you're like, "Hey, you know, try to do this. You know, think about this ahead of time." It is actually needs to be a first-class
[04:02] citizen. It's a first-class tool call. And you guys know this, Cursor does this. This is an example from Cursor, Cloud Code does this. I mean, they all have really good first call to do objects. And what we did was actually this is an image of Alex. We built a
[04:18] really good uh to do or plan every time that any user asked any question. And we actually had three separate to do to do's. Um we had a to do write, we had a to do update, and we also had a to do read.
[04:34] Um to do write actually kind of creates the plan. To do update actually, you know, changes the status of a task. So, if you're midway through one step or um you know, calling a specific LLM call, it actually updates whether the task was completed. And then to do read
[04:50] actually fetches the current state. These tasks also have states. So, you can see here there's pending, there's completed, there's blocked, there's also in progress. Surprisingly, in progress was actually the one that we added later. And it
[05:05] ended up actually becoming one of the most important states um that we learned. And this is because when you have only pending and completed, the agent doesn't actually have any working pointer. It doesn't know if it started a task or if it's completed a task. It doesn't even know
[05:22] what it's doing right now. And so, what we did was especially for these type of complex tasks, we actually, you know, stuff that takes multiple turns to actually solve, we actually added this in progress that gave the agent a really concrete anchor. This is my task. This
[05:38] is what I'm doing. Uh everything else can wait, but make sure you finish this. And actually, you know, in our evals, adding this in progress was something that drastically improved task completion rates. Another big critical architectural decision. The plan
[05:54] is not stored in the conversation history. So, how many of you guys have an agent? You have the human conversation where you're actually interacting with the end user, and then you also have your plan. Well, what happens is that it's conflicting because the agents now has the user messaging, it has the plan, and it can get buried
[06:11] or it can get truncated. Um what we realized and what ended up working for us is that we have to reinject the plan into the context window on every single LLM call. What this did is that it dynamically generated, you know, what the state was. Um
[06:26] we put it right after the system prompt and before all of the entire noisy tool call history. And this meant that no matter how deep the agent actually got, the plan was always right up front and center. This is what the agent sees. This is literally what the agent sees
[06:41] actually. Um we call this the plan message. And basically it tells the agent exactly what to do next. Um it tells it which tool call to make, which LLM call to make, what to do after the current task is finished. Um and it's not just kind of displaying
[06:57] the plan, it's actually coaching the agent on what to do step-by-step. And we learned a couple things actually in this. One is that we needed to have this thing called a finish gate. If the agent tries to call the finish tool, but it didn't finish some of the things on that
[07:13] to-do list, it actually we threw an error. It wasn't a suggestion, it actually wasn't even a prompt. This was actually a real error that was a structured error message that lists, "Hey, you didn't do these two other steps." Um and it bounces the agent actually back
[07:28] into the work loop. And we did this because then the to-do list can live outside of the context, it lives on the disk. The agent can actually, even if it tries to finish early, the system actually won't let it. Uh there's one big exception though, which was actually um what we call the
[07:46] blocks blocked status, um which is if the human responds, so a human jumps in and tries to add context, um that's the only one where it's actually pauses what it does and the plan gets restored with the context that gets injected.
[08:03] So, this is all a really big uh you know, one big lesson number one, which is lessons from planning. If you want agents to plan a rule, you know, I think we're now at a point where it's no longer, "Hey, the model's an issue, the tool call's an issue." It's really
[08:18] you're not guiding the agents in the way that is convenient to them. You need to put the rules in the code, you need to put um um not not just in the prompts, but you actually need to put them in code. The prompts are often suggestions, um but you need to put real constraints for the
[08:34] agents. So, for us the finish the finish gate was actually one of that. Um we had a lot of few-shot examples that actually showed the agent what to do instead of just telling the agent what to do. And what this did was actually showed the agent what really good planning was. So,
[08:51] all of this was one really big lesson um around a major improvement in our agent's performance. The second big lesson that I'm going to talk about is context management. So, what we learned was that the to-do plan that was in our agent needs to live
[09:07] somewhere that's really specific and reliable and can't be buried inside the conversation history. Because the conversation history can get truncated, it can get Imagine, you know, users store you know, millions of spans in a rise,
[09:23] they store hundreds of experiments. This can all the plan can get truncated by just or pushed out by just all the JSON that is getting returned back. And so, what we realized is that this needs its own home, and we decided to give it its own home.
[09:39] Um this is how we learned the lesson actually. In the very early days of our agent, we did I mean we took shortcuts. Like we realized that um if you tried to put two different experiments inside of the system prompt, one of the system one of the experiments
[09:57] can have hundreds of rows. And so we were like, well, let's just make sure it never compares more than two experiments at a time. But I mean users run thousands of experiments in our product. And so you know, we tried to solve this like
[10:12] context overflow problem with a band-aid. We just said, do less. Um but that's not a real solution. It's just giving up. Um and so what we did was basically um we started to think about how cursor and
[10:27] Claude code navigate through a large code base. They don't just dump the entire file into the context window. They read a preview. They use grep to actually find what they need. They read just specific lines. I threw the
[10:42] screenshot because it shows like plus 90 lines. Um and what's important about this is that you know, the file lives on the disk, but the agent holds a reference to it. And we realized that we actually needed a pattern like that for handling large
[10:59] data. So we built this abstraction. We called it large JSON. Um Don't tell us we're bad at naming, we know. Uh but when a tool returns a really large data set, basically our agent Alex doesn't just put all of that inside the
[11:15] context. It stores the full data server-side memory, and then it just gives the LLM a preview. Plus, you know, the ID to that large JSON. So that's a way to to handle kind of a stable way for us to go and retrieve that piece of data. So now what the
[11:32] agent can do is you know, it can read just the preview. If it needs the entire large amount of data, it has a way to go back and get that. Um and so, it's it's basically just like having a file name that you can then go back and pull from disk. Okay, cool. So, we did that.
[11:48] But then we realized something else. I mean, the obvious approach that we took was just take the first, you know, 10 lines. Just take the first 100 lines. Um and and then, you know, obviously one line can have hella tokens. So, we're like,
[12:03] okay, let's be smart. Let's just take the first end tokens and stop. But what that did was actually it showed the agent just a few complete rows, but with no idea about the actual structure of that data. So, what we decided to do was
[12:19] actually, how do we compress the actual value um but not the structure so that the agent understands what the entire JSON should look like. So, what we did, um what Alex does, is it now it
[12:35] compresses the values inside the JSON JSON, um but not the structure. So, it walks through the entire JSON tree and it truncates every single value inside of a cell to 100 characters max, which means that every field, every key, every
[12:52] array element um that appears in that preview, you know, except for like really long strings, everything um ends up getting truncated. But the agent ends up seeing actually the full shape of the entire data. And that's what you need to
[13:07] actually write a useful query. So, we empowered our agent with really small composable tools. So, things like, yeah, jq, uh the same jq you'd use on the command line, uh and grep JSON, which is just regex search um over serialized data.
[13:25] So, the agent can now write JQ expressions to slice, filter, aggregate, transform the data. And we also had some kind of hard budget, something like no tool call output can exceed 10,000 characters. Which meant that no matter how large the actual underlying data
[13:41] was, each query actually cost a predictable amount of context. Um and there was no overflow. You just had to make sure go through extra turns if it needed it. So, you're going to hear me say this again and again, but small composable tools is something we
[14:00] we realized and it's it's not a new idea. It's a really old idea. I mean, JQ's old, grep is even older. Um but these tools are are really small. They do one thing really really well and you can compose them on top of each other. They usually kind of accept, you
[14:16] know, the input, they produce some output, and basically this is all we needed in order to make sure that our agent handled context really really well. Um what this did was basically enable us to manage infinite amount of context.
[14:33] And you know, this was probably one of the biggest takeaways for us because before large JSON and, you know, enabling these small composable tools, every time a user wanted to go and, you know, do analysis over 100 spans or compare multiple experiments, we just we
[14:50] just were in this problem of context overflow. And we tried band-aid and band-aid and band-aid limit the smaller amount of data, but what we realized is if we were just smart about how we handled the data and and the agent could ask for more information when it needed it, um
[15:05] this basically, you know, got us out of this like context exploding dump problem we ended up in. So, now there's no limitation. You can handle hundreds of experiments and our agent can actually completely handle that. So, these are some of the lessons from kind of the the context management.
[15:23] One is hard token budgets on every tool output that actually gets passed into the context window. Um compress values, not structure. The agent should know what your data looks like, but it doesn't need to have all the text kind of, you know, shown to it
[15:41] in order to be able to understand that structure. This is an important one. Use recoverable exceptions when creating the the feedback loops. Basically, um every time we got into this problem where it failed, the tool call failed because of some, you know, context overflow, we started end up having these
[15:57] ways for the agent to go back and learn, "Hey, query a smaller and smaller piece of data until, you know, you don't overwhelm the context window." So, this is a this was lesson number two. Lesson number three is a really important one. This is um probably one of the hardest parts when you're building an agent, which is how
[16:13] do you test this? Traditional software engineering tests kind of assert deterministic behavior. You have X, expect Y. Agents really break that because your output is non-deterministic. The same prompt can
[16:29] produce different responses every single run. And all of those could be correct. So, you can't just assert on exact text matches. Um you can't even, and I know some people do this, which is did it call the right sequence of tool calls?
[16:46] I mean sometimes it might make an extra tool call and it still gets to the right answer. That doesn't mean that it's necessarily wrong. Um so, it's not I I personally don't think that asserting the same sequence of tool calls is the right thing um because some users might
[17:01] ask a really complex question that asks the agent to go through multiple tool calls. I mean, this is where we started, it's where everyone starts, right? Which is vibe vibe checking, right? We watched Alex, we eyeballed kind of the output. Um Um, we we asked ourselves, "Does this
[17:17] look right? And then we shipped. Um, and then it doesn't scale. This is where we started getting into all these problems where we were realizing we were just kind of dropping responses or not able to handle big big data inputs. Um, and this absolutely vibe checking
[17:33] doesn't you know, run in CI and you you just can't build a real production agent with vibe checks. And so this is something that we it was probably one of our biggest key insight, which is don't write expected inputs by hand. So,
[17:49] you might know what you want your agent to actually say or respond with, but just let production actually tell you what good looks like. When a user has a really good session, then you know, when when Alex actually does the right thing, we literally just take that session and
[18:05] we're we're like, "You're a golden example. Like you are what we should be trying to do." And those traces actually give you everything. It's what the LM did, it's what tool calls were called, uh, in what order, and we basically turned those traces into our ground truth. Um, we didn't write it. A real
[18:23] actually user and a real, um, it's a it was a real instance of Alex that was actually the ground truth. Um, and then what we did was we took that and we created levels of testing. The first level of testing, we called this like decision point tests.
[18:40] Um, and these are basically pie test based um, they're they're they're pie tests that basically validate very specific agent decisions. For example, like if you're building a message history up until some kind of like decision point,
[18:56] you we would actually run the actual orchestrator and assert that this output is correct. Like it has access to these tool calls, these traces, uh, this is the number of experiments. And we we had flexible logic, right? So like, you know, 2 seconds, 2,000 milliseconds.
[19:13] Like, there's there's some flexibility, but for the most part, it's pretty deterministic. That's, you know, this is awesome for when you're writing code-based tests, you want to make sure you're as deterministic as possible. The second stage is actually when you start getting into this idea of like trajectory-level tests. This is where
[19:29] you're measuring did the agent accomplish the task in front of it. So, entire sessions, we would just extract them from the production traces and replay it against our code. Pull the trace, extract basically what the user asked
[19:44] for, replay it kind of row by row, except in this case, it's like span by span, to go through the the real orchestrator, and we would score the output of those results with an LLM judge. You know, did it you know, did it get to the same place? Did it trigger the right
[20:00] next step? Did it actually do a different path this time? Try it with different words and see if it actually got to the right response, but um this is this was a little bit more about testing the trajectory of the entire agent to see if it accomplished its
[20:15] goal. And what we realized here was that LLM as a judge was a really good way for us to do this. Um but the evaluation prompt that we actually used mattered a lot. Um and it needed to be kind of flexible enough so that it can handle kind of all
[20:31] the nuances. And this is this is maybe a really I can give you some examples. Like, this is a really subtle one that happened to us, but we changed one of the tool calls under the hood that our agent would call. So, instead of like a you know,
[20:46] get trace preview, we we named it to something else like get traces. And when that tool call got renamed, the agent basically tried to call a function that didn't exist, and it made this whole runtime failure completely invisible if you're just using like unit tests or if
[21:03] you're stubbing data. Um and so, what we started doing was actually we put this into CI. Anytime anyone now makes a change to our agent, there's experiments in CI that run live. Anything from a prompt to a tool call. And Claude code review actually now runs
[21:20] on every single PR with those validation rules. And this is great because it means that somebody could change just a prompt within Alex and we're essentially using, you know, natural language to vet natural language. To be honest, this is the part that I kind of love. We get to
[21:36] dog food our own product. We're an eval's platform. Our entire test run inside of arise and this is a way for us to make sure that if we're able to bring our own agent's performance up by doing these type of CICD checks and evals and it's pretty good validation that we're we're on the right track. Here's some of
[21:52] the lessons that we learned from kind of this this lesson number three of crystallizing behavior. Build some infra to actually capture your traces and your golden sessions. Don't wait until, you know, you need to really, you know, set something on fire
[22:09] from from building tests. Run against real APIs, not just mocks. I think that this was this was kind of one of the the big ones that we we learned. This is my last lesson for you guys, which is so, we've done planning, we've got context management, and now we
[22:24] have actually tests running in CI. You deploy. Something goes wrong. You're running an LLM in production. I mean, something's going to go wrong. And one of the things we learned is that manual debugging is just really, really hard. A human's debugging Alex is opening up
[22:41] arise. You might have some stuff that's related to um code and that's in data dog and then you might have something that never even made it to data dog and that's in your GCP logs. And it's just it's a very tedious thing to debug because it's often cross system, really, you know,
[22:59] messy data to traverse. And this is something that LLM's are actually very, very good at. And this is something I want to share with you guys which is a bit of like a change that we're all kind of going through as an industry right now. Um of how software gets built.
[23:15] You know, phase one which is kind of like software 1.0, this is the world that probably at least I started engineering in, probably the world that most of you guys grew up in, which is a human writes code in an IDE, the application runs, it emits some telemetry, the human then goes reads
[23:30] some dashboard and some logs and then the human decides this is broken, this is what I need to go do to deploy a fix. Um but it's the human kind of completely doing this. Right now, most of us are probably in phase two, which is um the human's kind of prompting a coding agent inside of like cursor,
[23:47] cloud code. Um the agent's writing code, but the person who's probably still looking at the observability data, still the human. The human's reading the dashboard, it's making hey, this is a bug, and then it's actually using that to um you know, answer me
[24:04] and then they go back to the agent and use it to actually go fix code. I'm actually really excited about where this is going, which is phase three, where this goes autonomous. The agent should be able to have access to your observability platforms. It should query traces. It should have access to all of that telemetry and it should honestly
[24:21] implement the changes and then just let the human know. The human's role should really shift to now being, you know, monitoring over the system. And this is kind of my thesis, which is observability platforms were built for humans, all the like dashboards, whatever, the paradigm is now breaking when the
[24:37] primary consumer of all of this telemetry data is now an agent. Um agents don't care about charts. They don't really care about all the pretty visualizations. They care about structured data and they care about CLIs and that interface layer has to evolve. Um
[24:53] and this is this is why um you know, the source of truth is actually shifting right now. When you think about uh traditional software, and this is actually Harrison Chase from LangChain said this, and I think he's actually really right. Um when you think about kind of traditional software,
[25:09] you can actually go through and read every line of the code, and every path that the code can take is really visible to you. But here, it's non-deterministic. You don't know what the agent's going to do every time it's making a decision. Which means that
[25:27] you know, every single thing that actually happens is logged inside of the traces. Um and this has a direct implication for coding agents. A coding agent can't just look at your code and see what happened. The coding agent actually needs
[25:43] access to these traces to figure out what is happening at runtime. And it can figure out what the errors are, it can figure out when there's tool call loops that aren't working correctly. And if it doesn't have access to that, it actually can't do any of this self-improvement loop that we all
[25:59] want to kind of get to. The thesis is give your agent access to your observability telemetry stack. Um the way this works for us, this is actually what we do, is our entire Alex agent has access to all of the data inside of our
[26:14] platform. It has access to the traces, it has access to the evals that are running on the production traces, it has access to the feedback. All of this is accessible via our CLI, um also as our skills, and essentially that's the bridge between your
[26:31] application your your coding agent being able to debug and autonomously start to improve your actual agent. And so we're actually doing this today internally, where the coding agent now uses that data to figure out every time a user hits an issue, we know it
[26:47] happened because we're looking at the traces. We can can automatically go put up a fix, and we actually have a human just review that fix, which is really, really powerful. Um the skills it's available actually today in case you guys want to use it with your own coding
[27:02] agent. All you have to do is just NPX skills at arise, and basically like these are some of the things that the skills can do. It can just help you instrument your entire agent. It can go pull all of the spans of your agent, surface up summaries, it can go help you write evals, and the vision here is that
[27:20] this is going to become the AI engineering loop where your agent is going to have access to all of your data. It's going to observe it. It's going to fetch the traces. It's going to fetch the spans. It's going to hypothesize. What should I actually go do? Form a theory about all
[27:37] the times your agent's giving your application has some kind of bad response. Your coding agent should actually hypothesize why. Uh it shouldn't guess from the code. It shouldn't just, you know, guess from, you know, console logs. It should
[27:52] actually look at the traces, understand what happened, and then it should actually design and run experiments that then get automatically logged, um and and you can measure the outcomes. If it's good, ship it. If it's not, go back, try again. If you have logs in
[28:08] other places, like in our world I was saying that we actually have our data inside of We have all the stuff that the agent does, um you know, the agent's perspective, all live inside of arise. We do also have DataDog for all the servers perspective. So this is all the
[28:24] latencies, errors, HTTP status codes. And then GCP actually stores our entire infrastructure kind of perspective. So anything that, you know, GRPC errors that never even made it into as a span. And so all of these we now have exposed
[28:40] as skills as a loop. Um and we just go ask our our agent go figure out, hey, you know, Alex gave this user a wrong answer, uh and go figure out what went wrong, and go go make an improvement. And it
[28:56] goes through, it figures out what, you know, what skills to call, and it fetches all of the telemetry data, and you know, we've caught a lot of things. We've caught things like out of memory issues. We've caught bad prompts. We've caught things that weren't even related to our agent. Um something that used to
[29:12] take an engineer on our team maybe like half of half an hour to go do, the agent's probably able to do in like 2 minutes, which is which is really awesome. Um we do have to be, you know, mindful about the skills that are exposed though. We only gave skills that can do
[29:27] safe things. So, we do have like wrapper scripts around some of the more powerful CLI calls. Um and you know, some of them do involve, you know, asking for permission if it really is um need needs that type of guardrails. So, the TLDR from kind of
[29:44] once you're in production, how do you debug an actual agent? One, build skills. They're low cost, high ROI. Uh second, be smart about, you know, making sure that the skills are have are wrapped in something safe. And then this is maybe the big meta thing, which
[29:59] is debugging an agent is an agent-shaped problem. Like, let an agent kind of do what it's good at, um and debug itself. Like, let it sift through the traces, let it sift through the logs, let it go correlate IDs and
[30:15] timestamps, and go down a procedure to narrow down a root cause, and and and the last one is, you might want to invest in observability before you actually think you need it. It's incredibly cheap. You get visibility into your agent, and it's probably going to help you make sure that your agent
[30:31] improves. So, these are some of the big lessons. What does this all boil down to? One is plan to not lose sight of the goal. This is really important so that your agent doesn't just forget everything it learns. Second, you know, it's almost counterintuitive to lesson one, which is but even though it gathers all that
[30:46] context, make sure you only provide it just enough context. You can always have tool calls to you know, composable tools to go get more. Three, when something does go really well, like don't just talk about the stuff where it's not doing well. Like actually crystallize the stuff that it does really well. And
[31:02] then, I mean, it's production, it's going to go bad. You might as well get you ready to go and debug and have the data and the tools ready to understand how to make it better. These are maybe two big themes of things we learned, which is context engineering is basically everything, how you handle context. The second theme is just it's a
[31:18] very weird but fun problem as software engineers, you know, it's always very deterministic systems, but we're all probably handling non-determinism and it's fun and a new type of challenge that I think we're all trying to figure out how to how to understand. So, thank you guys so much for the time.

Learn more about the Databricks Data and AI platform.