Debugging Multi-Agent Systems: Tracing Cascading Failures and Distributed Complexity

Summary

Moving multi-agent systems from playground to production exposes failure modes such as cascading errors, deadlocks, memory poisoning, and split-brain scenarios that cannot be fully anticipated before deployment.
A three-level evaluation hierarchy covering system performance, coordination quality, and agent handoff quality — combined with MLflow tracing integrated with Unity Catalog — provides the observability needed to detect and diagnose these failures.
Treating multi-agent systems as distributed software first, with trace-based guardrails, circuit breakers, and architectural governance, is the practical path to balancing agent autonomy with production predictability.

Debugging Multi-Agent Systems: Tracing Cascading Failures and Distributed Complexity

Multi-agent systems shift evaluation challenges from individual model outputs to the integrity of the coordination layer. When a supervisor agent delegates tasks with flawed context, errors propagate through the chain, creating distributed hallucinations that bypass end-to-end testing. Treating multi-agent systems as distributed software first and agentic systems second reveals the real complexity: cascading failures, deadlocks, memory poisoning, and split-brain scenarios that require tracing, observability, and architectural governance to prevent.
this video covers how to monitor orchestration and agent handoffs, evaluate three levels of multi-agent performance (system, coordination, and agent handoff quality), implement trace-based guardrails that halt execution automatically, and design architectures that balance agent autonomy with predictability. Learn how MLflow tracing integrates with Unity Catalog for analysis, plus practical approaches to human oversight, remote evaluations, and circuit breakers for production safety.
🤝

Chapters

FAQs

What are the most common failure modes when running multi-agent systems in production?

Common failure modes include cascading errors where flawed context passed by a supervisor agent propagates through the chain, deadlocks where agents wait on one another indefinitely, memory poisoning where incorrect state corrupts subsequent reasoning, and split-brain scenarios where agents have inconsistent views of shared state. Recursive hallucinations and schema mismatches between agents are also identified as typical problems.

How does MLflow tracing integrate with Unity Catalog for multi-agent observability?

MLflow tracing captures the full execution history of agent runs and organizes those traces by execution context, allowing teams to inspect individual agent handoffs and identify where failures originated. Integrating this trace data with Unity Catalog enables analysis and governance of agent behavior at enterprise scale.

What is the three-level evaluation hierarchy for multi-agent systems?

The three levels are system-level evaluation, which measures end-to-end outcomes; coordination-level evaluation, which assesses how well agents hand off tasks and context; and agent handoff quality evaluation, which examines the fidelity of individual delegations. Evaluating all three levels is necessary because end-to-end tests alone cannot pinpoint where coordination breaks down.

How do trace-based guardrails and circuit breakers improve multi-agent safety in production?

Trace-based guardrails monitor agent execution in real time and automatically halt a run when predefined safety or quality conditions are violated, preventing errors from cascading further. Circuit breakers serve a similar role by stopping execution when a subsystem is detected to be in a failed state, a pattern borrowed from distributed systems engineering.

Full transcript

[00:03] Super welcome. Super nice to see you. What I would like to talk today with you about is basically a how do you move from single agent in playground to a multi-agent application in production? Just short disclaimer, I don't have a perfect answer. So, I will tell you
[00:21] everything I know about this topic, but at the end of the day, um we all need to figure it out together throughout a lot of errors and trial and errors. So, let's start with um basically with a statement. Over the last few years, essentially agentic
[00:36] development moved from uh um single agent application in playground to uh like moving those applications to production. At the same time, uh we started moving from a single single agent application to a multi-agent
[00:51] application. So, if you want to understand how it actually looks like, you remember what it is, right? Like you I'm not the only one who knows what this where this screen is about. So, imagine if this is a single agent application um
[01:07] in uh your playground, this is more or less your single agent application in uh production. Great, we're getting somewhere. So, this is how multi-agent application in production looks like. So, ultimately, the biggest challenge is that you don't
[01:24] um there are so many possibilities of what's going on with a multi-agent applications when you move to production that it's almost impossible to encounter every single possible scenario, every single possible error. At the same time, there are a lot of ways to be more
[01:39] prepared if those errors will happen. So, today, I would like to talk to you first of all about what kind of issues you can expect with multi-agent applications in production. And after that, how you can start seeing those issues, start preparing for those
[01:56] issues, and potentially even solve those issues. Let's start with a simple one. How single agent application actually fail or where like okay, not fail, but where they struggle. So, I don't think those four issues are super new to you, but essentially one of
[02:11] the typical problem with a single agent application is a recursive hallucination. Essentially, typical example the tools that agent called didn't return any data. Agent didn't really understand that it result of empty interpreted in different way and started doing some
[02:28] kind of stuff. Second one schema schema between how you transfer data between different agents. This is not particularly new problem, but in agentic application it becomes even more critical, especially like you
[02:44] just heard from a partner she was talking about how you send like how sensitivity of a tool name impacts um reaction of your agent. This is exactly that. So, small changes in API field name basically drives your agent
[03:01] completely nuts. Context decay, yes, long prompts, long context, basically your initial task is lost in that context. And probably the last one which is kind of interesting one in the inference
[03:17] runaway. So, essentially at some point your agent doesn't start hallucinating and producing wrong results, your agent starts burning tokens because it tries to find out a correct path, correct results, correct way, and so on.
[03:32] So, this is a single agent application. So, this is your chatbot that that you built. How about the multi-agent applications? So, to start with multi-agent failures, I would like to you to imagine the following scenario. So, we have a
[03:48] simple system. It has three agents um executed almost linear. So, you have an orchestration agent. Then this agent is called a financial calculations agent. So, this kind of like financial use case. And at the end when financial calculations are done, you would like to
[04:05] build some kind of financial advice that will go to a end user. So, what happens? Imagine this scenario. So, your orchestration agent sent correct but a little bit ambiguous instructions to financial calculations agent.
[04:21] At the same time that agent kind of interpreted the way it wants, but a little bit like misunderstood the business rules that you had in the system. What happens next? Your financial advisor agent actually
[04:36] uses the flow output to perform analysis. So, while you're trying to understand why my analysis, why is the final result is wrong, actual problem is not there. And this is the biggest challenge with, multi multi agent applications, it's so
[04:52] called cascading effect. So, if you know a like a typical power grid example, this is what cascading failure is. The problem has even more, challenges because it has a different flavor with those agents are sharing
[05:07] memory. So, for instance, you have some kind of like a use store as a context, or information that those agents share. And what happens, this memory get like poisoned or polluted at some point of time. And again, you're trying to you
[05:23] debugging your data, you're trying to understand what happened, but essentially you're looking in the wrong place. So, you don't know where the problem actually is. Another cool failure with a multi agent system is a deadlock. So, it's exactly same deadlock as you probably heard from a database systems. Who knows what a deadlock is in
[05:41] databases? I hope some people are Yeah, I I genuinely hope so because otherwise it doesn't really help. So, essentially you have an orchestrator, you have a specialist agent. While your orchestrator is waiting for an answer, your specialist agent is very waiting for a confirmation
[05:58] from an orchestrator. So, none of the agent can really proceed, none of the agent can really do something, but what happens basically your latency is um increasing. So, you you don't really understand what's going on, but but essentially everybody is waiting and you
[06:13] don't get any results. These two problems and many many more related to multi-agent applications are not particularly new. If you ever read one of these books, also these books on the internet, or saw those books somewhere at the office laying in the corner,
[06:28] this is exactly that. Those books are talking exactly about this problem because it's not really new, it's a problem of distributed systems. And you need to treat the multi-agent systems the way you treated the distributed systems. But, there is like a two There are two
[06:45] specific things that are very particularly important for agentic applications. Again, first of all, agentic applications are non-deterministic by nature. If you run the same prompt 100 times,
[07:00] you will get a slightly different results. Not exactly same every single time. That's um differentiator number one. And differentiator number two are costs. Essentially, if you if you're a software developer and you remember a phrase of
[07:18] 100 clicks solved my problem, with agents it can be the case, but it's extremely expensive. Even right now we all like a token doesn't really matter, but actually it does, especially when your context grows and when this is a solution to to actual
[07:35] problem. So, in the next 13 minutes, I would like to tell you how you can actually monitor your systems, how you can build a robust observability to detect those issues, what you need to detect, and how you possibly can prevent some of those issues from happening.
[07:50] Let's get started. So, let's first of all establish what is the fundamental difference between a monitoring distributed systems and monitoring a single single threaded single agent application. So, in single agent, we like our focus is very narrow. So, what we are looking
[08:06] at essentially, we monitor input and we monitor output. And yes, we do also check inference parameters. We do check We do monitor tool definitions and so on. But essentially, we are always looking at a very single threaded application.
[08:22] If we try to monitor distributed, so multi-agent system, we actually need to move beyond this. So, we we don't simply monitor input and output, essentially your context prompt and completion the result of your application.
[08:38] You need to monitor logic. Logic that happens between when when a supervisor agent decides which tool to call, which agents to send this information to, how the context is propagated. Do you have some kind of resource concurrency?
[08:55] Imagine you have a following system. So, you have a supervisor agent and you have three agents doing some work. Those three agents might access exactly the same information. How do you ensure that they don't change the I don't know, like a simple example, how you ensure
[09:10] that they don't change the same information for the same user at the same time. How do you prevent those those situations from happening? Then you have rate limits. Again, three agents trying to access the same API and instead of like doing nice job, they are
[09:25] just basically blasting the request to that API all together. And this is something that you probably would like to know. There are a lot of like more complex situations with a multi-agent system, like for instance a split brain scenario. When
[09:41] again you have a supervisor agent, you have a multiple worker agent. And at some point your worker agents have completely different view of reality. Why it happened? Again, nobody knows circumstances and so on,
[09:56] but essentially this is a very very difficult issue to resolve. So to be able to do it, you need a few things. Here I will be giving an examples and screenshots from ML flow, but essentially it doesn't really it's not about ML flow specifically, it's
[10:11] about how you need to treat your multi-agent system. So first of all, if in a single agent application you can technically rely on a your traces kind of like in a hierarchical or basically in a um
[10:28] expressed throughout the time, so you can sort it by by timestamp and you will be fine. When you working with a multi-agent system, you actually need um you need to organize your traces in a way that you can see what happens within that particular execution of your
[10:45] multi-agent system. Why it's important? Because you can have like a multiple multiple systems have multiple calls, multiple uh tasks happening in parallel and you need to be able to identify those bottlenecks. Another important thing
[11:00] that operational telemetry is not going away. Everything you know about like a through development, everything you know about development of an agent as of now, you still need to collect information like for instance per node latency. You need to know a token attribution, you need to know input and output between
[11:17] between different agents. The only um version where it gets more complicated is because in a single single agent application you need to know it end to end. In a multi-agent application you you
[11:33] need to know it of course end to end, but you also need to know it per node, per combination of nodes, per pass, per configuration of particular reaction or particular response. One of the hardest problem to solve when you are talking about multi-agent system is the
[11:50] difference between intent and execution. It is hard in single-agent system as well, but in multi-agent system it gets like a different flavor completely. Because it's very hard to tell is my agent fail because the model is bad or because it didn't get the
[12:07] instructions that it needed to get to execute this perform this task correctly. To be able to do it, you need to be able to link the instruction of one agent or like a supervisor agent and a input of a worker agent.
[12:22] If you're able to link if you're able to get that audit trail, then you're in a better place to actually understand what's going on with your multi-agent system. So, particularly in MLflow, you can get a lot of this information kind of automatically by using an auto log
[12:38] function. Again, it supports a lot of flavors with the different open-source frameworks, with the different frameworks that you used to develop your agent and so on. But essentially, that is only part of a problem. Because when you're building an actual
[12:54] multi-agent application, you need to to monitor much more than just what you get out of the box. Why? Because multi-agent systems are complex by definition in terms of like all passes that exist between different agents.
[13:09] Because of that, you also need to have a lot of custom tracing to different functions and different tool calls that you have in between. In MLflow, you can use it with a decorator, but essentially the point is that you can rely on out-of-the-box functionality, but you also need to have
[13:26] much more custom parts. Special flavor of monitoring a multi-agent system is for instance when your system is split through multiple services. And typically it's in a software development is not as that hard problem to solve. In a agentic
[13:41] application you actually need to be able to link your traces for your actual agentic application and the different services that you have next to it. How is happening in MLflow? MLflow is auto compatible so open telemetry um
[13:59] uh it has open telemetry SDK which allows you to actually um use the same traces use different spans and come back come combine them in a root span in a one single view of your platform. If you're using it within Databricks it
[14:15] also has a integration with Unity Catalog which is actually pretty funky because having that you can also get like an extra layer of analysis to your traces because you can actually analyze them as you would do with any big data um like
[14:32] outside of agentic scope all together. But now let's see how you can actually evaluate your system and how you can build the guardrails to prevent things from happening. Coming back to a single agent application the biggest um the biggest simplicity in a single
[14:48] agent application is that your path between different um basically between your input and output is more or less deterministic. It's not deterministic in the sense of like results of what your LLM returns to you and so on
[15:04] but you can kind of predict what kind of path it would be. In a multi-agent system the path can be anything and the the biggest challenge for us as human being and as a development developers of this platform is that we don't really know
[15:20] what the right path for that what that problem is. Example, for instance, I want to go from A to B. My view of reality is that I need to like, for instance, I'm going from Amsterdam to San Francisco. My point of view is that I'm taking a
[15:36] plane, 11 hours and I'm here. At the same time, um there are millions of different solutions how to get to San Francisco. And the challenge is that just because I imagine only one solution, it doesn't mean that there are
[15:53] no other solutions in reality. And here is actually what's important about agent. We all kind of blame them for hallucinations, but essentially hallucinations, it's also something that we actually want. We want We want them to be smarter than us and we want them to develop like a
[16:10] solution that we haven't imagined yet. But when you're building that system, you actually need to balance two sides. On one side, you want to it to be creative, but on the other hand, you want to know what it's doing. Again, a challenge.
[16:27] Special flavor of it is when your system actually has side effects. And in it's the same for a single agent or multi-agent applications, but in multi-agent applications, it's just bigger. Like the the the blast of radius of
[16:42] uh problems is bigger. So, imagine if you have like, for instance, if your agent is sending like a Slack messages, doing something or like reporting, I've done this good job. You can't predict that in multi-agent systems that your agent will actually do
[16:58] it. So, you need to be able to on one side enforce it, but on the other hand, be able to uh to monitor it in a way that you know what's going on. So, how you can evaluate your multi-agent system? To me, it's easier to see everything as
[17:14] a sort of a hierarchy. It's It's just pure Maya kind of mental model. So, I see it as a three levels. First of all, you need to understand like a system level um evaluation or system level evals. So, essentially, this is classic. You treat
[17:30] your multi-agent system as it was like a one big black box. So, you evaluate an input, output, user satisfaction, done. Second level is you need to evaluate coordination or essentially a conflict resolution. How different um
[17:47] different scenarios were handled. Were those scenarios an optimal scenarios? Did your um agent choose the most optimal path either in terms of a cost or in terms of um for instance, uh time like to to get to an answer or for
[18:04] instance, tools that it called and so on. And the third kind of the lowest layer, you need to evaluate an agent handoff. So, essentially, how one agent gives information to another agent. Is the context propagated correctly? Is the context complete? Is one agent just like
[18:22] a send the absolute insanity to another agent and it tries to make sense out of it. So, those are three layers. It wouldn't be like a technical presentation if I wouldn't show you a piece of code. So, this is an example how you can build like an agent handoff. So, essentially,
[18:37] the task is really simple. What the point I'm trying to make here is that everything that you everything that you try to solve here is not an extremely complicated task by nature, but it's a complicated because it's a lot of points of failure.
[18:53] So, here I will give you an example of how you can evaluate that this essentially your supervisor agent give a correct um basically correct um information to a worker agent. And for that, we will use an LLM as a judge. So,
[19:09] you you see here is I'm calling an MLflow um judges from a library. So, essentially what it allows me to do, it allows me to compare like information and logs from my supervisor agent, from my worker agent, and use LLM as a judge
[19:26] to understand was it good, was it bad, was it was it correct. Another example is you don't really want to only evaluate it. You also would like your to steer steer your agent in right direction.
[19:42] Right, like for instance, I don't know, I would like to set up guardrails in my system that my agent doesn't go beyond like a certain boundaries. Again, similar to evaluations, my guardrails also have certain form of a hierarchy. Right, my guardrails can be
[19:58] on a top level, meaning that I'm kind of preventing my system goes my that my system goes completely um wild. Then I try to build like a more narrow guardrails on a certain parts of a system on like for instance
[20:14] a certain path within my multi-agent system. And uh the most narrow focus is that I would like to have guardrails for a specific agent. To give you one flavor, I would like to build a guardrails that actually uh implements the following rule. It's like
[20:31] a more practical rule. So, no agent, regardless of a role, can call a wire transfer tool with amount more than 5K without MFA token. Again, back to our financial example at the very beginning. So, essentially to be able to do it, you
[20:46] need two things. First of all, you need to be able to capture that your specific tool is actually trying to do a wire transfer with a specific amount. On the other hand, you need to be able to enforce specific rule, basically your
[21:02] specific guardrail. Here you have two two parts of code where one is um essentially an eval, and another one is your guardrail. When we are talking about guardrails, there are a lot of guardrails that are much more
[21:18] let's say specific to your domain, but there are also guardrails that you have to have in your system. Like for instance, protecting from leaking PII data, protecting your your agents from knowing from getting too much information, and so on.
[21:34] So, in this particular example, this is an AI gateway within Databricks. But essentially, what um important here is not the fact that okay, you have this fancy UI, but the fact that those guardrails also should
[21:49] be part of your your multi-agent um monitoring and observability platform. How about human in a loop? We all say that human in a loop in the loop is important. Human in the loop is is good. We all want to have some kind
[22:06] of like a human supervision, especially when we try to prevent from cascading failures or different types of failures. What happened in reality, especially with a multi-agent system, is that indeed a human is still critical for identifying hallucinations, different
[22:22] nuances, and very complex scenarios within your multi-agent system. At the same time, don't be fooled. People can't really react um when your system goes completely wild.
[22:37] So, you are um when you try to evaluate your when you try to protect your system, when you try to build like a good observability toolkit, you cannot rely on a human being to be able to uh stop system from doing wrong things. What you need to do, you need to have
[22:54] like a very automated process that can actually do that for you. Essentially, manual intervention never stops a power grid outage. That's proven by by the fact that we still have power grid outages. Essentially, solution to that is that
[23:09] you want to have a very, uh, robust automated trace-based guardrails that halt execution. So, people will help you to figure out what happened, while then you are able to automate it in a way that it acts
[23:26] automatically every time when the system violates certain situation, or for instance, cascading failures happening. I talk a lot about observability, but essentially observability and guardrails and all that cool stuff is one part of a problem. If you remember one thing from this presentation, monitoring and
[23:42] observability is cool, it's great, it's essential, but essentially you also need to think about an architecture of your agents. When you're talking about a single agent applications, it's it you still need to think about architecture, but it's reasonably straightforward comparing to multi-agent systems.
[23:58] When you build a multi-agent, um, system, you really need to understand what kind of reaction, what kind of system you want to build. One simple example, do you want to allow every single agent to talk to each other? Or do you want to have a
[24:14] supervisor agent that delegates task only to specific specific agent with specific flow? Another question, would you like to have every single agent access to all possible information that is traveling through your system? Probably not. But if you limit it, how
[24:31] do you limit it? How do you create like that minimum scope for your agent that it can access only that particular information? Similar with tools. Do you allow every single agent in your system access everything that is available, every single tool that is available? Again, probably not. You
[24:48] would like to decide which the scope of those agents and the scope of tools that particular agent can access. Those are all architectural questions, and a lot of them are not particularly new. But essentially, those are back Those are questions that are coming back from a pure software engineering
[25:04] development software engineering best practices of how you build a distributed systems. And when you build a distributed systems, there are like a few things that you need to keep in mind. As first of all, it's a state persistence versus state diversions. So, essentially, how
[25:19] do you persist state of your agents? How do you implement the memory of your agent? On one side, it's an observability problem because you if you implement a memory, you have a lot more issues that you need to understand and you need to observe. But basically, you need to
[25:34] monitor what's going on with that memory. But on the other hand, how do you treat your memory of your agents from an architectural point of view? What do you store in your memory? Do you store its whole context? Do you store the whole conversation? Do you store only minimum information that the agent
[25:50] require to do its job? And so on. Second part is a deterministic routing and basically router accuracy metric. Again, you can build a system that everybody everything talk to everything and essentially, your system is
[26:05] completely like complete chaos and everything can do whatever it wants. It's probably possible and it probably even exist. I bet it exist already. But the most important part is that you would like to build your architecture as easy as possible because it allows you
[26:21] to have a simpler model in your observability. Because if your system is much more linear, you you need to monitor less complexity in your observability platform. And vice versa, the more like the more freedom your system has in
[26:36] terms of like how to interact between different agents, the more complex your observability should be to be able to handle all possible scenarios of where things go wrong. And the last part is basically like a circuit breaker. It's very interesting because it like
[26:53] potentially exist in a single agent system but it's much more complex much more common in a multi-agent system because sometimes you just need to be able to break a break a circuit. Essentially, what does it mean is that you have an extra agent that allows to stop
[27:11] execution of other agents. Example, you have a path like even if it's a linear path but you you see you monitor that your agent goes in the wrong direction. What do you want to do? Do you want it Do you want to allow it to go in that
[27:26] direction and see what will happen or do you want to break it at some point and basically ask your system to try a different route or try a different path to achieve the same results. This is um like intersection of observability and architecture.
[27:42] And essentially, if you remember one thing from my talk, I would like it to be this. Multi-agent systems are distributed distributed software systems first and agent systems second. For a single agent system, it's different. You talk about the lamps, you
[27:58] talk about um all possible cool stuff like the skills, tools, very impressive, very awesome, like absolutely fantastic. But if you move it to multi-agent system, those problems are already solved. And what you're
[28:13] facing right now is a classic distributed system problems. Which are complex by definition but essentially are um if you choose that route and it's probably also our future.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.