Multi-Agents in Production: How to Orchestrate Effective Agents

Summary

• Effective multi-agent networks: This video walks through techniques to build effective multi-agent networks in production, reducing error compounding.
• Engineering principles: Applying software engineering patterns like planning and task decomposition helps manage complexity across agents.
• Practical patterns: This video covers durable workflows, state management, and orchestration solutions for production-grade agent systems.

Multi-Agents in Production: How to Orchestrate Effective Agents

Talk by Tony Holdstock-Brown, ceo, Inngest
Agents solve complex problems, but building them can be hard. This talk will walk through techniques to build effective multi-agent networks in production deterministically, reducing the compounding error rates across each AI call.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to AI agents
• 2:53 Limitations of single models
• 4:41 Applying engineering principles
• 7:08 Cloud deployment challenges
• 9:40 Agent orchestration solutions
• 13:02 Implementing durable workflows
• 15:31 Conclusion and takeaways

FAQs

Why are multi-agent systems hard to build?

Multi-agent systems amplify errors—if each agent has 1% error rate, five agents in sequence compound to ~5% error. This video addresses error reduction techniques.

What is the planning approach?

Agents first create a plan and decompose tasks before execution, then pass each task to separate agents with proper state management for reliability.

What are the best tools for managing multi agent systems where agents can delegate tasks to other agents?

This video from Inngest discusses building effective multi-agent networks in production deterministically to reduce compounding error rates. The approach uses engineering principles for cloud deployment and orchestration solutions that enable agents to coordinate tasks while maintaining reliability at scale through proper error handling and state management.

What is durable workflow orchestration?

Durable workflows save state between steps, allowing recovery from failures and enabling complex multi-step operations to complete reliably.

How do agents use tools effectively?

Agents equipped with tools for reading, editing, and gathering context orchestrate these tools deterministically rather than randomly exploring options.

Full transcript

0:00 Introduction to AI agents

All right, good to go. Uh, hi, uh, the CEO of Inest and I'm here to talk about multi-agents in production. Um, how to orchestrate effective agents in your existing product using existing models without waiting for AGI, which is awesome. So, uh, essentially how you can build something that will solve really general problems uh, today. To kick it off, I don't know if you've seen this. This is one of the best talks ever.
Amazing talk by in 2013 talking about the future of programming. So, so good. Uh, he comes out dressing 70s gear. He's got like this overhead projector and he does the entire talk using slides. Um, and he goes over a crazy amount of history of CS between 50s and the 70s explosion in sort of the way that we build things. There's like data log, there's a whole bunch of like generative programming, there's flow, there's like flow-based programming.
One of the things that he talks about is like, hey, we've got constraint based programming in the 70s. Wouldn't it be crazy if in 40 years time we're still writing code and manually throwing together APIs? That would be wild. Um, and he specifically says it would be awesome and we're heading in this direction with law. You could say what you want and the system will figure it out because we have that right now in 1973.
And it turns out that we weren't there in 2013 when he gave the talk. Super forward thinking, but we're kind of almost thereish, which is awesome. Um, so I'll give an example now of a system that you've probably used or seen that is fairly general. You give it some input and it figures out what to do without you telling how to do it. Everyone has probably seen or used code. It's amazing. Um, the example on the right is me asking it to write some unit test for a package because I don't yet trust it for real code, but it does the thing.
Um, and how does it work? Well, agent with a bunch of tools really. Each agent has a whole bunch of tools for reading code, editing code, and getting context. And Claude first goes ahead and creates a plan for what you want to do. It doesn't actually just start yolo modifying code. It will read the code, create a plan, create a set of tasks, and then once it has that set of tasks, it passes each task off to a separate agent.
And the agent takes that task as context to figure out what which code it should write and then how to modify your system. So handoff, state, a whole bunch of stuff is required. Um, and all it does is execute this in a loop as you all probably know because you might have seen how to build agents or how to build AI. The interesting part is it does this directly on your own computer which is hard. I don't know how many people here are building desktop apps.
Cool. Yeah. None as expected. Uh that would be crazy especially at this conference. Um and so that's really tough. A question you might ask is why agents anyway? 03 yesterday was like the cost was reduced by 80%. Let's just use reasoning models. An analogy that I have in my mind is early versions of the web. So started coding in like what 2000 well like actually the 90s but for work in the 2000s and uh huge monolithic PHP files so good.

2:53 Limitations of single models

Uh don't know if you've any ever experienced this awful mess. You put your HTML your PHP directly together. Beautiful system. Um and it's a mess of spaghetti code. Uh it's really difficult to test a bug and fix. Uh and it doesn't scale well for complex products. So, uh, there's some similarities actually with the early versions of AI, it turns out. And with reasoning models, you get these huge monolithic prompts and you have a mess of instructions, context, and rules with the most important ones in all caps so that your your model knows exactly what to do.
Um, and it's difficult to test the bug off fix for general purpose use cases. You might have like evals that cover 80 or 100 cases. That's great. and then you get like a 101th case and then you realize that you need to change a prompt and it breaks 10 of the other cases you've got in your system because huge monolithic prompt in one reasoning model which is really tough and so agents in general help you scale for these complex problems uh probably seen windsurf this is windsurf prompts huge monolithic prompts don't even fix things if you promise to pay your agi billion dollars which is crazy this is literally their prompt uh or part of it so fun times agents are generally useful in helping solve complex problems foundationally.
Um, so we know how to fix this in code. I don't know if anyone's writing PHP anymore. If you are, I am very very sorry. Uh, and it's uh it's pretty easy. There's there's one quote. We've been learning it since like what the 2000s. This isn't meant to be going back to school. So I do apologize. single responsibility principle is analogous to where we're at. Um, 2000's solid. I don't know if you're all right in Java, but that was a big deal in Java.

4:41 Applying engineering principles

A class should only have one reason to change. And you can see a sort of similar thing with the early days of AI. How would this look if we took single responsibility principle in which you separate your billing off product packages um into agents realistically and I think Anthropic have talked through this a few times because they're they're really good at this. You um you split things up and then when you split things up things are easy to test, change, reason about and maintain.
So instead of having one monolithic prompt that does everything for you, you end up having a planning agent that has planning agent specific tools and then you run the planning agent to generate a plan and then you pass that off to a coding agent which has coding agent specific tools. Um really similar to what we were doing back in the day with regular code actually. It turns out specialized agents mean that you have specialized prompts that are really easy to test and eval independently of what you need to do.
So uh fairly analogous actually. um own tools, own prompts, own context. An example of this, if you've used FL code recently, I think this week or last, they released planning mode. It's very similar. You have a planning agent and it specifically plans the stuff you need to do into discrete tasks as talked about before it passes over to the other agents. So, if you're thinking about building agents, instead of having one massive prompt, break things down.
Pretty easy. um it leads to more effective solutions which is why claw code is better than a lot of other things. Cursor is great, wind surface is great, ADA is great, a lot of people say that claw code is the best. Um but it does require shared state and handoff and so going back to the whole context of building desktop applications a lot easier to do on one machine than it is to do on the cloud because on one machine you don't have distributed systems.
You can have literally a map if you wanted of state very easy in the cloud. Uh not not so fun. So Kubernetes pods with rolling deploys lovely. Uh agents might take what 20 minutes to finish. Deep research takes half an hour. That's pretty cool. But if you're rolling deploys and you do CD, great. You might have like 20 deploys a day. You don't necessarily want to wait for every request to finish for half an hour in order to do this sort of stuff.

7:08 Cloud deployment challenges

So challenging. You also don't want to guarantee that one single agent request is on one container anyway because containers die. Maybe it oomphs. Kubernetes is great at self-healing. Problem with running in the cloud is uh you know rolling deploys different machines, different containers. You also end up handling thousands of users or tens of thousands or hundreds of thousands at once, which is uh very different to clawed code running on your machine.
Uh unless you know you've been hacked and someone's doing some crazy stuff on your repo. This presents a big challenge because shared state, mutable state, many users. One user might end up sending if you're doing something like like day. AI has a CRM that's AI native. connect your account, connect your emails, and you might have to read through 100,000 emails and build up a graph of all of your connections for a CRM.
Another user might only have 10 emails that they received that day. And you don't want the 100,000 emails from a new user to outweigh the other 10 emails from another user in terms of capacity. So, it's not just like, oh, you have to manage hundreds of thousands of users, scale, scale your containers. is that like fundamentally AI capacity is constrained and you need to deal with those constraints at scale when you're building a real product.
Um that's this part imbalance load across users. No fun to deal with overall. So these are the sorts of challenges you'll end up facing. Observability is hard but like kind of a solved problem to a degree. Data dog is great. If you've got infinite money, use it. Yay. Um, and then you get the classic reliability issues of the cloud that we've dealt with for 20 25 years. Uh, networking rate limits, machine failures, all the stuff that we talked about.
Um, generally speaking, it would be nice if you could have the cursor for loop running over and over again, hitting a model, hitting a tool, updating state, evaluating that state to figure out, do we move on to the next agent? But foundationally that doesn't work when you're building your product in the cloud. Um it turns out that when you're building your product in the cloud a good mental model is to have uh layers of indirection which we've uh we've been using forever.

9:40 Agent orchestration solutions

Um good architecture. Your agents run they do some stuff and then underneath that you orchestrate them. Level of abstraction between the agents and the infrastructure. Relatively simple stuff. Uh the orchestration takes care of state as in this agent ran and it produced this output. Handles flu control as in this user can do 10 things at once so that you don't have 100,000 things from one user. Uh does all the reliability, the retries, that sort of stuff and observability.
I'll talk through this in a little bit more detail in a second. And the orchestration sits on top of your infrastructure and data. So it can pull out context from anything that you've got because it runs in your cloud. It's connected to your existing system. uh in general here's sort of how that works and I'll show you some example code in a second you can essentially think of orchestration Kubernetes does this for pods in this world you've all probably seen airflow daxter prefect that sort of stuff does it for your data pipelines same sort of stuff but for your application level code that runs your agents um step functions and each LLM call is a step that's that's basically good to go.
Um, each tool is also a step and you run them together in one workflow. Like honestly, pretty simple stuff. AWS Step Functions, as terrible of an as an experience it is, has existed since 2015. Something similar. I would not recommend using it. Please don't look at the API. It's awful. It's worse than PHP, actually. genuinely uh state management should be handled by your orchestration layer because when a step runs and it executes a tool if that tool returns results like for example I want to do a directory listing so I can understand what code exists uh for the coding agent the tool result should be cached in function state for that step function and it should run exactly once pretty basic and it should handle this reliably so as close to exactly once as possible because you don't want to create a PR twice um it should handle errors and retries because you will be rate limited and you might need to switch models.
Uh and then it should give you this sort of observability over one entire aerogenic flow because if you're doing an agent it is not as we discussed one massive prompt. It's a series of prompt calls or a series of tool calls and you want to see how those chain together and if you do it nicely and you split up each particular prompt or each agent into their own single responsibility actually becomes fairly easy to test.
Uh and then if you do this well and you use a decent orchestration platform, it should generate all the data sets that you need for evals and testing in the future because if you're writing your code in step functions, the orchestration platform knows your architecture. It knows the code and it knows which LLM you hit, the output that it was given, the prompt that it was given. It knows the tools that were hit. And then if you decide that step 10 of this particular thing was wrong, you have a data set and you can change the prompt for step 10.
Use the existing previous function run. Use that as an eval genuinely like easy, not too crazy orchestration as a separate layer instead of building your encuing systems and whatnot tends to solve a lot of these problems for building the cloud. Um, here is some example code. There are many platforms that do this for you actually. You can build a graph using langraph. Pretty cool. Uh you can also write step functions.

13:02 Implementing durable workflows

Uh this goes over the all I really want you to take away is the foundational issues and the and the patterns not necessarily this code. So I'll talk about the foundational patterns that this code introduces rather than the code itself. Uh steps as we talked about each step runs some code. That code could be pulling information for context from from your your data lakeink. It could be rag. It could be a model call or it could be a tool call.
It doesn't matter. A step is a step in a function that does some stuff and it runs once and the result of that stuff is stored in function state. You can chain them together. Type script is nowadays the lowest common denominator. So I figured you probably all understand the promise. which parallelizes two requests, does the thing um and then if these fail it will retry. So you get automatic state tracking, retries, parallelism, the whole durable stuff that you would expect and it's normal code.
The interesting and important part is and again the pattern to to think about is abstracting over your infrastructure so that you don't need to figure out how to chain one LLM call to one tool call to one another LLM call managing the state yourself. find something that abstracts the infrastructure such that you can just focus on a loop that calls the LLMs like that's that's basically the takeaway of the talk. Um no if you do that you won't need cues you won't need state you won't need a key value database you don't need to I don't know what people use nowadays foundation DB reddish whatever you use you don't need to deploy that because this should do it for you implicitly um and it should work with any SDK and any way of writing agents the important part is this doesn't specify and the orchestration that you use shouldn't specify how you build the agents themselves you should be able to run if you're interested in langraph or if interest interested in don't know how you pronounce it DSPY DSPI you should be able to use that too because those are pretty sick tools you still need to figure out how to solve the problems that we mentioned in the cloud which are durable state retries orchestration pod machine failures and in order to do that going back to 2015 the same principles that we already knew queuing systems distributed systems orchestration generally speaking solves that problem for you um if you do all of this and we take it to its logical conclusion.

15:31 Conclusion and takeaways

Basically the same slide just on the cloud. GitHub might send you a PR sorry you might create an issue on GitHub and then that sends you a web hook which contains information about that issue. You send that to the orchestration the orchestration platform which runs the entire flow as steps that does the thing reliably with all the flow control with all the state management and then you pass this off from the planning agent to the coding agent to eventually an SDLC agent that automatically understands the GitHub APIs using MCP or whatever fancy stuff you want to use and then it submits a PR automatically in the cloud without you having done the thing which is essentially and is basically or claw code with an orchestrator in the cloud abstracts away all of the complexity of running that stuff in the cloud.
The machines, the containers, the cues, the distributed state, the retries, you've basically built the same thing. Um, takeaways, we're not yet there with AGI, but people are building these products now. That's great, but it sort of lowers the moat. And it turns out that we also need to build these sorts of products in our companies which is kind of frustrating. So we have to build some stuff like this in general anyway and you have to think about state handoff reliability and all the distributed system stuff that you normally would and uh orchestration generally speaking helps you do that.

Learn more about the Databricks Data and AI platform.