Designing Agentic Systems Reliable Enough for Production

Summary

This video argues that agentic system reliability is an engineering problem, not a prompting problem—and demonstrates that adding proper control flow and a plan-execute-observe-replan design can raise a benchmark score from 20% to over 60%.
CrewAI's production architecture uses flows—event-based orchestration layers that combine deterministic nodes for programmatic tasks with intelligent nodes that apply LLM reasoning only where it genuinely adds value.
A pre-deploy checklist covers the essential components every production agent needs, including guardrails, hooks, human-in-the-loop checkpoints, and state checkpointing—components that most teams building their first agents forget to include.

Designing Agentic Systems Reliable Enough for Production

Most agents don't fail in production because the model is bad. They fail because the system around the model has no guardrails, no structure, and no way to see what's happening. In this video I'll make the case that reliability is an engineering problem, not a prompting one. I'll walk through the harness every production agent needs: control flow, a real plan, execute, observe, and replan design, and observability you can act on. I'll back it with production failures we've all seen and a benchmark where the same model went from 20% to 60%+ just by changing the system. You'll leave with a pre-deploy checklist: the blocks to include, and the ones most teams forget.
Talk By: Lorenze Jay Hernandez, OSS Lead Software Engineer, CrewAI ;

Chapters

FAQs

Why do most AI agents fail in production and what is the real cause?

Most agents fail not because the underlying model is poor but because the system around the model lacks guardrails, structure, and observability. This video makes the case that reliability is an engineering problem: the same model can go from 20% to over 60% benchmark performance simply by changing the surrounding system design.

What are CrewAI flows and why do production customers use them?

CrewAI flows are event-based orchestration layers that form the core production architecture shipped by CrewAI customers including PwC, DocuSign, and AB InBev. They combine deterministic nodes for programmatic tasks—such as fetching data from a CRM—with intelligent nodes where LLM reasoning is applied, keeping token usage focused on steps where it genuinely matters.

What is the difference between a deterministic node and an intelligence node in an agentic flow?

A deterministic node performs a programmatic operation like a Python function call or an API request—tasks that have been done reliably for decades and do not need LLM reasoning. An intelligence node applies an LLM to evaluate, score, or make a decision at a step where the range of valid outputs is too broad to handle programmatically.

What components should every production-ready agentic system include?

According to this video, production agents need a real plan-execute-observe-replan control flow, guardrails to prevent harmful actions, hooks for extensibility, human-in-the-loop approval for high-stakes decisions, and checkpointing so work is not lost if a step fails. Most teams building their first agents forget at least one of these elements.

Full transcript

[00:07] Pretty good. Pretty good. I think this is the last session for Data Bricks. So, thank you guys for coming. I know it's been a couple of good fun days um here today. But before we get started, my name is Lorenz. I lead our
[00:19] open source team at Crew AI. Uh but before we begin, I just want a raise of hands. How many people here have actually built agents? Raise your hands. Okay, most of the room. Keep your hands up if you've actually deployed that into
[00:33] production environments. Okay, half the room. What about scale that? Like tens of thousands of people have actually used the these agents? No one's hands up, really? Okay. Hopefully at the end of today, you're
[00:45] going to learn something about how you can actually do that reliably. So, my promise here is simple. I want you guys to just answer this question at the end of the topic. You'll have a framework that you could bring to your
[00:57] team or to yourself if you're building them to answer, "Is our agentic system actually production ready?" Not just something that works in a demo,
[01:09] not something that just works in my own machine, but something that you could actually bring and scale to a bunch of real users. At Crew AI, we've worked with many organizations from PwC, DocuSign, AB
[01:23] InBev, and many others. There are different industries, there are different use cases, they bring in different models.
[01:36] But all of our customers have this one thing in common. And that's that they ship something called flows. This core architecture of what they actually deploy is this orchestration like event-based
[01:48] orchestration layer. All it is, it's deterministic. It has start methods, it has listeners, and it has routers. Think of this as like a traditional workflow automation tool. However, you could also bring it some
[02:00] intelligence. You can sprinkle those in in between each node where intelligence or reasoning might matter. So, what do you have? Let me give you guys a
[02:12] quick example. Think of a lead qualification use case. I want to first pull data. Could be from HubSpot, it could be from like Data Bricks. This is a fetch request. This is not something that needs to burn tokens. This is
[02:24] something programmatic, something with Python. This is something we've been doing for many decades now. The ones in the green are highlighted are entirely programmatic. So, your pull
[02:36] data, I'm going to fetch data from HubSpot. I have this row now. Now I can score it. This is where an LM can come into play. Based off my company's qualifications of what a good lead is, we can use an LM to actually score what
[02:49] that looks like. Uh from there, we could have this router. It's a conditional. So, if lead score is greater than seven or less than seven, let's just give it a generic answer. This is programmatic, we could
[03:02] do a request post request, and we'll either do nothing with that or go back to our database and just flag it as not a lead. Or if it's greater than seven, let's do more research about them. Let's query
[03:15] more results from a database. Let's analyze if this profile is genuinely a lead or not, then create this highly custom campaign just for that uh potential lead, right?
[03:27] And in the red, these are where you have your levels of sprinkled intelligence or agency, right? Your first agent scores an LM, or your analysis now is like multi-agents. So, you have three agents
[03:39] potentially running more research about that user. But overall, this structure is what we call flows. So, what's the takeaway here? Like what do I If there's one thing that I want to
[03:52] go home with, this is the takeaway. Reliable agentic systems balance both determinism and agency. Agency is like intelligence, reasoning, things like that.
[04:04] The goal here isn't to eliminate agents. Agents are actually powerful and useful. But the goal is to really be intentional in where we place them, right? In each node or each step of the way, where do
[04:16] we actually want these intelligent intelligent models to perform from? So, how do we put all of these together, right? How do we design these reliable
[04:29] agentic systems? I think there's three pillars here. One is control flows. You have guardrails, hooks, human in the loop, checkpointing. This is how you constrain and control your execution execution
[04:43] flow. Then there comes the agent pattern, the design of how you actually build an agent. People think of it this as a loop, but agents also require some structure. It could be done with planning and we'll
[04:55] get to those details over there, but structure and structured outputs is pretty much all you need to know when it comes to more reliable agentic patterns. And I'm going to

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.