How to Improve Quality of Multi-Agent Systems with Agent Bricks

Summary

Multi-agent systems: Agent Bricks enables building, evaluating and deploying multi-agent systems combining Genie, RAG chatbots and MCP servers.
Natural language development: Use natural language feedback and built-in evaluation metrics to continuously improve multi-agent system output quality without code.
Agent patterns: Information extraction, knowledge assistants, Genie for structured data, multi-agent supervisors and custom agents cover key use cases.

How to Improve Quality of Multi-Agent Systems with Agent Bricks

In this video, you will learn how to build, evaluate, and deploy multi-agent systems on Databricks using only natural language. This video demonstrates creating a "Novel Ideas Bookworm" multi-agent system, which combines an AI Genie (for structured data/inventory), a RAG chatbot (for book recommendations/knowledge), and an MCP server (for general web search) to provide intelligent bookstore assistance. See how to continuously improve the multi-agent system's output quality by using natural language feedback from labeling sessions and built-in evaluation metrics.
🎯 What You'll Learn:
Key Agent Use Cases: Information Extraction, Knowledge Assistant (RAG chatbot), AI/ML Genie for structured data, Multi-Agent Supervisor (RAG + Genie), Custom LLM, and fully custom agents.
Build Multi-Agent Systems: Use Databricks to build and improve quality on multi-agent systems using SME feedback and natural language.
Evaluate with Traces and Custom Evals: Use the "Experiment" tab for detailed interaction traces and the "Evaluations" tab for built-in metrics (e.g., correctness, relevance) and custom evaluation prompts.
⏱️ CHAPTERS:
📚 Resources:
• GitHub repo - https://github.com/databricks
• Documentation:
Agent Bricks - https://www.databricks.com/product/artificial-intelligence/agent-bricks
AWS - https://docs.databricks.com/aws/en/generative-ai/agent-bricks
Azure - https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/

Chapters

FAQs

What is Agent Bricks?

Agent Bricks is a framework for building, evaluating and deploying multi-agent systems on Databricks using natural language. This video demonstrates creating a Novel Ideas Bookworm system combining AI Genie for structured data, a RAG chatbot for knowledge retrieval, and MCP servers for external search capabilities.

What are the key agent use cases?

Agent Bricks supports information extraction, knowledge assistants (RAG chatbots), AI/BI Genie for structured data queries, multi-agent supervisors that orchestrate multiple agents, custom LLM-based agents, and fully custom agents using external frameworks like Langchain.

What are the key characteristics of multi-agent systems?

This video introduces Agent Bricks and their use in building multi-agent systems on the Databricks Data and AI platform. Key characteristics include independent agents with specialized roles, communication protocols for agent coordination, shared access to tools and data, and monitoring to track system behavior. The video demonstrates how these components work together in production systems.

How do I build a custom multi-agent system?

This video shows how to use Agent Bricks patterns and templates on the Databricks Data and AI platform to build custom multi-agent systems. The framework provides pre-built components for agent scaffolding, tool orchestration, state management, and error handling. By combining these building blocks, you can create reliable multi-agent workflows tailored to your specific business requirements.

What are the best agent harnesses for multi-agent systems?

This video demonstrates Agent Bricks as a production-ready harness for multi-agent systems on the Databricks Data and AI platform. Agent Bricks provides the infrastructure for managing agent lifecycles, coordinating between agents, handling failures, and monitoring system health. The framework abstracts complexity while allowing customization for domain-specific requirements.

Full transcript

`0:00` Agent Bricks Overview – Build, evaluate, and deploy multi-agent systems on Databricks with natural language.

Hi everyone, I've been building with AI and ML systems for about a decade now and it's easier than it's ever been before. I'm excited to show you how you can build, evaluate and deploy multi- aent systems on data bicks using only natural language. All right, let's get started. Now the agents that you can build within data bicks and these are in the agents tab are mostly around key use cases. So this is information extraction.
So the ability to extract key information and insights into structured JSON formats and you can actually use PDFs directly. It will use our AI parse document and create a workflow. So you can upload PDFs as well as other unstructured documents. Knowledge assistant. This is our rag chatbot. So this is the ability to just have a Q&A over your documents. AIB genie. So if you don't have unstructured documents and you actually have structured data in tables, you're able to use AIB genie to ask questions and get insights.
And then maybe if you want to combine your retrieval augmented generation chatbot with your AI genie. So combining all your unstructured and structured data and ask questions over everything. You can create a multi-agent supervisor. This could also be combined with MCP servers and additional tool calling. Custom LM, this is your content or code generation specializes to perform these custom generation tasks and then code your own agent.
So this is the ability to bring in other open- source libraries, bring in agent frameworks like Langchain, OpenAI SDK and start building your own agents from scratch within data bricks. Now, in order to break down the multi- aent supervisor, let's see what agents are actually powering that behind the scenes. Starting with the genie space. So, I'm going to go to my sidebar and go into the genie tab. Here, I can see my novel ideas inventory space.
And for the sake of this demo, we are saying that we are in the small independent bookstore space. Our bookstore is called Novel Ideas and we're really able to compete against larger bookstores because we have incredible customer service. We have an amazing community and we're able to automate a lot of these processes that tend to slow down larger companies. So, this inventory space is the Genie space that delivers intelligent bookstore assistance.
It integrates with real-time inventory. It has aisle locations, product catalog, sales data, supplier information, stock updates, and if you want to add any additional tables in my configure tab, you could see where I have all the data linked. All right. Now, I have some example questions available. So, let's click one. Let's see what are the total sales for each title in the sales velocity table. So, it's going through the sales velocity table, seeing the title, seeing the amount of sales, and then it also throws in a histogram.

`2:53` Architecture: Vibe-Coding Stack – Overview of the Novel Ideas multi-agent system

Maybe I want to see this information as a pie chart. And there I can see the information as a pie chart for the total sales distribution by title. Now I can add in different categories. I can add in different years of sales. I could add in different locations if I have different locations for the novel ideas bookstore and be able to just interact and have that information available. All right. Next, I'm going to go into our knowledge assistant.
So, going back to my agents tab, going into my bookstock bot. So, the bookstock bot again, this is your knowledge assistant rag chatbot retrieval augmented generation. This is going to combine expertise from bookstore stack with datadriven insights. So if a customer asks for a book recommendation, it's not just an LLM pinging the internet to see what is a logical next book choice. It's using domain specific expertise, curated staff recommendations, trending titles, new releases, that customer purchase history and book metadata.

`4:00` Knowledge Assistant (Bookstock Bot) – RAG chatbot for personalized book recommendations.

Um that's all stored in unique catalog volumes and vector search. just pulling from that information in order to give tailored recommendation which helps make novel ideas so successful. So the current knowledge sources I have available. So I have a few volumes. So I have a volume that has all the staff recommendations. So featuring personal reactions, reasons to read, suggestions for similar titles, really helping discover new books.
We might also have paid book reviewers that are subject matter experts that are leaving reviews. Annotations. So these are key notes and they're all stored in a vector search index. And then I have another volume for recent releases. So this volume contains the most anticipated books of 2025 as well as trending books. So these could be on book talk, these could be national bestsellers and kind of combining those lists.
Now, if you already have a vector search index built, you can plug that directly into your knowledge assistant and use that information. Otherwise, if you use a volume, uh, this vector search index is already created in the background. Now, I'm not going to go into testing right now because I built a multi-agent system that leverages both of these. So, going back, going back now, I have my novel ideas bookworm. So, I'm going to click into this bookworm.

`5:50` Multi-Agent Supervisor (Bookworm) – Orchestrates Bookstock Bot, Inventory Space, and Tavly MCP for broader queries.

And the novel Ideas Bookworm enhances bookstore operations and customer service by providing employees with instant inventory tracking and intelligent recommendation tools. This helps quickly assist customers with up-to-date stock information and tailored book suggestions. And what's powering this multi- aent system? Even though I can have 20 different agents currently contributing and powering this multi- aent system, I have the bookstock bot.
Again, that's our knowledge assistant react chatbot and then our inventory space. So, this is our genie inventory space as well as an MCP server. So, I have the Tavali MCP server. Tavali helps enable LLM based assistants like Claude to connect to real-time web tools. So, for search extraction, mapping and crawling. Now, I want to make sure that the novel ideas bookworm isn't overutilizing the MCP server. So, I have an optional here. only use the Tavi MCP server if the question is broad and we don't have the answer in our novel ideas reference material.
All right, so now getting into a question I want to ask the novel ideas bookworm. So if I work at the Novel Ideas bookstore, I probably have customers coming up to me all the time that are looking for instock book recommendations. So if I say a customer just finished and loved it, which instock title should I reserve for her next? What we would expect to happen here is that the novel ideas bookworm is first going to ping the books stopbot, which it looks like it's already doing because this is going to give the tailored recommendations.
And then once it has those material, it's probably going to do a reference check to make sure that those titles are in stock. So let's see what it's come up with. Right. So novel ideas bookworm handed off to the bookstock bot. Here we can view thoughts. These can get kind of messy because these are each individual process that the agent's taking to find relevant book recommendations. But you can see it's going through book reviews.
It's verifying knowledge sources and then it's able to figure out top recommendations. So we're seeing things like Janeire for, My Cousin Rachel, also by. And let's just scroll down. So, it picked the relevant books and now it's pinging the inventory agent. So, that's our Genie space, seeing what's in stock. So, seeing some top book recommendations, seeing what's currently in stock, and then it gave top three recommendations.
So, Mexican Gothic, My Cousin Rachel, Jane Air, giving a few alternative options as well. And so, now let's look into improve quality. And I've already started a labeling session. So the labeling sessions are you can create a golden data set essentially with a variety of questions, guidelines, and send these out to subject matter experts. So if we have folks in the book community, people who are paid book reviewers, those in book club, those that are in our community staff, I can grant them permissions, send out this review set, and collect feedback from them.

`8:51` Improving Quality – Use “Improve Quality” to label sessions, add expectations, and merge expert feedback.

I just have to open a labeling session. Now you can see in this labeling session we have the question I just asked the final results and the final results may vary because these are non-deterministic systems. So the exact answer I got when I first asked the question is probably not the exact same answer I'm getting now. It's why we collect all this information in aggregate and use that feedback to continuously improve our model and improve our outputs.
So going into expectations, recommend no more than three books, don't recommend the sequel if they haven't read the original. And as you saw from the result, there were no more than three books recommended. And so those were the guidelines I already had in place. Now I might decide, hey, I really think you should add in the aisle location to the books. So, I'm just going to add in add the aisle location with the books.
So, if anything's recommended, you know, not just that it's in stock, but you know, the aisle in which it's located. Save that information. And then for feedback, I really like the books that it recommended. So, if we scroll down, I'm seeing Mexican Gothic, Jane Air, My Cousin Rachel, but maybe I'd also recommend, you know, Weathering Heights should be in the top three. So, the feedback I would give is these are great wrecks, but I would also recommend Weathering Heights.
Save that feedback. And then when I'm back in my improve quality tab, all I have to do is merge that feedback. Right. So now we can see the question, the new guidelines, the feedback records and as you collect more and more of this feedback, you can improve the quality of the output from your agent. Now let's see on the configuration. So going back to configure, clearing the chat. Now, if I'm an employee at the Novel Ideas Bookstore, I want to make sure that I'm giving good recommendations to my customers and making sure they're leaving satisfied and with a good book.
Now, I could just reask the question, but I'm curious if I change the book, what would happen? Because we just had a subject matter expert say, "If you liked you should also like Weathering Heights." So I wonder if we change the book title to Weathering Heights if we would get the same expectation. So as you would expect the novel ideas bookworm would first call the bookstock bot to get the recommendations which it does and then we use the inventory space.
So our genie space to check that it is in stock and including the aisle locations now which we see the in stock number of books and the aisle location. And so we say perfect here are all three uh recommendations and as expected you know is in there. So this is a perfect iteration of using the improved quality tab, leveraging that information and being able to use the different expectations to make sure that the responses are consistent.

`12:20` Wrap-Up & Next Steps

Now you might have noticed we haven't actually used our MCP server. For a question like that, you it would be nothing domain specific, but maybe a customer would ask, is there a movie adaptation of Weathering Heights? So, is there a movie adaptation of Weathering Heights? Now, I know for a fact that this information isn't in our genie space. It isn't in our knowledge assistant. So, I would expect this is the type of question to ping the Tavly MCP server.

`12:47` Using the MCP Server – Routes general questions to Tavly MCP when not in references.

As we can see, it's already pinging the Tavly search. Now, we can just look through the retrieved JSON files from the Tavly search and then go into the final response. So, the notable film adaptations and we even know now that there's an upcoming 2026 Weathering Heights that features. Now, if you wanted to dive deeper into your multi- aent supervisor and continue to improve it using evaluations, you can go into the experiment tab.

`13:42` Evaluating Performance – Use “Experiment” for traces, “Scores” for custom evals, and “Evaluations” for metrics like correctness and relevance.

It takes you to your traces. So every interaction with your multi- aent supervisor, so is there a movie adaptation of weathering heights shows you what tools were used, how it was called, and every span that went into that trace. We also have capabilities like our scores. These are our custom evals prompts. So prompt storage, version control, labeling, and then into evaluations. In our evaluations tab, you can see the runs that we have done for evaluation metrics.
These evaluation metrics are such as correctness, relevance, retrieve groundedness, and we have a large variety of built-in evaluation metrics. But again, we also have customized scores as well. You can look at different runs to really see how your performance has changed over time. If I want to do a comparison to see aggregate how these eval started to improve, I can do direct comparisons. And I can even look into individual evaluation runs and see how they have changed.
So, what book should I recommend to readers who liked Mexican Gothic? For this, you might have an expected outcome like I probably expect these outcomes. Janeire, My Cousin Weathering Heights, and. And then you can see the what the model output is. And here our model output suggested Frankenstein by. Now, that's actually a very reasonable recommendation. It goes into why this is a perfect match and who on the staff recommended it.
So then you can change your expected output to then include Frankenstein. So you now have resolved that evaluation. So constantly monitoring your evals, having them evaluated in aggregate helps you understand how your model is performing and what you can do to improve the performance of your multi-agent system. Thanks everyone for watching. As you can see, in just 15 minutes, we were able to build and evaluate a multi- aent system using just natural language and data bricks.

`15:55` Conclusion – Build and evaluate a multi-agent system in minutes with natural language on Databricks.

This system combined a genie space, a knowledge assistant rag chatbot, and an MCP server in order to help improve customer experience. If you're looking to build these agents for yourself, check out the links below for additional educational material, courses, and a link for how you can sign up for your free edition of Data Bricks today. Thanks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.