Evolving Agent Complexity: Building Multi-Agent Systems With Mosaic AI

Summary

• Multi-agent architecture: This video explores building specialized agents versus single monolithic agents, examining the tradeoffs and challenges of complex multi-agent systems.
• Greenlight's AI Assistant: Built on the Mosaic AI Platform, this assistant powers Greenlight's family finance app which evolved from a debit card tool to a comprehensive financial safety platform.
• Production deployment patterns: This video covers vector search, model and feature serving endpoints, and evaluation frameworks needed to build, deploy and monitor AI agents at scale.

Evolving Agent Complexity: Building Multi-Agent Systems With Mosaic AI

Talk by Shanduojiao Jiang, Machine Learning Engineer, Greenlight Financial Technology ; Tim Mullins, Staff Machine Learning Engineer, Greenlight Financial Technology
this video dives into building multi-agent systems on the Mosaic AI Platform, exploring the techniques, architectures and lessons learned from experiences building Greenlight’s real-world agent applications. This presentation is well suited for executives, product managers and engineers alike, breaking down AI Agents into easy-to-understand concepts, while presenting an architecture for building complex systems. We’ll examine the core components of generative AI Agents and different ways to assemble them into agents, including different prompting and reasoning techniques. We’ll cover how the Mosaic AI Platform has enabled our small team to build, deploy and monitor our AI Agents, touching on vector search, feature and model serving endpoints, and the evaluation framework. Finally, we’ll discuss the pros and cons of building a multi-agent system consisting of specialized agents vs. a single large agent for Greenlight’s AI Assistant, and the challenges we encountered.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and context
• 3:20 Fundamentals of AI agents
• 7:08 Evolving agent complexity
• 13:12 Multi-agent architectures
• 16:42 Greenlight assistant design
• 21:57 Production, monitoring, and evaluation
• 25:54 Final architecture and outlook

FAQs

What is Greenlight and why does it need AI agents?

Greenlight is the number one family finance and safety app, evolved from a debit card for kids. It now offers diverse features including chores, allowance, investing, safety tracking, and family monitoring—all requiring intelligent AI assistance.

Should you build multiple specialized agents or one large agent?

This video examines the pros and cons of both approaches—multi-agent systems with specialized agents versus single large agents—based on Greenlight's real-world experiences and challenges.

How do you architect systems where multiple AI agents work together to solve complex problems?

Multi-agent systems require a coordinator that manages task delegation, monitors agent progress, and handles inter-agent communication within a shared context. The Databricks Data and AI platform provides the infrastructure to implement these patterns through MLflow for agent lifecycle management, real-time tracing to observe agent interactions, and unified storage through Delta Lake for shared state. This architecture enables organizations to decompose complex problems into specialized agents while maintaining observability and governance across the entire system.

How can you evaluate and continuously improve multi-agent system performance?

Multi-agent systems require evaluation frameworks that assess both individual agent behavior and emergent system properties like coordination efficiency and solution quality. Databricks MLflow provides tools to trace agent interactions, collect human feedback on outcomes, and compare performance across agent versions and configurations. This systematic approach enables teams to identify bottlenecks, optimize agent specialization, and validate improvements before deploying evolved systems to production.

What visibility and controls are needed to safely deploy autonomous multi-agent systems?

Autonomous multi-agent systems require comprehensive monitoring to detect anomalies, prevent unintended behavior, and maintain safety guardrails across all agent decisions. The Databricks Data and AI platform enables this through real-time tracing of all agent interactions, automated alerts when agent behavior deviates from expected patterns, and audit logs for compliance and incident investigation. Integration with governance tools ensures decisions made by agent systems are traceable and reversible if needed, providing the oversight necessary for production autonomous systems.

Full transcript

0:00 Introduction and context

Welcome everyone. Um, today we're going to talk about evolving agent complexity and we're going to start simple but work our way up there to build a multi- aent system today. So, first things first, this is our forward work uh looking statement and also don't forget to complete your survey after this session. So, my name is SJ. I am a machine learning engineer at Greenlight. I focus on the end-to-end machine learning and AI development all the way from model deployment all the way from model development to model deployment as well as integration to our back-end system so that we can um surface the model to our users and I also hold a master's degree in computer science from Stanford University and bachelor's in computer science from UFI.
I'm the uh staff machine learning engineer and manager at Greenlight of all of the team you see before you. Um yeah, work on all kinds of machine learning uh and genai work and previous to that did a lot of data engineering and traditional data science. Um and yeah uh graduate of Georgia Tech. Go Jackets. So let me tell you a little bit about Greenlight. One because it's a great product. to give you context for our talk.
Uh Greenlight started off as the debit card for kids. It's a way to give your kids money that's not cash because who uses cash anymore to teach them to spend and save wisely. But since then, it's evolved into the number one family finance and safety app. Uh you can set chores and allowance on it to teach your kids uh how to earn money, how to be responsible with money. Uh it has an invest section to teach kids how to learn about the stock market and learn to invest to grow long-term wealth.
Um we also have safety tracking uh where you can keep an eye on your kids. Uh driving scores built into that. Uh geo fencing, all of that good stuff. And more recently family uh a new product called Family Shield that lets you monitor other members uh of your family's account. So maybe you have a elder parent or uh uh an adult dependent that you need to monitor their monitor their finances as well. You can do that with a new product and you can just see uh a walkthrough of some of the features of the app in here.
All right, thank you so much for walking us through what Greenlight is. And as you can probably see, Greenlight is a very complicated product where a lot of users might have a lot of questions about our products and features and all of our um offering. So today we're going to build a multi- aent chatbot together. Um this chatbot will be able to handle all kind of questions including green light features and products, customer service and questions about investing concepts and um as well as spending and saving questions based on users um spending and saving data.
And finally we will have function calling which we will talk about um you know like one of the one of the um examples would be here is dispute a transaction based on um the user's spending data. All right. So here's today's agenda. So first of all we're going to give a quick recap over what is um an AI agent and then will will walk us through um the evolving complexity of AI agents. um starting from a really simple prompt agent all the way to one that's capable of handling um tool calls and everything else.

3:20 Fundamentals of AI agents

And then I'm going to talk about multi-agent architectures as well as greenlights multi- aent um assistant that you guys just saw earlier and then at the very end will put everything together into production and we'll um talk about evaluation and monitoring. All right, awesome. So what are AI agents? When we're talking about AI agents, it's very important to also talk about the difference between LLM and AI agents. So for example, AI agent um is a goal oriented system that is built around LLM.
When we're talking about chat GBT, for example, it's not an LLM. GBT is in fact an agent. So um when it comes to capacity LLMs are only language uh are only models that are trained on um uh large text corpora and it only has language understanding and generation. However, AI agents can have autonomous decision- making um and as well as for the memory and state. LMS are very stateless and has limited memory. While you can configure the AI agents to be holding long-term memory and doing long-term planning as well.
The TLDDR here is that LLMs are just brains. However, AI agents are the doers. They are built around LLM and they control them and you can add tools and memory and autonomy. Here are some of the very um common components that we will see when we're talking about AI agents. So obviously we have the brain here. Um some of the very popular brains these days are chap uh GPT41 claude etc. And obviously we can have prompts which are basically instructions to tell the brain how to um how to perform a certain task.
And also we have data whether this be a structured data or unstructured data. This kind of data will serve as the knowledge base or context for the L uh for the AI agent. And lastly we can have tools um which can be external or internal APIs. So here are um after we know all of the common components for AI agents um we still have a lot of decisions to make when it comes to the architecture for AI agents. There are so many questions we need to ask ourselves.
For example um what is the purpose of the agent? What is the end goal? And what are we trying to do here? And obviously what data does the agent need access to? Do it do we need to access structured data or unstructured data? And also do we need to access tools? Will this be external APIs or internal APIs? And we also need to think about if we need to make multiple LLM calls or just one LLM call. And obviously once we get the tool outputs, what are we doing with them?
And at the very end, is this agent iterative or will this be a linear agent? And of course there are very popular um very common tools that we're using for assembling um the AI agents these days. The most common ones are lane chain, lingraph and llama index for example. And so here will talk you um through all of the complexity we have for AI agents. Cool. Thanks SJ. Um yeah, like SJ said, we're going to start out really simple, but we'll get complex quick.
And this is actually the flow we went through as we started building AI, an AI agent or an AI product in our app. So step one is a very simple agent. This is what you would do in chat GPT with a custom prompt. You define the prompt, give it instructions how you want it to respond, what task you want it to do, and then you can reuse that without having to reprompt it every time. This is actually what we did. The very first idea of using uh AI in our app was to help kids understand investing better.

7:08 Evolving agent complexity

Help them uh convert investing terms uh and research stocks in terms they could understand. So it's a prompt like your helpful assistant helping kids and turn kids and teens learn about investing like don't use any financial jargon without explaining it and keep it relatable to them. So that was our very first agent a year and a half ago or more. It's called Invest Copilot. Um, it's since evolved beyond just simple prompts, but this is actually how it started out.
So, you can see what's EPS and we give them a kid-friendly uh answer to it. So, then you move on uh to needing access to other data. So, we want this in the app. We want kids to be able to invest, learn about investing with it, but we also need it to know about greenlight. So, that's where vector search comes in. Uh we have unstructured data public facing help docs um that we use the mosaic uh vector store and vector index to uh ingest so that we can do contextual search um so that now the chatbot can answer questions about how to invest in green light uh and I show all of this too to talk about the complexity um it's not just the contextual search and the vector store you have to have a pipeline for that so it kind of all ties back to data engineering feature engineering whatever you want to call it you have to have a pipeline to uh parse the data, embed it, and keep it up to date as well.
Um because we've changed, we've added new plans, we'll add new features. We want the chatbot, the AI assistant to know about all of those. Um so here's an example of that. Uh it's just uh referencing our public facing help docs and it could answer questions about uh Greenlight like about what features are in a plan or what the cost of a plan is. So that was our green light help bot which was sub agent number two of the green light assistant.
And then number three most organizations probably every organization has a wealth of structured data because that up until recently is how uh most data was processed. You know you got reporting you've got BI you've got tons of structured data. How can we also get that into the agent? So, we took advantage of the Mosaic feature store um and feature serving to serve up some of our structured data in particular, our users spend and save data to help them uh learn about their spending and saving habits and find out how much they've been spending, what where have they been spending the most, how can they be better about spending.
So, same thing here, you need a structured data pipeline, data pipeline or feature pipeline. Um we actually in the current version of the feature store we actually flatten this data uh into like a JSON representation per user or family and then retrieve it at the time of the query or at the time of the question to the agent and then the agent actually loads that into memory in a duct DB instance and write SQL against it.
Um so here's an example of that. Where did spend the most behind the scenes? It's retrieving the data from the feature serving endpoint very fast and then it actually writes a fast query or a query against duct DB which executes uh very fast very small amount of data since it's isolated to that user or family. And then level four or what we got to next was like how do we let users take actions inside G assistant if they are figuring out how to do something and they're like okay can you do that for me?
That's where tool usage or function calling comes in. So that can hook up to APIs internal or external, can hook up to web search, uh you can hook it up to really anything. Um we wrote our ouruh functions and tools uh explicitly wrote them ourselves for this case. Uh but now there's a wealth of things in Langchain uh and other libraries that you can plug into uh you know pre-built tools already. Uh so here's an example of that.
It's plugging into our green light uh chores micros service. So you can ask how many what chores does have for this week. It's going to hit the API, retrieve the chores, and then answer it. And then you can also create chores, check off chores, uh and schedule new like recurring chores as well. Um the thing we ran into that with that though we still haven't actually expanded that to a lot of green light functionality because it is a lot of uh writing custom code uh both uh you know the JSON representations of the tools and the code to execute the tools.
But then MCP comes along. So this is our next step. We've used it in some other apps but not in this one actually. But you know there's a lot of pre-built tools now. So you don't have to manage these all yourselves. Um, you can either use pre-built ones for external source sources or databases or you can write your own for your internal APIs once and then reuse them. So, that's actually what we're in the process of doing.
One of our engineers has already spun up a green light uh MCP server. Um, and we just need to fully test it out and get it all the way into production. So, this is what you end up with. Um, you can have an agent that has one or more of these components. It doesn't have to have all. Um, and then I'll say the only thing not shown in here is uh the type of reasoning or the type of uh prompting you give the agent. Uh, this shows kind of a linear flow, but it could be a react agent that hits multiple of these and then decides if it uh is ready to respond or it could be a uh, you know, linear decide which chain to go down and execute those actions.
Cool. Let talk about our multi- aent architectures. Yeah, thanks. Um, so as you can see there are so many different levels of difficulties when it comes to building AI agents and unfort well fortunately I guess um these days many system require more than just one single agent. So today we're going to talk about three core techniques that we're using for building multi- aent systems. The first thing we're going to talk about the router based agent and then we're going to talk about meta agent which is also known as agent of agents and then lastly we're we're going to go through the supervisor and uh reviewer agent.

13:12 Multi-agent architectures

So when it comes to routerbased agent, you can think of it as a two-layer architecture where the first layer is a router and then it will route um individual requests to specialized agents and based on the simple uh the classification rules that you specify in the router. And this kind of um routerbased agent is very linear. It's lightweight and handles onetoone task which makes it very simple and fast and also predictable.
However, the cons of this kind of architecture is that it's not very hard. It's pretty hard to scale with complexity and not super adaptive. So now we can look into um an example router prompt that we have um at green light. Basically, you're telling the prompt uh you're telling the router what it does and also gives it options to choose from. And one of the one of the use uh I guess um kind of like hacks we're doing is that we're basically sorry this clicker is not working.
Um basically adding few shot examples um to help with the router um classification performance. So now let's look at the agent of agents. Um agent of agents architecture can be viewed as more than just two layers. Um it can't it's basically a dynamic coordinator agent at the top of that router layer and it could route to still numerous specialized agents. However, it can also um select composes or spawns agents based on the context.
So this kind of multi- aent system can call multiple agents at once instead of just onetoone relationship compared to the router agent and um it's flexible and can handle really complex tasks. However, it can be way slower compared to the routerbased agent. And then we have supervisor and reviewer agent. and supervisor and reviewer agent will um review the answer that the specialized agents answer and then determines if we need to retry for a better performance.
So the supervisor the supervisor agent um can act as after the task execution. So it is another layer of quality control and this kind of a this kind of system improves the reliability and quality. However, it can also add latency because you don't exactly know um if you're going to trigger retries or not. So those were the three type of core techniques that we have for multi-agent systems. And now I'm going to walk over Greenlight's multi-agent system.

16:42 Greenlight assistant design

So here's the architecture that we have right now. already walked us through all of the individual agents um that we have in the earlier involving complexity slides. So obviously we have the function calling agent which is hooked up to our internal services and then it can handle um questions such as hey can you help me create a chore um for my kids or how many chores do my kids have and then we also have invest co-pilot which is hooked up to external APIs that can answer questions such as how much is Tesla today and um what is EPS and also we have customer support agent which is hooked up to our internal knowledge base and um this customer service agent will be able to answer questions um such as our green light products and features.
And lastly, we have the spensave insights agent which is hooked up to our structure data which will have all of the user spending and savings um data so that we can answer questions like hey how much have my kids spent in the past month? And here is another slide that basically shows an example of the green light assistant that we have using the multi um agent architecture that we just talked about. So right now we can see that it can answer questions related to Tesla stock and as well as greenlight's um features.
And this routerbased mod agent architecture um has the four agents that we showed and it also handles dispute um transactions. So I want to also talk about why we chose using the routerbased architecture instead of the other two. The first thing is that um we need expert sub agents to handle simple tasks instead of using a giga agent that can handle everything together. Um separating everything into individual agents makes it easier for us to build, tune, and debug.
Um and also we don't expect to handle really complex orchestration tasks because this is only a chat interface and we're not trying to do anything crazy. Um and very lastly low low latency is very important in a chat experience and we don't want the user to wait a long time for us to generate the answers back and also obviously um similar to every kind of development projects there are so many pain points that we went through and I want to highlight some of them here.
So the first thing is that it was really hard to ch well not really hard but it is definitely a chore for us to change the router prompt every single time when we want to add one or even more agents and also routers can really struggle with classifying a vague follow-up. Um, this can be this can be um I guess improved if you're if you're adding another layer of LLM inside that router to summarize all of the chat history and rephrase the question to make it a better follow-up question.
However, that also adds latency and also we encounter several challenges when we're trying to surface streaming from agents back to the router agent. And um so now that we kind of talked through the pain points that we have in the past, um we also want to look forward to see what we're trying to do in the future when it comes to green light assistant. So the first thing we want to do is to add a supervisor layer so that we can ensure the answer consistency across all of our specialized agents to make it kid-friendly and um easy to understand.
And also we want to add more customer service um actionable tool calling into our green light assistant so that we can handle more automations. And um we also um this is tied to the second point. We basically want to add more tools and APIs into this um green light system and also we don't want it to just be a simple chatbot that is so 2024 we want it to be um proactive so that we can generate insights for our users.
So for example, instead of the user having to actively in enter the chatbot and ask question, we generate spending and saving insights such as here on the left hand side, we're seeing um spent this much of uh this much less um that week um than the last week. And so we can generate this kind of insights by having a predefined set of questions sent over to our green light assistant and to display the answers into our assistant and as well as having an invest insight as well.
So instead of the users having to go into our chatbot asking questions about stocks, we can also have that kind of question already predefined and showing up in our invest page. Cool. So now we've got our assistant uh we've built it out, you know, in our notebooks. It's deployed uh in data bricks. Looks great. Even maybe we've even got it in integrated into the app at this point, but getting it into production and stable is, you know, that's half of the problem still.

21:57 Production, monitoring, and evaluation

So we have really, we're a small team, as I said. We've really relied heavily on data bricks, the data bricks platform and the mosaic AI tooling uh to help us do that. So inference tables uh if you've deployed any uh serving endpoint on data bricks you're probably aware of logs all of the inputs and outputs for you. Um so everything is there. We need it for compliance because we work with kids and families and uh financial data but we also use it for reporting.
So everything's already in the inference tables. we can quickly build dashboards to monitor and report on the usage. Um this is yeah one of the dashboards I look at you know almost every day to see how green light assistant is doing. Um and that's built directly on top of the inference tables. Um, MLflow tracing was huge. Uh, before this came along, the print statements everywhere that the print statements had to go and debugging an agent, especially a multi- uh multi- aent agent, uh, was very tedious.
But this has been, you know, a lifesaver. You can dig into every subprocess of your agent. You can see the inputs, outputs, and very importantly, you can see the latency. Where is it taking the longest? Because like SJ said, uh, we wanted this to be low latency. We knew the barrier people are going to compare any chatbot against is chat GBT. So we wanted to be as fast as possible and this helped uh a lot in debugging that.
But then how do we ensure that our agent does not go off the rails because like I said we work with kids. This chatbot is primarily for kids but also for their parents. The last thing we want is green light to be in the news because our chatbot tells a kid something it shouldn't. So we use the uh Mosaic AI offline evaluation. Uh it runs daily. We have a reference set that we continuously add to evaluate and see make sure our agent is responding consistently and safely.
Um we use some of the or we use the built-in judges. You can see relevance and safety are the built-in judges that uh Mosaic AI provides for us. And then this is a screenshot of two of our custom agent. One is to make sure the language is kid-friendly. You know, we don't want anything too complex when we're talking about uh financial concepts. And two, very important for compliance is we can't give anything that can be considered uh investment advice.
So that is one of the agents or custom LLM judges that we implemented as well. So like I said, this runs uh daily um and uh we get Slack notifications for it. So, here's a screenshot of the Slack notifications we get each day, but we also use the online monitoring in the same way we use the offline evaluation um to evaluate our uh live inputs as they come in. So, we're using the built-in judges there, safety and relevance, as well as our custom judges.
And uh this these some of these numbers are high. We really needed to t to tune the uh global guidelines here. But you can see it flagging uh kid friendliness on the left screenshot. Uh and then the global guidelines which is some of the uh relevance and uh uh relevance and like uh just the how how we expect uh the green light assistant to ask. That's one of the only issues we ran into with the built-in custom judges is they don't uh necessarily have the context of what your agent is supposed to be doing.
So we took advantage of the global guidelines uh and regular guidelines for uh more granular control over how we wanted the agent to respond. And then it basically looks like this. Our architecture looks like this. At the top we've got our agent workflow all in data bricks. We take advantage of the notebooks MLflow. We deploy on mosaic AI model serving and we log to Unity uh log to inference tables on our data pipelines structured and unstructured.

25:54 Final architecture and outlook

We pull data in from our data warehouse and other sources as well. Those notebooks are orchestrated in uh the workflows in data bricks uh and they write to delta tables uh in the feature store and we serve those via API and like I just went through on the right we take advantage basically of everything we can in data bricks uh agent evaluation lakehouse monitoring and uh dashboarding um and mosaic AI gateway has also been very powerful to us as well because we don't have to change any of the code when a new model comes out uh we use the models host hosted by uh data bricks, the cloud models, the ll ll ll ll ll ll ll ll ll ll ll llama models as well as openai and some models from Amazon Bedrock and that's how we built our production ready assistant.
Um skipped ahead a little bit but we are hiring. We're a small team but we are looking to expand. Uh this this presentation has mostly been about our uh userfacing chatbot but we're working on a handful of other projects. can't even read these. Uh we have AI video generation for stocks to help kids learn about uh specific stocks and companies. Uh we're doing personalized insights which SJ showed for spending and saving and investing.
Uh we recently implemented uh suspicious transaction monitoring uh using AI. Um SJ's been focusing a lot on customer service automation. That's a big cost center. So we believe we can bring down cost uh through a through AI. And we get kind of bombarded all the time with internal requests for, you know, uh, rag chat bots or or specialized assistance from accounting, marketing. Um, so like I said, we're a small team, but we're we're looking uh to hire if you're interested.
Happy to take questions, but also happy to uh connect as well. Thanks. Thank you.

Learn more about the Databricks Data and AI platform.