Skip to main content

Accelerate End-to-End Multi-Agents on Databricks and DSPy

Summary

  • End-to-end workflows: Production GenAI applications require more than the agent framework—they need data ETL, governance, versioning, evaluation, and hosting infrastructure.
  • Multi-agent orchestration: DSPy enables multiple agents to interact and reason together, ensuring questions are answered correctly even on specialized topics beyond the base LLM.
  • Practical design: Smaller models and classical machine learning remain effective; agents serve as 10x assistants for solving complex problems without frontier models.

Accelerate End-to-End Multi-Agents on Databricks and DSPy

Watch: Accelerate End-to-End Multi-Agents on Databricks and DSPy
Talk by Austin Choi, Delivery Solutions Architect, Databricks
A production-ready GenAI application is more than the framework itself. Like ML, you need a unified platform to create an end-to-end workflow for production quality applications. Below is an example of how this works on Databricks: Data ETL with DLT and jobs Data storage for governance and access with Unity Catalog Code development with Notebooks Agent versioning and metric tracking with MLflow and Unity Catalog Evaluation and optimizations with Mosaic AI Agent Framework and DSPy Hosting infrastructure with monitoring with Model Serving and AI Gateway Front-end apps using Databricks Apps In this video, learn how to build agents to access all your data and models through function calling. Then, learn how DSPy enables agent interaction with each other to ensure the question is answered correctly. We will demonstrate a chatbot, powered by multiple agents, to be able to answer questions and reason answers the base LLM does not know and very specialized topics.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What makes a production-ready GenAI application?

Production GenAI apps require more than an agent framework; they need data pipelines with DLT, governed data storage with Unity Catalog, versioning with MLflow, and monitoring with Model Serving.

How does DSPy improve multi-agent systems?

DSPy enables agents to interact with each other, coordinating their reasoning to answer questions correctly and solve specialized topics that a single LLM cannot handle alone.

Can AI agents build machine learning models end to end?

This video demonstrates that multi-agent systems on Databricks can orchestrate end-to-end ML workflows—from data preparation through model training and deployment. Agents can autonomously make decisions about algorithms, parameters, and infrastructure choices within defined guardrails.

What infrastructure do multi-agents need on Databricks?

Multi-agent systems leverage Databricks' unified compute and storage for reliable tool execution. Agents access the Databricks Data and AI platform's APIs for data access, model management, and orchestration, enabling end-to-end ML automation.

What role does DSPy play in enterprise multi-agent deployment?

DSPy provides production patterns for agent development including structured outputs, tool integration, and error handling. On Databricks, these patterns scale to handle enterprise workloads while maintaining governance and auditability.

Full transcript

0:00 Introduction to agents

All right. Hello everyone. Welcome to this session on multi- aents with DSPI and data bricks. Uh my I. But before I actually go into that some forward-looking statements and as well as you know do your surveys. Um it always looks good to see how you felt about the session itself. So kind of try and go a little slowly so the people wait listed can come in. But a little bit about myself. I'm a delivery solutions architect here at data bricks.
I live in Los Angeles and I have a huge passion for kind of machine learning, data science, generated AI, of course. And on the personal side, I'm a big swimmer and I play a lot of video games. I'm sure that one game you see in that slide is very familiar to some of you. And these are my two beautiful cats. So all that to say, you're here today to learn about how you can build multi- aents with ESPI on data bricks. And this actually originated from a Medium article that I had written um a few months back to kind of articulate the power of using your agents to not just use generative AI resources, but classical machine learning, raw Python, kind of whatever you need.
So if you want to kind of see that example here, it actually goes through like a Pokemon agent as well to really highlight the fact that it's um using uh different tools. So this is the original Medium article, but we've revamped it since. So the first thing is we need to level set what exactly is agent or even a multi- aent. And I'm sure there's many people in the crowd that are like okay agent is just a glorified way of orchestrating prompts.

1:36 Defining AI agents

Um it could be it could even be just a rag agent. I know some of you have been saying rag agent but you know it some people's definitions differ and sometimes even in our official documentation we say the industry is still defining AI agents. This is a very fast changing landscape and I think as long as you're able to use these tools to accomplish your use case or solution, I think that's what's most important rather than debating on what makes something multi- aent or agent.
So all that to say agents don't need to be complicated. um what we've learned and then some of you have may have seen the anthropic art uh paper that came out that talked about their experience talking to their customers and uh trying to build out their own uh agent systems. And what we've learned is that smaller models and classical machine learning still re supreme. Why is that? The powerful large language models are very expensive.
Frontier models are very expensive. um hosting anything higher than like a 70 billion parameter large language model is pretty expensive at least until uh we get more GPUs which is still a problem um this is not going to get cheaper and these are good alternatives to accomplish your use case um number two don't make it too complicated just because you have more than don't like force the issue to have more than one agent in order to accomplish your use case that just that creates a larger black box for you that makes it even more difficult to um debug and iterate over.
And number three, genai alone is just too inconsistent when you integrate things like function calling and other uh tools and just good Python code. Um you'll find that you will fi get better results than just using genai by itself. And that kind of leads into four and five where especially five the code itself is very difficult to maintain. I Don't know how many of you, but I'm sure most of you do not like writing large blocks of text.
I did not like doing that for my college essays. That was a very miserable experience, but I don't know why we're doing that today. We're writing gigantic prompts to accomplish what we want out of our large language models. And some people are kind of like breaking it apart and trying to use it like more modularly. But ultimately, there's still a lot of people out there who are making like paragraphs, pages and pages of prompts just to force like GBT 4.
1 to do exactly what you want and multiple things at once. So, Faxstead actually ran into this problem and some of you in other presentations may have seen this where they were trying to you trying to have GPT4 do everything and they did not get the accuracy that they were looking for. As you can see in the slides, it was only 59% and the latency was actually 15 seconds. But when they transitioned it to an agent system that was broken up into multiple agents, which again totally depends on how you define agents, um they were able to increase their accuracy from 59% to 85%.

4:32 FactSet case study

And then the latency from 15 seconds to 6 seconds. Now you're wondering there's four more steps to this. How did they drop it from 15 seconds to 6 seconds? Well, larger large language, very large large language models take a much longer time to run inference on. And if you don't have the latest and greatest GPUs, it's only going to take even longer. So, and plus classical machine learning models like a CNN is way smaller than the billion parameters that these uh large language models require.
And so because of that, you're able to get much faster latency, much faster responses by actually breaking it up into these smaller pieces. Also, the each task is much simpler. So you don't have to wait for this model to go through. It's a reasoning task which will take longer in order to finish uh answering or completing their request. So I bring fact set up because they chose data bricks to build this agent system on.
And this is not theirs, but this is roughly what you kind of need to build like a production ready uh agent system, right? I'm not expecting you to like look at this and be like, "Oh, I got it." Um, but I am here to pitch data bricks, of course, and this is all available on data bricks. The hosting, the compute, uh, where to put your code and all that, all that good stuff. So don't worry too much on this, but that you do need to you should pick a platform that kind of has everything you need in one place and you don't have scattered tools kind of everywhere.
So with all that to say, let's try and get the most out of your system here. So tip number one, like I've been mentioning this entire time, use classical machine learning. Um if you're intimidated with machine learning, don't. There's so many good open source models out there, especially on hugging face that you can take advantage of. We're actually going to use one for our demo later after this slide. Um, but just take a look at a very simple example, right?

6:39 Multi-agent tips and tricks

Image classification. Like there's so many good models out there that can do this like instantly, right? But if you send it to a multimodal model, depending on the size of your image, the quality of your image, it it's going to be expensive. A dollar for 20 requests is not too bad. may take a few seconds, but when you compare it to hosting a vision transformer model where it can do that um classification in like half a second and it costs only 70 cents an hour to host this model and you can do hundreds and thousands of requests within that hour.
It's a no-brainer. That's so much cheaper and you can run through so many more images by doing by hosting a classical machine learning model. Number two, and this also relates to tip number one, fine-tune. Fine-tune your uh language models, fine-tune your models to exactly your use case. It's a lot more powerful, especially with a smaller language model when it know when it's fine-tuned on your context and on your data.
And oftent times we are starting to see smaller language models used in these agent systems because they're because they're able to handle the simpler tasks like text extraction or summarization much just as well as a 70 billion model. You don't need to use a 70 billion model. You don't need to use the trillion billion. I don't think there's one, but you don't need to use the trillion billion model for u something so simple.
And so fine-tuning it only gets you the per a higher level of performance from the um smaller language model. Essentially making up the performance gap between the bigger ones and the smaller one. Tip number three is use function calling to hit different services. Uh I'm sure by now you all know well actually I'm not going to make that assumption. Function calling is very important. You should use function calling to um hit any other data sources out there via API, via just Python.
This could it can be like a Python calculator function that you call to do math so that the large language model itself is not doing that. Um there's many many different things that you can do for function calling. Um I'm not going to go deep into this. But for this example on the screen you can see there's an agent that kind of triages it to the right function gets an output take takes that into consideration and then gives that out to the user.
So now it's actually becoming data intelligent and able to respond to your queries based on valuable data that you likely have. And tip four again the pitch data bricks one place data bicks has a place to do your model evaluation uh to build your models to serve your applications to make vector search to do your rag applications serve that data serve those vectors uh do governance on unity catalog there's a whole data platform with delta tables iceberg all that cool stuff and finally mlops llmops is the core of data bricks and so all this to say data bricks has everything on that one platform and we don't really try to lock you into a single model or anything like that like you're we more we welcome you bringing open source models and models from other providers so that you can use it for your agent system.
So quick pause so I can take a breath but we're going to do a demo and it's going to be live so please bear with me. So um but before I actually like run into the demo this is what's going to happen. Let's set the stage for this demo. We're going to be uh pretending we're some kind of healthcare provider company. We're going to be receiving some kind of PDF. No idea what's in it. Uh and then we're going to have a vision transformer image classifier classical model from Microsoft.

10:20 Live agent demo

It's pre-trained for a thousand uh image labels to see if we can just based on that model identify what this image is. So that's this part right here. And then based on that output, we're going to send it to the image analyzer agent and have Llama 4 Maverick review that output and then let's decide is this good enough or should I send it to another vision model to get more information out of that document so that I can determine what exactly this document is.
And I'm going to use good old testeract here. I'm hoping if you're in the back, you can read that. Um, but we're going to use good old tester to do text extraction. I'm sure many of you have many of you have used uh testact in the past. And then we're going to do that one more time. We're going to pass it back over to Llama 4 and have it review that output, which will be a lot more text for it to review and then decide, okay, I have enough information.
I know what this document is. Now, let's cycle through the other agents that I have set up. So, we'll hit and I don't know which one it's going to actually pick. Every time I run this, it's a little different, but it will either hit the genie agent, which will query one of the genie spaces that I've pre-created and then also use the results of that genie space because we're looking for we're going to pro we're going to look for uh any relevant information in that genie space and then more relevant information in that vector search based on the outputs of that genie space.
Hope you're still following along. uh we're going to use data bricks cloud 4 because why not? This is more of just like a showcase of like I'm model switching within the uh system itself and then uh text ETL will help with keyword extraction that we can use in the summarization or passing it back into the genie agent using those keywords so that we have a more accurate query. And then finally, the Spark final agent will take all the information that's gathered based on the information from that PDF and write it into a delta table.
Okay, hope you're following along. So with that, let's end this and we're going to jump over to here. So I will zoom in a bit. Okay. So, for further context, we're actually going to put in a medical discharge summary, but you as the user don't know that. You as the audience know that. So, our goal is going to be and the query is going to be get an updated doctor's note column for. is in this document. So, the agent system doesn't know and I'm not specifying.
He's just that's just the patient name in that document and the agent has to figure that out. So I'm only showing you this query because we have one row of where we have a doctor note of sleep more. I'm sure a lot of us can benefit from that. And um no deductible information. I'm going to add in the query that I want to see deductible information, but in a medical discharge summary, you're not going to have that information.
And you can see in my structured data, I don't have that information either. And that's where the vector search comes in because uh before this presentation I actually have a vector search index that contains insurance documents based on the uh insurance name as well as the insurance type. They're all examples. Um they're really just kind of examples I found online. And we're going to use that and match it up to what uh uh insurance information is to find the right document and then find a deductible.
And this is where maybe a multimodel large language model would be more important because this is a much more complicated task. So we're going to use this information right here. Okay. Now let's give this a try. So I'm going to upload that discharge summary. Okay. And let's start processing. So first we're going to process those PDFs into images. Okay. It's going pretty fast. Uh, and now it's this first response is saying after the vision transformer, I'm realizing I don't have enough information about this.
So, I'm going to actually move it over to the test OCR and I'm trying to keep up. Now, we're at the patient lookup genie. Uh, where it's going to based on the information from that testact OCR out text output. Uh, we're going to send it to the genie space and then we're going to send it to the vector search index to find the insurance information. So, this one takes a while because the genie space and the vector search index uh actual calling takes a bit more time.
Um, but I'll actually pull up what the medical discharge summary looks like. And this is what it looks like. This is the PDF that I sent in. So, there's a patient name here named, some other in information about hypertension, etc. , etc. Um, you can see I did find this online as well, but is s it is a sample discharge summary that um has some pretty good text in here to kind of describe a typical discharge summary. Okay, looks like it finished the uh patient lookup uh agent.
So, we're going to move over to the final agent that's now writing it to a delta table. It's saving it as structured data so that I can use it later. Um, we'll give this a bit of time to run itself. Finalizing. Okay, great. There there was three pages in there, so that's good. We got patient information, insurance details, visit information, and so forth. It found the deductible, and this was the original query that I had put in.
Update my delta table right here with information from the document, add their insurance deductible to the table. So, this was sent alongside this PDF. Okay, so this is what the large language model or the agents had uh took took a look at. But as you saw in the beginning, this is just one row. So let's actually verify if the table itself has another row with in it with an updated doctor's note column and an up and a new column called deductible.
So I need to turn my Gradio instance off. Let's rerun this query and pray. And there it is. Uh there is a there is another column here with my doctor with the doctor notes from that document as well as a deductible for 2,000 individual and 4,000 families. So I again if you're imagining yourself as a user I have no idea what's in this document. someone had just uploaded it and we trusted the agent system to make the decisions to extract what I needed based on the original query which was this just update my table with information from the document and then we add their deductible to the table.
So thankfully my live demo worked so we're in good shape. Um, so with the time that we have left, I do want to quickly go uh through the DSPI part of this. Um, this uh, okay, we h this took a little longer than I thought, but I'm going to essentially run through uh some code to really highlight how DSPI has made this a lot easier for me to develop a multi- aent system. Um, so I'm going to skip all this. This is a notebook that I will make public in a repo, so you're more than welcome to come and try it.

17:45 DSPy implementation details

But uh let me skip all this because this is all like ingesting the data ahead of time. This part is like a this part would be good in the workflow like if you had a PDF land in your cloud storage you want to do like an like a job to like actually convert it you know that would be a good step before this. Um this is tool tool setup getting the genie spaces the vector search the model serving endpoints and so on and so forth.
And now we're in the DSPI part. So DSPI is great because if I was trying to test and use different models, I just need to assign them like this. I can use the anthropic provider. I can use data bricks. I can use OpenAI. I can use Bedrock. It doesn't matter. You don't have to reinstall another library. You just need to change this string to the provider and then the model endpoint name. That's all you need to do. The rest of your code will be compatible and you don't have to change anything else.
So very very helpful for development and iterating through over that. DSPI base type is a new um object that's coming out as part of DSpi 3. 0 which you should attend that session that's tomorrow. Um but the this u is pretty much paidantic based model. So just look that up if you don't know what uh I mean by that. So we're going to uh blow through that because I want to focus on DSPI signatures. Signatures is how DSPI takes your uh code and that's how you're describing what the model should do to accomplish it its task.
Right? So this first one is a text summarizer extraction. That was the uh OCR a text OCR agent that you saw in the diagram. But all I need to do for this is define a dock string. Our recommendation for this is that if you do good documentation, which I hope you do, um you would write something similar like that to this. Obviously, this isn't a good example because I don't write good documentation, but you can see here that will influence and that will uh allow DSPI to adapt that as a part of the final prompt that it sends over to the uh large language model.
And we can define uh input fields and output fields. This is just telling VSPI that I expect two string inputs and a memory history input. Without these three, it will reject it. And then I need to see four outputs, the response, the summary, keywords, and the next agent or tool, which I use to uh identify which agents to go to next uh based on what the large language model thinks. And so I repeat this process for each agent.
This one was a little longer because well there's some there was an extra tool that I wanted to write and I have a lot more different values that I use here and so on and so forth. Repeat that process until you have kind of everything you need. But the best way to think about it is if you need to accomplish a task, try and write a signature if it's very specific and it's not duplicative or not redundant. Um step five is just testing to make sure that each of them work the way it tends.
So again, it's just testing. So I'm not going to focus too much on this, but I do want to show you the implementation of this where I put it all together in a DSP a DSPI module, a custom one where I define a ded uh document in DSpodule initialize some of those agents using DSp. react predict and so forth. These are pre-built modules that I can use to uh interact with a large language model and then I use that throughout this class, right? it becomes uh a DSPI module object.
I put the tools that I need to use here in this class as well. And if I scroll down, there's the genie uh tool itself to actually interact with the space. I'm going to scroll all the way down to handle question because this is where I use the uh structured outputs from DSPI. It is a declarative framework. So I can use the outputs from what it generates. So I'm going to call the text processing agent here. this text agent response as you remember I had like three outputs for this signature so I can call them directly using response uh dots summary so far and next agent or tool the best part about this is that I don't have to rerun this again I can just use it downstream in my other agents like this text agent response summary so far and now I have that values that that very specific value and the same for the genie agent response Here you can see that I'm using the outputs that I define in that signature within the code itself so that I can use it through my Python logic.
Um, and you can see more of that logic happening in the forward method which is actually what gets called when you call this module. So lot going on trying to cram it all in like 30 35 minutes. But to then use your module you just you know def instantiate the class and then put provide the query and medical summary. So, um, just so you can see a little more detail into what was happening in that Gradio instance, I do have some print statements that I wrote that you could take a look at.
But, um, let's go here. Here the text query being sent for the vector search insurance company 1 PO deductible. So, the agent knew that, okay, I'm sending it to the ve uh, the vector search tool. Now, it would be better to send some keywords in. So it created some keyword to send to the vector search instead of the original text query that I had put in. And so it's using the context from that document to better uh call the vector search tool.
Um and then I think it and if we go more and then this is more of like a summary of like everything it found right the genie agent uh pulled this information and then it used that to do more things. This was the original query that I sent. This is the summary of the tessoract uh text extraction and this is the deductible found in the insurance documents themselves and then it wrote it to this delta table that it found in the original query.
Okay, that is all for the live demo. Um there's six minutes left. I did want to leave it open for Q&A and I ran a little over. So with the remaining six five minutes there are mics on that side and that side if you would like to ask any questions but if not thank you for attending and I hope you learned thing.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.