Streamlining DSPy Development: Track, Debug, and Deploy With MLflow

Summary

• DSPy framework: DSPy simplifies GenAI application development with automatic prompt optimization, addressing the complexity of manual prompt engineering and reducing maintenance burden.
• MLflow integration: Combining MLflow's tracing and autologging capabilities with DSPy brings full observability to development, enabling tracking of module calls, evaluations, and optimizers.
• Production readiness: This video demonstrates how to debug, iterate, and understand DSPy workflows before deploying your entire program end-to-end on the Databricks Data and AI platform.

Streamlining DSPy Development: Track, Debug, and Deploy With MLflow

Talk by Chen Qian, Senior Software Engineer, Databricks
DSPy is a framework for authoring GenAI applications with automatic prompt optimization, while MLflow provides powerful MLOps tooling to track, monitor, and productize machine learning workflows. In this lightning talk, we demonstrate how to integrate MLflow with DSPy to bring full observability to your DSPy development. We’ll walk through how to track DSPy module calls, evaluations, and optimizers using MLflow’s tracing and autologging capabilities. By the end, you'll see how combining these two tools makes it easier to debug, iterate, and understand your DSPy workflows, then deploy your DSPy program — end to end.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to DSPY
• 1:06 Compound AI system challenges
• 2:52 DSPY framework overview
• 3:47 Core DSPY concepts
• 7:15 Optimizer and workflow
• 9:33 MLflow integration
• 13:08 Practical code demonstrations
• 19:35 Optimization and conclusion

FAQs

What is DSPy and how does it improve GenAI development?

DSPy is a framework for authoring GenAI applications that includes automatic prompt optimization. It addresses the "prompt engineering problem" by reducing the need for manual prompt tuning and providing better abstraction for AI systems.

How does MLflow improve DSPy development?

MLflow provides tracing and autologging capabilities that give you full observability into your DSPy workflows. You can track module calls, evaluations, and optimizer behavior to understand and debug your AI applications more effectively.

What are the main challenges in compound AI systems?

Compound AI systems combine multiple LLM calls with custom functions, making them powerful but complex. This video shows how MLflow and DSPy solve challenges around prompt management, versioning, and understanding which prompts work best.

How do I deploy a DSPy program after optimization?

Once you've tracked and optimized your DSPy modules using MLflow, this video shows how to deploy your DSPy program end-to-end on the Databricks Data and AI platform for production use.

Full transcript

0:00 Introduction to DSPY

Welcome to my session. So in this session we are going to explore how to use DSPI and ML flow to build gen applications or agents in a fun way. And today's agenda like we'll talk about like we do a full introduction to DSPI then do a brief introduction to MLflow only covers the part we need to use in DSPI development. Then after slides, we'll see a few code demos so that make that make more sense. Cool. I always start with this question.
What is DSPI? So DSPI is a geni authoring framework that simplifies the development of gen applications. I mean this simple sentence doesn't make any sense to me to be honest because the bar for building a geni framework is super low comparing to building like a PyTorch or TensorFlow. So we have so many so many options over there in the market. So are we just reinventing the wheels? The answer is no. But let's explore together why it's no in this session.
Let's talk about the problem we're facing when we started to build DSPI. Since the success of CH GBT in 2022, we are seeing the rise of compound AI system. Compound AI system is like system has multiple modules and each module could do RM calls or like run custom functions and combining all them together makes you a very powerful system that can do like rag knowledge based QA or generate code of high quality. But we are all facing a problem called prompt engineering.

1:06 Compound AI system challenges

We do prompt engineering because we know that with a better prompts we get a higher quality response from the LM. But the problem is prompt engineering is very messy because we do changing the prompts based on what we think and make a better prompts but may not think in that way. What that means that means we may end up with like 50 to 100 very long prompts stored somewhere and it's hard to trace back which is good one which is a bad one.
Making that even worse if you change your behind the scene you start over. So all that you did was nothing. And we have another problem coming from the frameworks. I've been hearing people saying like frameworks are powerful and powerful troublemakers because like it's like with the frameworks instead of focusing more on writing my customer logic I'm more focused on learning the frameworks which is nonsense. All the purpose for having a framework is easy to use save my time so that I can focus on my on my business and what that means is like to migrate to some framework it's a lot of time if I want to get to another framework to migrate off the current code it's a lot of time again that is what DSP trying to solve so let's answer the question what is DSP again so DSPI is a lightweight flex is a lightweight flexible framework network which is easy to migrate into and easy to migrate off.

2:52 DSPY framework overview

We simplify the interaction with the language model by transforming the prompt engineering into something like calling a function with wellestablished inputs output outputs format and DSPI provide you automatic program optimization. We do prompt optimization and we can also fine-tune the LM weights behind the scene through something called DSPI optimizer and DSPI is ready for production. We have streaming async high concurrency cache out of the box so that you can focus more on like building a custom logic.
This may not make sense to people right now and after talk I hope that makes more sense. Okay, let's dive into how to use DSpi. So we start from DSP programming which means we how to build a gen application with DSPI. So DSPI programming is based on the signature and the module system. Signature is DSP signature. Module is DSpodule. Let's take a look at them one by one. So signature defines the inputs and outputs format of your LM interaction.

3:47 Core DSPY concepts

What that means in normal context when you quality LM is string in and string out. But like in the context of DSPI, it's more like you are calling a wellestablished restful API, but you define an input and output at the client side rather than a server side. This is through the DSP signature. Let's take a look at an example. So this is some signature I built for myself to learn Japanese. I want the input to be a simple English text.
I want output to be Japanese text and some other like grammar information here. So there so this one is called translation subclassing from DSP part of signature and there are five important parts in the signature the red box the dockstream part it's like a high level description of what we want to get from the LM interaction and orange box is the is a field name like it could be like input field or output field input field is how you set a value when you call the LM say like I give that English text And the output field name is how you read the response from the LM.
Like if I get response from LM, then I can do response Japanese tax that gives you the text without needing you to parse anything. The purple box is like some additional information to the fields. Sometimes the field is not self-explanable and the green part is a type hint so that like the output is already in the type I want that to be. No pars no parsing is required. So signature we have the static information which is not enough because we need to call the LM to use the signature.
This is through the DSpot module. DSPI module is a core unit of a DSPI program and we write all the logic inside DSpi module. The simplest module but also the most important is DSpot predict. It is bundled with a signature and simply cost the LM with a signature. No other logic there. modules are configurable. It has fial examples and you can specify the LM to call that has some other fields. And I'll say that one important thing about module is module is super flexible.
We don't restrict you something like you can do or cannot do with a modules. You can simply subclass from DSP module right to your own program like or custom module and override the forward method to put anything there. This is pretty wild. You can do anything in the forward method. You can of course call call DSPI modules but you can also call like lanchin modules agonal agents and or like lanchin to mcp tool if you want to train a model you can also do that in the forward so this is why we say like dspi is easy to migrate to because you just put your code into the forward function you don't need to change anything if you want to use the pi module you can change part of your your code to dspi module and one day if you hate the pie I hope not but you want to migrate off just take the forum method This gives you the code you want.
Let's put them together to form an end to end workflow. So like the first step we need to do is pick a L behind the scene. Then let's say we build a sentiment analysis app like so we define a signature take a text output sentiment we feed a signature into DSpot predict which just a dom calls then call that call call a predict with the input field. You see like we do text equals to something then access then read the response with a sentiment output field that's all you need with the end to end workflow and we're missing one thing what is actually we only have one slide for this but this is actually the most important slide in DSPI I call this DS my magic this is called DSP optimizer so like what that does is after you build your program you spin up a DSP optimizer and prepare a data set data set could be as few as like 20 data records and also give us a evaluation metric.

7:15 Optimizer and workflow

The purpose is that we need to understand what is a good program and what is not a good program. With those two we can start to search for a better prompts or even twin um our weights behind the scene. So like to use the optimizer you just like send the like training data set and the evaluation metric to the optimizer and call the compile method to kick off the training process and then the return the program. will of the will be of the same structure as before but I have prompt being a better one.
So the one thing I want to call out here is I've been seeing a pattern people like the first reaction hearing this they think okay this is good this is cool but like come to real production they still sticking with like doing custom prompt engineering because if they don't want to spin up like data set this is like overhead and then they don't do DSP optimizer but the problem is if you do serious productization you want that to evolve along with the time you want to keep understanding the new data and to do manual every time to do to keep that manual every time is really time consuming and unreliable.
But DSPI optimizer is built by our research team with lots of experiments. This is trustworthy and that can also help you build a high quality fusial example not just like V coding or something twin the prompts. So this is really good. Give that a try. That is everything about DSPI. Now let's talk about like MLflow. This will be very short. So what is MLflow? ML flow is open source ML ops and a ops framework that can help you like build gen applications throughout the full life cycle and uh the both the server and client ML flow is fully open sourced you can spin up your server but you don't if you don't want to do so you can use the like cannot see that it's in the bottom data bricks has some like free trial edition and also we announced that today you can give that a try at signup. databicks. com databases. com.

9:33 MLflow integration

I'm not sure if that's all right, but yeah, cool. And we have two things we want to bring into DSpay from MLFlow. The first one is MLFlow tracing. So what is tracing? Tracing basically means we record the inputs and outputs of the intermediate steps and captures a hierarchical call stack which means which function calls which function. This is very important because like in DSP after you build a DSPI program you only get the inputs and outputs.
You don't know what's happening behind the scene. Like if one LRM call fails for some reason it's hard to tell why that's failed. But with a trace it's easy to see what happens behind the scene. To turn that on it's just a one line. It's malflow. dsp. log. And this is what a trace looks like. We trace four things inside DSPI. We trace every module call no matter if that's a top level or like child module call. We also trace adapter.
We didn't mention adapter but it's very simple. So like in DSPI we call the DSPI program with structure inputs but there has to be one place we transform the structure inputs into a prompt. This is through adapter and the adapter also parse the resp the alarm response into a structure output. Even if the model doesn't support native structure output and ALM call is traced of course and tools also traced tool is important for building a gen agents.
So every tool call has been traced. We'll see that in a demo and this is just a demo on how to inspect the actual prompt to send to the LM through MLflow tracing. And uh yeah like this is a like something we just built is a DSP optimizer has also been tracked by and traced by ML flow. So like many people like if you ever used the DSP before DSP optimizer looks like a black box but like with the MLflow tracing you can understand what's happening behind the scene can see how we evolve along with the time.
We log every candidate program we're trying to evaluate. So we can you can see the matrix evolving along with the time and this is another size optimization and the last thing we want to bring from MLflow is the packaging and deployments. So like after you build your program you want to serve that for productionization purpose just do mfo. dsp DSP log model put your model there and give the input example then you just do a few clicks right now just do a few few clicks then the serving endpoint will be up on data bricks platform it's very easy even more like a no-brainer there I'll show you in a demo as well and if you know something called agent framework in data bricks you can also deploy that to get a playground UI to play with a program cool that's about the slides now let me show a demo Cool.

13:08 Practical code demonstrations

Let's take a look at the demo. Yeah. So, you first get get to configure the LM behind the scene and this is following the light like format. So, it's provide a name followed by the model name. You can pick whatever is supported by light. Then we turn on the MLflow auto logging which say turns on the tracing for you through MLFlow. ds autolock. And as we say we just say we need a let's take a look at a very basic workflow first we build a build a signature do sentiment analysis we take a context and context has a inputs and the sentiment which is integer at output and some description and we restrict the range between 0 to 10 and this is called a pent this is called pentic constraints which is fully supported by DSP signature and we just send that to DSP. predict predict and call that and we can see the malf tracing is up.
So you can see what happens behind the scene. So like this is the adapter formats to a prompt and this is actual um call you sent to the LM and to read the outputs you can just like one second you can do a dot accessor or access it as a like keyword. Cool. And we have seen like with the amo tracing you can see the prompt we send to the LM can be found here it's multi-term message very familiar to people but if you don't want to spin up the tracing you can do that by dos inspect history to print out the like the alarm interaction history here we go and you can also easily try a different built-in modules like say we have channel of thought which gives you reasoning besides inside the defined output and you can also like see the tracing tracing tracing here or inspect that by inspect history.
This is just all basic workflow. The next thing I want to show here is building a customer module and uh the demo here is like a nameless celebrity game. If you're not familiar what that means is I we have two players. Player one thinks about a name and player two keeps asking yes or no questions to get to the to the final name. could be a generic generic question or director guess. So as we said we subclass from the s module we override a forward method to do that.
This is actually a human in a loop. Sounds cool right? But it's just like asking for a yes or no here. And we have two module here. We have question generator. I want to show the signature here is we taking the pass of question and answer generate a new question and indicating if that's a direct gas or generate question. So we just keep going and this is a running history I find a name and then keeping tell the tell the LM if that's yes or no until reaching the end I mean this one that get got it.
So like doing a ref action the demo is just a for fun but like the real thing I want to show people is it's very flexible to do anything with a custom module you can just write Python function so it's just all Python you can write anything you can call lenient tool MCP tool query your database anything here no restriction and uh yeah we kind of after the time but I want to see uh the next thing I want to show is more complex I want to build a agent so this agent is doing like a airline and customer service.
I want that to be able to book a flight for the user and being able to cancel the itinerary or modify itinerary. So we first define a data model. This is like the database schema if that's a real production. Then we create some dummy data so that we can actually call the call agent. Then we define a tool. This is important. When you define tool in DSPI, you want to give the like dock string to describe what that does and also you have the type hints for it arguments so that the LM can help you decide arguments.
So after you have the tools, we define a signature which is input and output of the whole agent. Here we're just taking a user request and we output a process result and we bundle them together through something called DSpal react. We just let our decide to if that to call a tool or draw a decision and we call that with something my name is something I book a fly from this to that then here is what when MLflow tracing becomes really valuable you see we have multi-turn calls and every turn has a different tool calling decided by without ML flow it's just black box without with ML flow you can see what we do behind the scene the first one thinks we need to get a user information.
So we pull the user information and send all this back to the LM. Next to a call will be fetch a flight information. So we just keep going calling the tool until we are able to book the itinerary. Then we finish and tell the tell the user this is the final outputs with a confirmation number or so. If something is going off, you can see the exception here. Like for this one, for this query, we don't have a fly from SFO to JFK on this day.
So you'll see an arrow here. It's very easy to debug to see what happens behind the scene. All right. And uh probably the last thing I want to show is to deploy that on data data bricks for just deploy the airline service agent. So just call the DSPI log model and this is log for that one. Then you open that you will get this like this is your log model. You just click on the register model that'll be registered in Unity catalog.
Then on the Unity catalog, click on serve this model. That brings you to the serving UI. You just give a name, configure something, new create. That's all you need to serve the model. Very simple, right? Like you don't need to know anything about packaging, Kubernetes, nothing is required. Just do a few clicks to serve the model. And this is uh so I deployed that before the talk. So this is what that looks like. So you can use it theory at any place use your own like token and you can use Python client.
We just use curl. Cool. And if people want to stay around for another minute, this is how we use a DSP optimizer. So like we download, we define the evaluation matrix. This one is a little complex because this is math equivalence. Then uh you download data set. This one we have a ex existing data set but in real production you may need to find your own data set but I say you don't need a big data like a fine train something you just need 20 or 30 which is totally enough and we get a data set and we yeah and we can turn on the maflow tracking with a few more flags in autolog function and we eva we can evaluate the baseline before before optimization we see a score of 72 two then we all right this is cool then we spin up the optimizer nipple optimizer is one of our most popular one which does the prompt optimization doesn't tw anything so like we can we call evaluate well we use optimizer use a compile call to get a better one and then we can evaluate it after optimization process we still get to 80 81 this this takes about like seven minutes in total I and you can get like free bump.

19:35 Optimization and conclusion

So this is just showcase how to use that end to end. Cool. That's all of the demo and because we don't have a lot of time this is very quick. If you want to like learn more about DSP and ML flow I do have a course on deep learning AI published from last week just for DSPI space course space deep learning. ai AI and it's like a it's a short course. It's one one hour and I have more details for all of this. And I do have a brand data session from 400 pm today.
If you want to learn more, ask questions, you can find me there. And my colleague who is also the creator of DSPI will give a talk from 520. You can find location. I forgot the location but like it's about DSP 3. We are releasing a beta today and so it will be it will be really quick. You don't want to miss that one. Cool. That's all about the talk. Thanks for coming.

Learn more about the Databricks Data and AI platform.