Skip to main content

DSPy 3.0 — and DSPy at Databricks

Summary

  • DSPy 3.0 release: Databricks and the broader DSPy community released version 3.0 with improved prompt optimization, reinforcement learning, and MLflow integration.
  • Prompt optimization: DSPy moves beyond manual prompting to automated optimization—adapting prompts and fine-tuning to work reliably with any model.
  • Productionization: Native MLflow integration enables observability, versioning, and deployment of DSPy systems at enterprise scale.

DSPy 3.0 — and DSPy at Databricks

Watch: DSPy 3.0 — and DSPy at Databricks
Talk by Krista Opsahl-Ong, Research Engineer, Databricks ; Omar Khattab, Research Scientist, Databricks
The DSPy OSS team at Databricks and beyond is excited to present DSPy 3.0, targeted for release close to DAIS 2025. We will present what DSPy is and how it evolved over the past year. We will discuss greatly improved prompt optimization and finetuning/RL capabilities, improved productionization and observability via thorough and native integration with MLflow, and lessons from usage of DSPy in various Databricks R&D and professional services contexts.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is DSPy and why do we need it?

DSPy is a framework for building reliable AI systems that adapt to new models without manual rewrites. Instead of writing longer prompts for each new model, DSPy optimizes them automatically.

What are the key improvements in DSPy 3.0?

DSPy 3.0 adds better prompt optimization, reinforcement learning capabilities for fine-tuning, and deep MLflow integration for production observability and versioning.

How does DSPy use reinforcement learning?

DSPy applies RL to fine-tune models based on task-specific reward signals, rather than requiring expensive labeled datasets. This enables continuous improvement and adaptation to your use case.

Who developed DSPy and where is it used?

Omar Khattab (Research Scientist) and Krista Opsahl-Ong (Research Engineer) at Databricks present DSPy, which is used across Databricks R&D and professional services for reliable AI applications.

Full transcript

0:00 Introduction to DSPy

Welcome everyone. Thanks for coming to the talk. Um. Um I'm a research scientist at data bicks. I started the DSPI project as part of my PhD at and I've been working at data bricks for about a year where we have a nice um open source DSPI team that collaborates with the broader DSP ecosystem. And I'm excited today to uh tell you all about what we've been building and DSPI 3. 0 which we just um open sourced earlier today.
We have a lot of the DSPI um contributors at data bricks and maybe even from outside in the audience and um I'll you know be doing a lot of shout outs to a lot of the work that they've been doing. So the context of this talk and of a lot of the DSPI work is that it's actually kind of impressive how clear it's become how we as a as a AI community and data community and and more broadly should train large you know language models and large models that do AI stuff.
But as you've been hearing all around the you know DICE so far and maybe elsewhere the field is actually still guessing how to build reliable AI software. So, we've managed to build this little foundation, this essentially, you know, processor for a lot of the uh behaviors we want to exhibit. And of course, there's a lot of improvements that are to come. But how do we actually take this and start building all kinds of applications on top um that actually do the stuff that we care about?

1:15 Limitations of Prompting

That's still unclear. Um and you know, if you're here, it's because you're building with LLMs or because you work with people that build with LLMs. And the kind of name of the game is that, you know, every week a new large language model comes out and a new interesting one comes out, not just like, you know, a copy of what was there before. Maybe it's more efficient for the quality it gives you. Maybe it's higher quality for the cost it's at.
You know, maybe it works better in certain domains, whatever. There's a lot of, you know, reasons you might care about those. And, you know, they don't just come out and they're interchangeable. Every time one comes out, there's new quirks. and you think that, hey, you know, it's like next year is going to be different and it ends up being the case that like the prompting guides only get longer and longer and that's only if you're lucky.
Um, if you're less lucky, they don't give you a prompting guide and so like you kind of just have to guess at it. And the surprising thing is that the providers when they write these guides, they change their minds all the time. And they change their minds all the time because nobody is um engineering these LLMs, right? The way they're they're grown essentially, right? they're trained on a lot of data and they emerge from just the total sum of what we train them on.
So everybody's basically interacting on the basis of a lot of trial and error um even though we have a really intuitive interface and every month beyond the LLMs um a new learning algorithm comes out. So nowadays is the excitement around you know the well-deserved excitement around reinforcement learning. You know a lot of algorithms that are sort of being released all the time in open source and that you know um companies are building internally that you hear about that might be you know exactly the sort of thing that you need to find that you make your app a lot better.
Um at the same time a lot of inference strategies hey agentic architectures or inference time scaling methods or ways of doing reasoning or all other kinds of sort of architectural decisions are being released in research or announced from companies or announced in tutorials or elsewhere that seem to claim uh that they have the keys that will unlock greater reliability for your usage of large language models in building applications.
And you know this is just everywhere. So I have here some uh you know um screenshots from just OpenAI's you know um guides on prompting. So there was a there's a general one, there's one on 4. 1, there is like here is how you build agents and make them reliable. Here's how you organize your prompts and they tell you things like hey you got to repeat if they have a long instru you know if you have a long inputs you got to repeat the instruction at the top and at the bottom. um they tell you things like hey if 03 comes out here is how you should structure your layout for 03 which is you know the latest reasoning models from open AAI and my intention is not to pick on any particular company or anything like that my intention is to say that when you look at all of this you're getting lots of low-level guidance that seems to change every week right as these new models come out and that was just one LLM provider so there's actually differences across all of these LLMs um so the question that like motivates a lot of what we're going to talk about is like are you supposed to like I don't know stay on top of all of these just to do like just to maintain the software that you're building with LLMs and are you supposed to do that forever?
I don't know. It doesn't seem particularly sustainable. It doesn't seem like the right approach. So let's actually make this concrete. Let's say that you're writing an internal productivity app in your company that every employee can use um whose job is to take their emails maybe and their Slack messages or Teams messages or something and digest them for every employee um to help them maybe plan their week. So maybe it's going to look at all of their emails, you know, privately for each employee and maybe look at all of the events that um have come up that might be interesting to them, maybe things that they were invited to or things that have have showed up in their email and maybe extract all action items and things that they promised or things that they were asked to do or things that they, you know, kind of brainstorm with other folks.
Um and then maybe kind of with this information block some focus time blocks in their calendar so that they can work on these things. just as a you know just a simple idea you might have that you want to build internally in your company how should you do this well you know architecturally and conceptually it's actually a great fit for LLMs right you have a bunch of email threads maybe you take one at a time you give the LLM the subject of the thread you give it the email thread all the attachments maybe images and PDFs and whatnot you find a way to give them to the LLM and you just ask it hey give me the events and give me the action items and I'll do some like processing with those and maybe the other processing will also do you So we'll also call LLMs.
It's incredibly simple and clear and crisp and you know we can have this beautiful diagram. The problem is you start thinking well what is an event exactly? Um you know it has a name um it has you know a location it has a time um and in order to connect this to the rest of your system you got to be able to actually get that stuff out of the LLM. It can't just give you like a response like in a chat format right you got to actually be able to extract this.
So you start saying things like always format your answer following this JSON schema right I want these little pieces in JSON format and so far it's actually not that harmful you know you're just designing your system and then you realize well I don't know people are talking about all these prompting techniques and the LLMs are actually fairly chatty um you know it works on some emails but not others so I'll start saying things like hey if you listen to all of what I said I'll tip you $1,000 um or like please take a deep breath and I've heard that reasoning is good for LLMs so take a deep breath and think step by step in think tags because then like I don't know I I hear that's supposed to make LLMs more reliable.
Um, and then you you think about it. Well, wait a minute. Um, I have these complex threads. Sometimes they're long. Maybe they don't always fit in the context memory, but sometimes they also have attachments. Um, they have all these little pieces. Some threads, you know, have multiple emails. How should I actually feed that data, which is I have an object, right? I've loaded the dictionaries and the, you know, like the, I don't know, Python objects, let's say, or JavaScript objects.
I've loaded them in memory, but now I actually got to give them to the LLM, which just expects one big blur, right? One string. how do I serialize this into the LLM? And the problem is the way you serialize matters, right? Because if you give it in a format, it's actually kind of not ergonomic for the model, it'll do a bad job. So you start thinking about this and say, hey, people say that XML is pretty ergonomic for LLM.
So I'll start doing some XML manipulation here. And then you realize, well, I have a whole bunch of emails and even if the LLM does a reasonable job on one of them, I actually want it to be consistent because I want to take these things and process them. And if the LM is doing different things on different emails, you know, it's going to be pretty hard. And you do this for a while and then you realize it's actually still not working.
So you ask around and you hear that, hey, maybe personas are the answer. You are Professor Einstein, a PhD in information extraction from the LLM Institute of Technology. Uh, you do not hallucinate events. My job depends on it. Right? So these are the types of prompts that you might see in the wild. And nobody writes this prompt up front. This always happens as a result of like scars in the battle, right? Like working with these with these models.
And the problem is um you know, you end up with a prompt that looks like this. I didn't even bother assuming that you will read it, so I just put it sideways because it was wouldn't fit. And here's the issue I I have. Here's like the thing I'm very passionate about. Prompts are a horrible soup of a canvas for programming. They are incredibly bad canvas for programming. They're not actually that bad for management. If you have an agent who you know who you want to treat as essentially a you know a cheaper employee maybe a prompt is fine.

8:31 Engineering over Prompting

It's like having a Slack channel with a remote employee. But if your goal is to build a system that you can rely on which has modular pieces that interact and interact reliably with good contracts between them right the way we build software prompts are a really bad abstraction for this. And the reason for that is not just they're ugly. They're pretty ugly. That's fine. Um it's that they entangle and mix up your actual system design. the beautiful di diagram we looked at before, for example, with many tiny incidental choices you don't actually mean, but you kind of are forced to make them.
Um, and maybe you don't even really have a preference, but you're forced to make a lot of these decisions. So, when you're prompting, you're hard coding, which is kind of a bad word, right? Hard coding dozens of little incidental choices like the exact wording and examples, which can be very long, that happen to appease your specific LLM. The LLM changes tomorrow. Nothing says that any of that wording was the right wording uh for the new LLM.
Um the inference strategy we said we said things like reasoning it's actually a really beautiful idea reasoning but in practice you're say you're saying things like take a deep take a deep breath and write this XML format this JSON which is not the fundamental idea we're trying to express how you format your data how you specify the output format that the LLM should produce and how you actually enforce it um and then how you actually parse and when the LLM inevitably fails to actually um you know follow these instructions how you do the retries or validation or whatever it is that will allow you to handle the failures.
Right? So there's a lot of code and a lot of strings and it's still an unreliable mess. You know, this reminds me of how we write bad code if you want to write bad code. So I've come across this u snippet on uh social media. It's unrelated to LLMs. I looked at this and I had no idea what it does and I don't expect you to look at this and know what it does. But apparently it was computing a square root that was supposed to be efficient for a certain machine architecture with a certain floatingoint representation at some point in time.
And I don't know how this computes that, but all I know is that this is clearly not the most future proof program. For one, it's not correct the moment your floatingoint representation internally changes, right? Because this seems to be an ancient representation. Um, so it's not even portable to a new hardware. But second of all, if you didn't have the name at the top square root, right, and you're looking at this and you're supposed to maintain that code, even if you wrote it, you have no clue what this is supposed to be doing, right?
Um, and in the prompt land, that's what we get. You get this huge sort of block. you have no idea what really they are actually trying to say. I mean, yeah, sure, there's email and extract, but there's a whole lot of stuff going on here. Um, and so you're losing track of the actual fundamental stuff. And a solution in software land, by the way, didn't happen immediately. That took a while to get things like structured programming.
Uh, people were building, you know, all kinds of soup with, you know, might have heard of go-to statements considered harmful. So, I'm telling you, prompts considered harmful. Um, the solution was higher level languages that ask you to say what you actually want. I want the square root, right? And people building compilers and runtimes that take this highle expression and then figure out whatever the machine wants to hear in order to do it efficiently.
And sometimes they'll do a great job, sometimes they'll they'll do a mediocre job, but whatever it is, we l we're raising the level of abstraction together that we work at so that we can build actually reliable and performance software. So I think we have the same problem in prompting dozens of agent frameworks. I don't know why there's so many coming out all the time. Some of them are actually helping with accessory behavior like orchestration, validation, observability.
Great stuff, actually important, you know, almost non-negotiable perhaps. But the problem is they never tackle the central abstraction problem we've been talking about, which is that when you use any of them, still your app's primary logic is still in sloppy prompt soup. Um, and I don't understand uh why this is not a problem that has been fixed at scale yet. And so we're here to tell you about the solution. And so the question that we're going to tackle concretely is what if building AI systems was more of an engineering discipline?

12:27 DSPy 3.0 Release

What if you could actually bring back programming and software engineering into the land of building uh AI software because it's not any different. So this is the SPY a lightweight declarative framework for programming not prompting LLMs that we've been building for the last three years after another three years of building uh LLM software since 2019. Um what DSPI does is uh it handles evolving the lower level layers that we've been talking about as changing all the time for scaling for learning and also for plumbing.
And I can tell you the plumbing is one of the hardest pieces uh below your app. So the first thing we do the first foundation is we try to turn language models from you know string in to string out um into actual function processors right because we want to be able with functions that we can rely on to have a contract of input and output. Uh and then the ne next thing we do is we build inference strategies that encapsulate behavior like reasoning or agents or you know tree search or whatever it is that's the latest interesting way of actually taking language models and turning them into the best sort of version of themselves in terms of quality.
And we also handle automatic optimization which is a question of well you've told us what your task is and maybe you've also given us some data that helps us sort of understand your domain. How do we take this and actually come up with the final prompts? Or how do we take this and fine-tune your LLM weights so that you get sort of the the behavior that you're anticipating? Here's what we cannot help you with, but what we make much easier.
Um it's declaring your actual sort of AI system. So this is where you're describing um the flow between all these pieces like the diagram we saw. Uh this is where you're describing a natural language what are the inputs, what are the outputs, what is the transformation between them and also where you supply evaluation sets or maybe synthesize them or otherwise um define the objective that you want to climb closer a little bit to uh to more traditional machine learning and of course it's entirely optional.
You can do a whole lot without this one. So um I'm excited to tell you that you know today we've uh open sourced the 3. 0 uh beta first beta release. You can already sort of use this with pip install preu dspi. Um, and I'll tell you today about the culmination of the last nine months of work uh by the DSP open source team uh and contributors all around the world from all institutions including here at data bricks. Um note barely any breaking changes.
Uh DSP is incredibly stable in terms of the fundamental abstractions. There are three fundamental abstractions. they've been there basically all along and they're still the same but we're evolving a lot of sort of their ability and the the functionality and features that we give you. So I'm incred incredibly grateful to over 300 contributors many of whom are here in the audience. U I would like to thank a lot of those folks you know uh by names you know, Delara,, Arnav, Tommo, Lakia,,, Shangen and many others.
Um and um you know along the way to 3 2. 4 we've had over a 100 intermediate releases which is just an incredible number maybe almost absurd. um allowing the library to grow to over a million downloads per month and over 25,000 stars um um in in the meantime. So, let me tell you about like the main concepts in the S5 because I imagine that some of you have not used it before although I I see a lot of familiar faces. So, I know some of you have been using it for like I don't know two years.
Um and along the way, what's new um from the runup to uh DSP 3 3. 0. So, there are way too many features. I actually don't even expect you to like completely grock every single one of them. Um my goal is just to give you a flavor because I understand that some people here like have not heard of any of this. Other people have heard of some of them. So like, you know, the point is I'll give you a flavor of those, but follow us on uh on X Twitter or LinkedIn to stay sort of, you know, in touch with a lot of the detailed announcements for a lot of these things.

16:00 Signatures and Adapters

So the first concept in DSP, not a new concept, it's basically three years old at this point, um is DSP signatures. uh which are our way of allowing you to express how to turn the language model from string in string out to a structured function definition to a structured function right we had this nice diagram and a DSP signature basically is that um it's just a it's just a Python class in which you describe the task right the task is one line it didn't need you know the 70 line prompt that I I I showed earlier it's like extract all events if any from the email provide pro provided um you describe the inputs and you can also give them types.
So you can say here um I have a a subject I have a thread or an email thread and I have a bunch of attach attachments which is just like a mapping from strings to images. Let's say these attachments are images. Um and I want you uh the LLM to give me a bunch of outputs. I just want a list of events. Um and you can define an event like maybe it's a type maybe a pantic type or something. um and I want action items which is just a mapping from the name of the item to just like a priority you know P 0 P1 or P2 that is the whole thing right so like as assuming you define an event which could be just like you know five or six lines of like I want a name I want a title you know be a similar class to this one uh that is basically all you needed to describe and here is what we didn't say anything about we didn't specify the inference strategy do I want it to think step by step do I want this to be an agent that has tools do I want this to be self-reflective that wanted to do research.
All of these are valid. All of these are interesting. They don't belong in your signature because these are strategies you apply. They're not the task you want to solve. Um I didn't say anything about how to teach an LLM to do this. I just said what I want, right? I'm not fighting with the LLM. I'm not saying pay special attention to the attachments. I'm not saying, you know, like um if you don't do this, I'll, you know, I don't know, I'll I'll charge you $5 or something, right?
And I'm doing nothing about the plumbing. I'm not saying here is how to format in XML. Here is how I want structured outputs in JSON. these are none of the this is none of the business of the of the signature right that's defining the actual uh behavior I want that this is kind of a timeless abstraction that's meant to evolve that's meant to sort of stay the same as everything else around it evolves so for the past few months one thing we've been doing is we've been building a whole ecosystem around these signatures that allows them to stay the same and gives you even more control if you want it um so a lot of the time people approach us and they're like well I really like the spy I like the way it works but I still kind of can't adopt it if I don't know I can step in and actually change the prompts that get that get applied.
It almost never really needs to happen but we've exposed a whole lower layer uh below the spy called adapters um that are lower level abstractions that give you the ability to control and to export out of the spy the behavior of prompting. It's still decoupled from signatures. So you're still actually maintaining the same modularity, but we're giving you back a lot of that control. Um we've introduced several of these adapters built in so you can actually swap them and see what happens if you really want to, right? but they're still kind of decoupled and general purpose.
Uh so you have chat adapters, XML adapters, JSON adapters. Um we've tuned those out of the box for a wide range of LLMs. Um and they are intelligently able to fall back to a lot of the native behaviors that come in in LLM. So a lot of the time people approach us and they're like hey this LLM provider for certain types of you know pyantic models or something they offer native structured outputs. It's not for everything, but you know, does DSP you how can I use it in DSP?
And the answer is you don't need to do anything. DSPI internally can tell, hey, this this LLM provider supports this thing now. Um, if I can't get it to work in a way that I know is effective, I'll fall back to the structured outputs um uh in the in the right place in the right time. Um, we've also universalized signatures a lot more relative, you know, to to before. So, right now we've made them multimodel. You can give you can specify input as images.
You can specify um audios, you can pass PDFs and you know PowerPoints into your signature thanks to a an extension by um for just about any other type of document. We've also made sure that these things compose the way you would expect anything else in Python to compose. So you can give um your your LLM a list of DSP images or a list of pedantic types. Um and we've also introduced a number of other advanced types that help you customize and you know make make the best use of LLMs.

20:41 DSPy Modules

The second concept in uh in DSP after signatures is modules. So those are the the functionalities or the functions that can take any signature you described and then actually give it a strategy for how to do for how to actually do inference. And the nice thing is there's only a few of those. they evolve very slowly because you don't need to actually sort of um you know adopt every minor variation but the thing is you take your signature a new one comes out you simply slot it in and you get that new behavior for your signature and I'll show you some examples of this so this is the signature we had before I didn't change it it's the same thing and let's say I want the LLM to reason about it right some models are already reasoning models other models you actually have to sort of structure them in in your in your calls um for them to reason it is this right so you say I want the uh I want the chain of thought with extract events which is the name of my signature and I can then give it a batch of emails and say hey process these with 32 threads that gives it the chain of thought behavior.
There's a lot that's going to happen under this and we'll discuss that when we when we talk about optimization u but it's like hey this is the behavior I wanted to express. Oops. Let's say I want this signature to be an agent because I want to give the um event extractor I want to give it some OCR tools to read the attachments. Maybe I want it to um be able to read the attachments, turn them into text, think about them for a while.
You know, I want I would like an agent behavior that sort of u iteratively explores things and maybe it should be able to search other emails. Maybe the only way to understand each email is to go like get a broader picture of what's going on. So I want to give it these tools. So as long as you have these two functions in Python, just two functions OCR and search other emails, you can take the same extractor signature unchanged entirely and now it has become an agent.
Um, and people say, "Hey, how do we do multi- aents?" Well, you can pass tools to an agent. Some of your tools can themselves be other agents, which is another oneliner like this, right? And you can simply pass it in the tools. So, now you have a multi- aent system, right? We did not even need a new concept or a new um you not need to learn a new thing. You simply have this highly composable actually proper software abstraction.
Um, let's say you hear about reflection, right? I would like um the LLM to think about it. look at the output it produced, think about it again, right? And retry. And maybe it should have access to a way of like assessing that, right? Maybe I have a way of like saying validating the input or something. I can give my signature to a a refined uh uh method and you know basically uh a refined module and basically ask it to um um apply this behavior.
So we've been refining uh no pun intended the reliability, productionization, and observability of modules out of the box for the last nine months. uh been aggressively working on that with tight feedback loop here at data bricks um but from dozens of teams including in R&D uh including ourselves uh in agent bricks uh especially knowledge assistant and uh the information extraction uh bricks and professional services so if you've uh if you've not been to the talks by Austin and please watch them online and if you're here tomorrow go to Moros's uh talk on how he's using the SPI as part of uh you know his his data science work and professional services work at data bicks.
We work also with a lot of data bricks customers. Um I have a number of data bricks customers here that are sort of okay about going public about their their DSP usage, although there's a lot more uh that we didn't get to ask. Um and the rest of the DSpy developer community and sort of user community, which you know um is is pretty large. We have a nice discord if you uh sort of want to want to be part of that. You've probably heard today in the keynote the announcement of the Asian bricks.
Uh this includes a lot of mosaic magic and tools, but it also includes plenty of DSPI magic um in there, especially in the two I mentioned earlier, knowledge assistant, information extraction. Um and having discussed modules, let me tell you all about sort of what's new in them. Um we've been working on better scalability. So been really focused on sort of making sure you can productionize these things at really large scale.
Um so we can now batch process with your modules. um thread safe out of the box. It does a lot of heavy lifting for you. DSP is now natively async. Uh we have a high concurrency cache built in by default, but it's incredibly customizable. Um and we've been working on a much smaller package size. We've been also uh investing in the developer experience so you can do streaming of structured um outputs uh in DSP. You can get usage tracking in all kinds of granularities um of the per par per language model globally for a single program for a part of a single program.
Um and you can get and track history of each part of the DSPI system that you have as well. Um and for deployment we've been working closely with the MLFlow team closely so closely in fact that we share some of the same people um to make sure that um you know saving and loading and loading in data bricks and you know exporting the prompt management layer and things like you know all kinds of deployment concerns like getting all the versions and whatnot um is handled very smoothly.
We've revamped the entire docs while working on all of this and we've been working on integrations along the way for you know interacting with tools from lang chain or MCP servers out of the box. We've been improving um how DSpay allows LLM to execute code because we think a lot of the future is going to be LLM generating and executing code as part of their thinking processes. Um and we've been building modules precisely for this.
Um, if you were at the uh at talk earlier today, you would have heard that MLflow 3. 0 um includes a lot of native integrations with DSPI that allow you to see the prompts, allows you to see the optimization processes, um, and see how the, you know, the gains are are ongoing and allows you to also kind of track all of the, um, systems that you've saved. Um, we've also heard a lot of requests for courses. Um, and so we did a nice collaboration with um, uh, deep learning.
AI AI team and announced last week that there is a nice short course on DSPI, maybe the first of some to come and we're interested in your feedback. Um, it's a great learning uh it's a great learning resource thanks to from the team. Um, and now finally the section that many of you are probably looking forward to. Um, so the third concept in DSPI and the one that is like we're often quite known for is DSP optimizes. Um these are the components that you give say your program could be you know a single single prompt program could be a really complicated agent um and you give it a metric you know what is this thing supposed to do and do well and maybe give it a bunch of inputs like some questions that people have asked and it figures out how to how to prompt your LLM or even if you have a you know an LLM that supports fine-tuning APIs how to actually update the weights of the LLM to fit your task.

26:58 MIPRO Optimizers

So we've seen um you know the optimizer that's been around in DSPI for the past year and is most popular is called Mero. We've seen hundreds um maybe even thousands really of MIRO runs and use cases over the past year and we've used that to refine a lot of the little details um sort of that matter for reliability of MERO. Uh we've also automated its configuration out of the box. We used to hear hey I don't know how to configure this parameter or that parameter.
So right now this code here is a complete piece of code not only to use mipro which is this two lines right it does not have it does not require any hyperparameters you just give it the judge you ask it to compile your extractor program that we defined earlier but it also here has um the ability that you know I'm here using defining a judge uh which is GPT4. 1 that's going to teach my llama 1 billion parameter model um how to be as accurate as possible um for this task and this is actually a complete piece of code in terms of that functionality.
Um, a related, you know, similar task of information extraction that I worked with and this is actually, you know, like it's a little public snippet I I I posted a while back actually allows the model to learn um to jump in performance basically doubles the quality of the small model and this is, you know, something that you can do quite often. Um, so thanks to a lot of the team's work on this and the reliability of MERO, Atlacon, a couple of months ago, Meta announced a new prompt management tool and migration tool that they used to um allow people to move off of, you know, other models to the latest Llama models.
And lo and behold, at the core of it, it was built in the SPI and relies on MERO um and delivers up to 14% uh gains on their rag customer evaluation. The same day, I don't know how these things happen, uh, Amazon AWS also announced a migration tool uh, based on MERO, but this time only 13% gained uh, on summarization evals. Um, Colombia researchers and researchers everywhere are using these things and sort of writing about them in public.
So, Colombia researchers announced Papion uh, a while back, which is a privacy oriented system for coordinating large and small LLMs and um, you know, they use MERO out of the box. Um I I know because I was slightly involved. Um and it improved the quality of the systems by up to 41% and decreased leakage of these small LLMs leaking information to large models by up to 68%. Um physicians um uh at the data lab um significantly improved the quality for chest X-ray and head CT reporting by over 10% comparing with other approaches for prompting um and then outperforming all of those uh thanks to mero and so many other instances.
I just really didn't have the time to cover, you know, even a a small fraction of these, but a lot of them are public and a lot of them, you know, you can find on our accounts on social media or our discord. People like to post the stuff there. Um but to get a lot of this in the same place, we collaborated with folks at UC, Shangi Tan and Lexia um Agro um to build this lang probe language model uh uh program benchmark um that basically brings together a lot of the types of AI software that people build and defines them in with you know um in DSPI and then runs a whole suite of LLMs with a whole suite of optimizers with a whole suite of the types of different programs on a bunch of different data sets and tasks so that we get a sense of where we understand and what matters and how well do different things perform.
Um I don't have the time to actually explain uh the whole figure but the short uh idea here is that the program you build really matters but at the level of the structure of the system design not the little prompting techniques and then applying a an optimizer on top gives you you know that final really important sometimes uh cherry on top and in their evaluation mero was the uh best optimizer in their in their comparisons on average but moving beyond mero which has been great uh we've been developing the next generation of PSI optimizers for a while and we've been thinking about it really hard for quite some time.

31:24 SIMBA Optimizer

Um and so for the past few weeks u and this is something you can bet on. We always do this. Um we've quietly released and we've been evolving uh the next prompt optimizer. It's been there uh called DSpi. simba for the DSP power users which are whoever just we have a lot of those folks just find it. they look at the release notes or they just look at like the new directories and they start using it and they start complaining or they start you know giving us feedback and it's a really helpful way for us when we stand today to tell you hey you can use this thing it's been it's been used for a while it's a great way we do betas but in the open source community right it's based on extensive exploration from um and Arnov so um what is new about Simba one of the key fundamental things that are new here is that we want to highlight using reasoning to do the actual learning or to do the actual optimization.
And we found that this can be really essential for tasks that have a lot of steps, long horizon tasks. A lot of people call this agentic tasks. Um the way Simba works is well beyond the scope of today, but I still wanted to give you a flavor. I promised, right? So the way this is going to work is it takes your data set. It samples a mini batch. Uh it samples a mini batch like a bunch of examples and it takes your program and it runs each example through your program a bunch of times maybe you know um six times and then it looks at those um runs these rollouts which is very similar to how certain reinforcement learning algorithms work. um it looks at those and then it finds which examples had essentially high variance, right?
Which examples had the largest spread uh between runs that really worked and the ones that really didn't. And the reason it looks for those is that essentially they're going to teach us a lot. Um they're going to have like there's some behavior that you do that's unreliable but that could actually really help. Um and then it looks at those and it runs certain strategies for learning whose job is basically essentially to ask an LLM, hey, it worked here but failed there. you can see exactly what happened.
Can you tell me like what I should do in the future so that I get more of the behavior that works? Right? How do I kind of amplify the behavior that works? Um, and there's various ways of doing this that we've been exploring. Um, and adding to Simba. How do I use Simba? It's complicated. Maybe you don't really care how it works. It's just the same two lines. You switch Meo, you replace it with Simba. And that's the whole point of DSP.
It's like, man, we're really proud of you, Meoro, but it's probably time to move on. How do you move on? You just change the name of optimizer, right? And then you have the latest and greatest um u of the optimizers. Another really cool thing about Simba is that we've really gotten a lot of feedback of people where they're like, "Ah, my metric is giving the optimizer a score of zero, but I actually know why it was wrong.
I wish I could tell it." So we allow you in Simba to actually return feedback that tells the optimizer, yeah, this thing is getting zero, but the reason it's getting zero is that in step five, it called that tool and it's not supposed to or something like that, right? you have a judge or an automatic way of detecting that the optimizer can actually look at this and leverage it in its reasoning. So um on the tool hop task which is a task where you know basically you wake up in the LLM you know agent you're given a bunch of tools and you're given a goal and your job is to basically call these tools until you've uh you have the answer for the question.
So a simple GP GPT 4. 1 agent on this task gets 47%. Um you simply apply Simba on it out of the box and you get you know you jump to 70%. Um, and you know, there's a nice interesting kind of prompt here. I simplified it to fit in the slide and generalize it slightly. It says things like, "Hey, if you attempt to use a tool that raises an error, do not give up, right? Do not simply assume it failed. Instead, here is how you should try alternative things.
If all of the programmatic attempts fail, well, sometimes a lot of these questions are about popular items, you know, that you might already know in your internal knowledge. So, like maybe try to leverage that and see if it gets you any closer." That's the prompt that it teaches itself, right? um maybe uh you know you have a different task but it will learn something very different. Indeed, Simba has been picked up by power users um even though it's not really something we've sort of like officially kind of publicly announced.
So Mike Taylor, author of the book prompt engineering for generative AI by O' recently published a case study at his company Ascarelli for using Simba um for user simulation studies. It was the best performing out of the DSPI optimizers he worked with all of them being way better than the baseline. Um, and this was the prompted learn or part of the prompt at learn. Um, it's actually very interesting, but obviously I I'll let you kind of find it online maybe and and look at it.
Uh, we've also been seeing a lot of the same in anonymized R&D results we have here at data bricks where you know Simba is is you know clearly up and coming in terms of uh you know being an improvement on average across you know relative to me pro but it's still it's still evolving. So, um, kind of close to wrapping up. Um, I told you optimizers can optimize your prompts, but one of the things we're most excited about in 3.

36:09 Reinforcement Learning

0 is that we're doubling down on being able to optimize your weights. We've always been able to do it, but it's always been kind of a more peripheral thing. We've been investing a lot to make it incredibly reliable. if you have a local language model or if you're working with a remote LLM that you can that has fine-tuning APIs like OpenAI for example or data bricks model training um how do we actually get fine-tuning to sort of be uh applicable in your programs and so everybody is super excited nowadays about reinforcement learning there's no reason why if you wrote the beautiful DSPI programs that you can't apply reinforcement learning on top of them without changing them there's no reason you couldn't that's always been the case but now you actually can uh because we built this sister library arbor um led by and Notredam, Delara soil at and Lakia at um which allows you to do reinforcement learning over DSPI programs.
How does it look? Um you probably guessed it by now. It is simply you swap Meoro and Simba with DSp. GRPO which is the name of the specific reinforcement learning um algorithm that's actually behind a lot of the LLMs being trained nowadays. um but we've generalized it to allow it to fit any agent and any system you've built in DSP and you give it the same metric you usually give um to a DSPI optimizer give it the same uh inputs you give it to the DSPI optimizer and if your LLM supports you know working with arbor for reinforce reinforcement learning you get a reinforced uh a reinforced version um of your system or agent and you know like kind of a I I don't want to spoil kind of the details so I've anonymized a lot of these results and kind of just um you know we've taken them um without the task names, but it's very promising and in some cases it can be quite a bit better than prompt optimization, although it's a lot more expensive.
Um other times we find that actually composing the two, so allowing the SPI to do prompt optimization and then doing reinforcement learning is exactly what you need to do. And so we're excited to tell you all about um these in in subsequent releases um that the team is is is excited about. So very quick recap and to wrap up with this um pip installed 3. 0 um release. You can already do that today. Um some of the major things is we've been working on a mature ecosystem of adapters and base types that give you a lot of control really full control uh and extensibility.

38:13 Conclusion

Uh we've been working on multimodality making signatures the universal interface to LLMs. Um and we've been making sure that modules are as robust uh easy to productionize observable with a lot of native uh integrations with MLflow 3. 0 very very tightly. Uh, so we've automated a lot of the stuff that made Meepro slightly harder to use in the past. It's basically just, you know, kind of you just call it at this point.
Um, and we've released two really powerful optimizers, the future um, of the SPI prompt optimizers. And then we're investing very heavily as an open source community um, in the reinforcement learning direction um, for whenever you're using open models or models that you want to um, to fine-tune. And stay tuned for a few really key updates that are coming in 3. 1 and 3. 2 too that sort of didn't really make it in 3. 0, but we're very very excited about those.
Um, so I'll leave you here with the with the with the closing slide. I don't know how I finished on time. I'm sorry this was really fast. Uh, but if you leave with one idea, it's that like, hey, prompts are a soup. They're not really a good canvas for programming. We've been thinking for three years about how to actually sort of give you an abstraction um that allows us to actually turn back to engineering for these types of systems.
And that's backed up by three earlier years before that of building LLM systems from the days of BERT. Um DSPI fundamentally in terms of the signature abstraction has stayed the same since text Dainci 2 if you know if you remember what that was um up to 04 um or 04 mini nowadays. And the reason for this is that we're building abstractions around the fundamental ideas of programming like functions and the types um that allow function contracts to be defined um instead of like the latest fad or something.
And that allows you to get really powerful ideas like multi- aents as simple compositions of the three or four tools that we give you um that are highly interchangeable. Uh thank you all very much. I'll stick around for people that you know have any questions or discussions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.