What Is Automated Prompt Optimization? GEPA, MLflow and Agent Bricks Explained

Summary

• Prompt tuning: Treats prompts as tunable parameters to improve accuracy without fine-tuning.
• GEPA optimization: Reflection model and Pareto frontier achieve up to 50% accuracy gains.
• Model downshift: Optimized prompts let smaller, cheaper models match larger ones.

What Is Automated Prompt Optimization? GEPA, MLflow and Agent Bricks Explained

Talk by Hanna Moazam, Senior Solutions Architect, Databricks
Automated prompt optimization improves LLM accuracy without fine-tuning: instead of hand-writing prompts, algorithms such as GEPA treat the prompt as a tunable parameter and optimize it against your own examples and an evaluation metric you define. This video explains why manual prompt engineering breaks down at enterprise scale, how optimized prompts let smaller, cheaper and faster models match larger ones, and what the research shows, including accuracy gains of up to 50% in a Google DeepMind study.
Using a review-classification example, it shows how prompts fragment across models such as GPT-4 and Claude. You will learn the optimizer workflow (task definition, examples, evaluation metric), research results such as up to a 50% accuracy gain in a Google DeepMind study, and surprising optimized prompts, including one tied to Star Trek. GEPA uses a student model, a stronger reflection model and a Pareto frontier of best prompts, and can work with roughly 20 to 50 examples. Three ways to apply it on Databricks: Agent Bricks, MLflow prompt optimization (framework-independent, including LangGraph and LangChain), and DSPy for full customization.

Chapters

• 00:00 Introduction
• 01:11 Why manual prompt engineering breaks down: a classification example
• 04:16 Treating prompts as tunable parameters
• 04:40 What is automated prompt optimization
• 06:54 Smaller models, lower latency and lower cost
• 07:45 Research evidence: accuracy gains from prompt optimizers
• 09:27 Counterintuitive prompts optimizers discover
• 11:58 How GEPA works: reflection and the Pareto frontier
• 15:03 How much data do you need to optimize a prompt
• 16:40 Prompt optimization in Agent Bricks, MLflow and DSPy

FAQs

Why does manual prompt engineering break down at enterprise scale?

This video shows that prompts fragment across models like GPT-4 and Claude, forcing teams to maintain different versions. Manual prompt engineering doesn't scale, and hand-optimized prompts often fail when requirements change or new models arrive.

How does automated prompt optimization work?

Rather than hand-writing prompts, GEPA treats the prompt as a tunable parameter and optimizes it against your own examples and evaluation metrics. This video explains that GEPA uses a student model, a reflection model and a Pareto frontier to discover optimal prompts.

What accuracy improvements are possible with prompt optimization?

Research shows up to 50% accuracy gains from GEPA in a Google DeepMind study. This video covers how optimized prompts let smaller, cheaper and faster models match the performance of larger ones.

Where can I apply prompt optimization on Databricks?

This video describes three ways: Agent Bricks, MLflow prompt optimization (framework-independent, including LangGraph and LangChain), and DSPy for full customization. You typically need roughly 20 to 50 examples to optimize a prompt effectively.

Full transcript

00:00 Introduction

So, I'm Hana. I am a Senior Solutions Architect at Databricks. For the last 5-6 years, I have been working with corporate clients, helping them design and build AI solutions. During this time, I have seen again and again that making an initial effective demo is very easy and enjoyable . But translating this demo version into industrial operation, ensuring its reliability and scalability, is much more difficult. The reason for this is that enterprise AI solutions consist of many components.
One of the main components is the prompt, that is, the instruction that we provide to our large language models (LLMs). The most natural thing we are all used to when working with LLM is communicating with them. We speak to them in natural language. But it is precisely in this paradigm that the problem lies. So that's what we'll be looking at today. Let's understand the pitfalls of modern prompt engineering, why we need a more algorithmic approach, and delve into what it is and how to use it.

01:11 Why manual prompt engineering breaks down: a classification example

I think the best way to understand the problem is to look at an example. I'll show you a very simple task: let's write a small program to classify reviews. We want to get two classification results. We need to determine the tone of the review, as well as the category of the product in question. Imagine the reviews on Amazon. At first glance, it's very, very simple, is n't it? At least, that's what I thought. So, I start by giving my LLM, let's say GPT-4 in this case, an instruction.
I ask her to classify the review, give me a tone and category. But I get a text in response. Because I want to use this solution in a data processing pipeline. I need the source data to be structured. Well , that's not a problem. I can go back to my LLM and clearly specify the output format. So I'm adding that to my prompt. And that's great. This solves the problem. But now the model is coming up with other classification categories that don't fit.
I don't want to be bothered by things like " experience." I only want to classify a specific product. So I'm making the next version, the next iteration. I clearly limit my prompt. I am specifying models for what the categories should be . And eventually, after iterations of testing and working with beta users, it worked. But then I remembered that Claude came out, and it's quite productive, so let me test it with Claude .
And Claude decides to convert this into XML output. So, I want to be able to compare different models, but that means I'm now maintaining two separate queries. And then when I support these two queries, when I work on them, trying to get them to recognize sarcasm or other aspects or edge cases, I have to edit each one of them. And it's getting harder and harder. This is an example that I showed you in less than 2 minutes, but customers, engineers, you see them doing this for days, weeks, and months.
And the hardest part is that you don't know when to stop. You don't know if changing two words will make your query better or worse, and how to measure that? So, summing up the problems we saw on screen, it's really, really difficult. And that's very frustrating when you're building enterprise systems. It's perfectly fine to chat with ChatGPT about which restaurants are good in San Francisco, but when you're building an enterprise solution , you can't do that kind of manual iteration.
And if you think about it, I'm sure there are a few software engineers, data scientists in the room. Would a data scientist ever sit around and adjust the weights of their model one by one, just randomly? Of course not. So the paradigm is changing: we need to think of queries not as a static artifact, because as you've seen, they're not static at all, but as a tunable parameter, something that we can improve, and ideally do so algorithmically.

04:16 Treating prompts as tunable parameters

So we eliminate the human factor from this process. So, the solution to this is query optimization. And for those who haven't heard of it before, the simplest definition is this: it's the automated process of improving your queries using data-driven algorithms. Very simple and very similar to what we have in traditional machine learning as our optimization and training algorithms. And before we talk about specific algorithms or how they work, it's helpful to just visualize what this flow looks like and what it takes to perform query optimization.

04:40 What is automated prompt optimization

So, the first part on the left side is your task definition. This is your request instruction. Instructions for LLM, sorry. Much like how we started that classification exercise, simply telling LLM what I wanted her to do without doing anything special to make sure she followed that instruction—that's the definition of my task. Now, moving on to the optimizer, I will consider it a "black box" for now. We will go into more detail on this later.
But the two blocks above and below are extremely important. So, examples are what your optimizer will learn and test on. And what does the optimizer need from you to be able to improve your query? He needs you to tell him what " good" is. Because if you can't define your metrics, your evaluation criteria, the optimizer doesn't know in which direction to improve. Unfortunately, we don't have time today to delve into the details that evaluation metrics require.
But if you've heard about using LLMs as judges, about defining metrics— all of that is embodied in that block and is a core component of creating LLM-based solutions. So, with all this input, the ideal scenario is to get your optimized query. And if we compare this to the previous process we went through when building queries, from manual approaches to optimization, it's almost line for line. You see the difference in how your workflow has changed .
One thing I want to emphasize that we didn't talk about in our initial example is the last line on the slide: sometimes when building enterprise solutions, we strive for the best quality. And to do that, we are ultimately forced to use the biggest, most expensive, and slowest models . But you don't have to do that . Ultimately, you're just trading latency for quality. By being able to iterate and refine your queries, you can get advanced model-level performance from smaller models that are cheaper and faster, thanks to the optimized query for this LLM.

06:54 Smaller models, lower latency and lower cost

I'm sure there are skeptics in the room . So don't take my word for it. I will provide some evidence and research results on query optimization, as with the development and popularity of LLM, the field of query optimization has become the subject of much research. Honestly, I don't know when the next optimizer will be released , but it's a very active area of research. So, first of all, we all think about accuracy, right?

07:45 Research evidence: accuracy gains from prompt optimizers

We think about quality. And I took fragments from various research papers. Here's one of them—from a Google DeepMind study, where on a certain dataset, a benchmark, they demonstrated up to a 50% increase in accuracy. And this is an increase compared to manually created queries developed by prompt engineering experts. And this is not just done on this dataset. Similar results are observed in other industry tests. Similar to what we talked about on the previous slide, it's about cost.
So, especially when you start implementing LLM into your corporate workflows, into your daily work, the value of the tokens grows exponentially. Therefore, being able to optimize a query for a smaller model and get similar performance is a huge improvement for any company. Additionally, we have examples where we know that LLMs have a lot of good general knowledge, but how do they work for a specific topic or industry?
Previously, the answer was fine-tuning. But those familiar with this know that training requires hundreds and thousands of data points. You adjust the model weights, and frankly , this is mostly done for research purposes right now. Very few corporate clients will be able to afford additional training. So, by optimizing the query and setting it as a parameter, you can achieve similar, and sometimes even better, results than with retraining.
I wanted to give some examples to show why this is important, because initially when I came into this field, I thought, "Well, I'm sure the person who sits down for this—because we have the language, right? We speak English. If I had sat down, I'm sure I could have done better. I'm sure I could figure out what this LLM would take to get the right result." But one example from the research paper demonstrates how they tested solving elementary school math problems.

09:27 Counterintuitive prompts optimizers discover

This, again, is an industry standard. And the model they used was Llama 2. So this is data from 2024. And the query that turned out to be the most accurate was generated by the optimizer, and it was suspiciously related to " Star Trek". As a prompt engineer, I would never have thought that this would help the model reach a new level of quality, right? So, presenting this in a scenario where everything happens in a command center on a spaceship.
Similarly, another example from another scientific paper, I do n't remember what model it was, but the same dataset , and also a different dataset, and what this group of researchers found when they ran the prompt optimizers is that for this model and this task, the prompts that worked best were quite emotional, like asking the LLM to take a deep breath before doing the task, or saying that this is very, very important for my career—that is, appealing to emotion.
And again, that's the last thing I would think about as a prompt engineer. Sorry, I had some nice visual material that didn't show up earlier. So, I would never think that LLMs are emotional, and I don't know if Claude is emotional, but GPT-4 is—no, right? I don't know which iterations are worth going through. And sometimes if you go to some of these model builders' sites, you see these documents about optimization for different models, what might work and what might not, and it's just really, really hard to scale, as we've already discussed.
So now that we hopefully agree that optimizers are a good solution and, you know, they can solve the problem of enterprise scale and iteration, I think it's worth taking a closer look at one prompt optimizer to understand what's really going on "under the hood." And, as I said, there are many optimizers. Many of them appear very, very quickly one after the other. The current modern standard is Jeppa, so I thought we'd take a look at Jeppa to conceptualize a little bit what's going on inside.

11:58 How GEPA works: reflection and the Pareto frontier

So, Jeppa works on the principle that it needs a language model-student. This means, in simple terms, the language model that you want to use in your solution and for which you want to optimize your prompt. We call her a student because she also needs a language model for reflection. And this language model for reflection, or the "teacher" language model, is the one used for the reflection that Jeppa does, and I'll explain that in a second.
But this will help you get back to where we were talking about stronger, larger models versus smaller, weaker models. You can use an algorithm like Jeppa to tune your smaller, weaker model since you are not running prompt optimization every day. You run it when you collect a lot of data, you run it, you know , on a schedule. But you can use the power of a really, really strong LLM to optimize the prompt for a smaller LLM that you use for tasks that need to be done every minute, every hour, every day.
And we already covered in the initial thread that you need feedback, i. e. your metrics and a training dataset. Jeppa is unique in that when I say " feedback," it also takes a trail of reasoning. So, we talked about our reflection model. It doesn't just take your metrics. Let's say I defined accuracy as a metric, right? And I have a labeled dataset where I can measure this. It will take not only the accuracy score derived from my metric, but also the reasoning and path that led to that score.
So let 's see how Jeppa on the right goes through the optimization process. Um, she uses a complicated term like " Pareto frontier." It simply means the subset of prompts that are most effective. This is a series of mini-packs where it tests the different prompts it generates against a mini-pack of examples from your training set. And if one outperforms the one already in the set, it is displaced, and so on. So, it supports this frontier of the best prompts.
If I have more than one metric, it won't over- optimize for just one. It will leave prompts in this frontier that perform well by all metrics. This means I can see the results, review the optimized prompts, and decide which one to prioritize or vice versa. So, in the end, you get your best prompt or prompts, depending on how you set it up. Now that we've gone through the theory, looked at how Jeppa works, I want to point out one thing, sorry, before moving on to the next section: the first question I get asked when I talk about optimizing prompts is how much data do I need?

15:03 How much data do you need to optimize a prompt

Yes? We use LLM because we don't have such a clean dataset on which we could simply train our own model. And here I will return to Jeppa again. Jeppa is truly unique here. So, before Jeppa, most prompt optimizers needed over 200 examples to really ensure quality and measurable improvements. Thanks to Jeppa, because it uses a reflection level, it does not depend solely on the numerical results of your metrics. You can get tangible benefits by having 20 to 50 examples, which is the optimal number .
But it's worth remembering that there is no single answer to how many examples you need. However, it is important to note that your examples should be representative. So, going back to the basics of machine learning, if you over-optimize or your dataset is not representative, you are optimizing the wrong thing . This is what you should know about. The advantage is that you can continuously collect data as your agent or system is deployed, and your dataset for optimizing prompts will grow over time.
So tomorrow, if I decide to switch to a different model, I'll just run my prompt optimizer . Or, if I have a different data set, I'll just restart the optimizer and my entire focus will be on building the system, not writing prompts. Regarding the use of optimizers, since this is a Databricks conference, you may not even notice that prompt optimization is built into many products. For example, Agent Bricks, which Casey talked about in his talk yesterday , uses prompt optimization along with other techniques to create an optimized version of your agent.

16:40 Prompt optimization in Agent Bricks, MLflow and DSPy

I call this the first level. This is the easiest way to take advantage of prompt optimization , as you don't have to do anything other than provide input. This is a kind of "black box". She does everything for you. If you want to go to the second level, where a little more control is needed, you have the option to use the prompt optimize function with MLflow. This is a managed function. Now "under the hood" she uses Jeba.
If a better optimizer comes out tomorrow, we will upgrade the system to it. It is framework-independent, so whether you use LangGraph, LangChain, or anything else, you can apply it to your workflow to optimize prompts . Finally, if you need full customization, DSPy is one of the packages that pioneered the concept of prompt optimizers. You can build solutions for your agents using DSPy and host them on Databricks. Thank you.

Learn more about the Databricks Data and AI platform.