From Code Completion to Autonomous Software Engineering Agents

Summary

• SWE-bench standard: This video introduces SWE-bench, a benchmark built from real GitHub issues that measures AI's ability to autonomously resolve complex software tasks in large codebases.
• Agent fundamentals: Discover how SWE-agent—a simple yet powerful framework for software engineering agents—operates and how it adapts agentic AI concepts to coding workflows and problem-solving.
• AI in software development: Explore current state-of-the-art, limitations of today's AI agents and what separates AI-assisted tools from truly autonomous developers solving real GitHub issues.

From Code Completion to Autonomous Software Engineering Agents

Talk by Kilian Lieret, Research Software Engineer, Princeton University
As language models have advanced, they have moved beyond code completion and are beginning to tackle software engineering tasks in a more autonomous, agentic way. However, evaluating agentic capabilities is challenging. To address this, we first introduce SWE-bench, a benchmark built from real GitHub issues that has become the standard for assessing AI’s ability to resolve complex software tasks in large codebases. We will discuss the current state of the field, the limitations of today’s models, and how far we still are from truly autonomous AI developers. Next, we will explore the fundamentals of agents based on hands-on demonstrations with SWE-agent, a simple yet powerful agent framework designed for software engineering but adaptable to a variety of domains. By the end of this video, you will have a clear understanding of the current frontier of agentic AI in software engineering, the challenges ahead and how you can experiment with AI agents in your own workflows.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is SWE-bench?

SWE-bench is a benchmark that evaluates AI agents' ability to resolve real GitHub issues. This video discusses the benchmark itself and how it became the standard for measuring agentic AI in software engineering.

How does SWE-agent work?

SWE-agent is a framework that orchestrates language models, code tools and reasoning to autonomously navigate codebases, understand issues and implement fixes—similar to how a human developer would approach a problem.

How far are we from truly autonomous developers?

The speaker discusses current capabilities versus the frontier, highlighting that today's agents can handle specific tasks but face challenges with complex multi-step problems requiring deep domain expertise.

Can I use these agents in my own projects?

Yes, SWE-agent is open source and adaptable to other domains. The video provides hands-on demonstrations and guidance for experimenting with AI agents in your own workflows.

Full transcript

All right. So, it's really great to see so many of you here. I honestly didn't expect to get so many people since this is the 8 a. m. talk. Uh, to be honest, if this was an academic conference and your talk is at like 8 a. m. , you might just as well talk to yourself. So I hope that either you are morning people or you're really excited for this uh topic which I think is a really exciting topic. So maybe before we begin how many of you are working with software are writing code maybe quick show of hands.
All right. Okay. That's basically all of you. So in that case uh it might very well be that all of this research has already affected your daily life or will so uh in the future. So I'm really excited to talk about this. Um all of this is research mostly done at Princeton University even though now some of our team members has also gone to or to OpenAI. So um um but it's all research uh not trying to sell you anything.
Uh all the software we show is open source. You can play around with it yourself. Uh you can read our paper even though it's mostly outdated by this point. So this will be like the big update basically um one year after um we published uh uh this paper with a similar title which got a lot of stones rolling and um brought to you the first um software engineering agent. Um so a quick note on the structure of this presentation.
Um since this you know my background is academic we're going to start with what to measure which means talking about coding benchmarks um especially one called sweep bench um then there's going to be a brief section on agent basics what's an AI agent and you know like what are the variables that go into uh building one um then the third chapter is um our agent sui agent um software engineering agent and finally I'll talk about uh training language models to be better agents and also what's next in um you know coding systems with AI.
So um also if you have any questions in the in the middle just raise your hand even though you'd have to probably shout to get the question to me. Um I hope there will be time for questions at the end. Um, also there's going to be some live demonstration of su agent in the middle. So you can already place your bets on, you know, like whether that's going to work or not. Um, but with that being said, um, let me just, uh, jump in.
All right, so let's talk about um, benchmarks. And one of the oldest coding benchmarks in AI is or one of the best known coding benchmarks is called human eval. And what you have is basically you have this little um Python stop solution. You have an input variable and it tells you you know basically what that function should do and the solution to that um you know is to put in the function body. So this is what people have quiz AI to do and uh this is an excellent benchmark because you can evaluate it very quickly. you just give all of these problems to an AI, you wait for whatever it writes and then you just run unit tests to check if it's um if it's correct.
Um so this is also very on point for like when you when you want to test it for autocomplete for example because if I'm going to write this um function in my code editor and I've already like maybe filled in the dock string, it would be really great if um whatever AI I have running in the background can just complete the body. Um so very well-known benchmark, really great benchmark. Uh however, it's also you know it got saturated really quickly.
So you know like already like maybe two years ago or certainly one year ago. Um all of the AI system were at like you know 95% plus. Uh at which point you know you could say like okay maybe this problem is solved or at the very least you know adding even more small percent points on top probably doesn't correlate so well with uh user experience. Um, so you could say this was easy. Um, and one of the reasons that it is easy is that um, there's relatively little context in there, right?
So this is like a very, you know, self-encclosed problem. Like I only I don't need to know about any other files. This is all kind of contained just in this dock string. Um however uh you know when we since we we already and we are all software engineers or or people working with software uh you probably have the same experience that most of your time is not really spent you know writing software as much as this is like reading other people's code reading through large code bases thinking a lot uh about things and then sometimes we might spend a day and we only change a few uh lines on our very big um code base.
So the story um of agents for software engineering really starts with what do software engineers spend most of their time with and if we model a little bit this you know process of software engineering like especially if you're like uh working on a big project or an open source project uh often times you have a really big existing code base and at some point someone finds a bug so you know like maybe they they erase a GitHub um issue and there's one example here uh on the screen and at first you know like if it's a bug report you might be like I don't know works on my machine uh and then people you know like double down and you're like okay I guess I have to to look into this stuff so you know you start by uh reading code uh you start by trying to reproduce um the problem and at some point you'll probably have reproduced it and hopefully you've narrowed it down to where the problem is coming from and so you make some edits and then hopefully you test them that they really uh resolve except that this is kind of the very idealistic case and in reality or and then you're done.
In reality, it looks more like this. You're just, you know, wildly trying stuff. You're reading, then you're editing something. You're trying to fix the problem. Maybe it fixed it, maybe it didn't. Anyway, you probably have to, you know, iterate through this whole um process. So, this is really complicated because there's a lot of steps with that. And uh Sweetbench, the benchmark that I'm going to start this talk with, basically tries to capture this process or tries to give um problems that usually require such a complicated process to be solved.
Um and then we're going to task AI systems to do the same. Um so the way this benchmark worked is actually relatively simple. So we basically crawled uh GitHub for a lot of repositories. um starting with the most popular Python repositories and um essentially we collected pairs of um issues and the pull requests that um belong together. Um and then we now try to get a AI system to solve the same. Uh we basically give uh whatever AI system um we have we give it the issue and the entire code base um at the point where this um issue was first raised.
Um then we have the language model um do some changes do whatever it wants to do really. Uh and after it has done that it basically gives us also something like a pull request but in reality it's just you know file changes. Um and then the trick with this benchmark is that we evaluate the solution from the language model by running the same uh unit tests. And um basically in the initial crawling phase to build this benchmark, we looked for pull requests that introduced new tests that weren't um passing before.
So we can test that at least, you know, like some of the unit tests flip from failing to passing. And that's how we validate that the language model really um solved the issue. All right. So it's all um real GitHub issues. It's real GitHub issues from um top Python repositories on GitHub. For example, Django is uh actually a big part of the benchmark. Um and we validated with the unit tests that come directly um from the pull request that were called um itself.
So we can actually you might have questions about this. Um so you know like there's lots of you know subtleties um about this and maybe sometimes the coverage of the unit tests is uh not so great and sometimes they might be overly specific um and so on. So, so it gets tricky once you look in the details, but in reality, you know, in essence, it's it's still basically um just that. All right. So, first of all, real world GitHub issues.
And second of all, this is super challenging for AI systems. Uh especially was um you know, like one and a half years ago when when this first came out. Um basically one and a half years ago when when this was published um I think the baselines were like three or four percent um that AI systems could solve at that time which was so low that people didn't even really want to work on that because they were like okay this is hopeless.
Uh and the reason that this is so difficult um for language models to work on a problem like this is first of all you have an extreme amount of context right so we're basically passing in the entire code base doesn't mean that we just you know like dump all of this into jet gpt but you know the entire uh context um that the language model has to explore in some way or the other uh might be hundreds of thousands of lines of code.
So that's a lot to take in. um then it's a reasoning heavy task right so I mean most programs try to solve some more or less complicated problems and you have to reason your ways through it to understand it um and also if we think in terms of um agents there's an enormous action space right so even though the tools that we'll give to the agent in the end will be um super simple you know just the fact that there might be a thousand files and then once I get to edit um a file you know I can write arbitrary words in that, right?
So, it's not like it's the exact opposite of like let's say like a multiple choice question where like an AI has to decide between five things here. You can literally do you know like you can write books, you can do whatever you want, right? So, this combination of having an extreme amount of context of requiring reasoning and having this very large action space uh made this benchmark really really hard um at the beginning.
Um however you know like now one and a half years later it looks very um different and we are like starting to even saturate this benchmark. So this is basically the revolution that took place uh in the last one and a half um years. So these are a few spin-offs also of this benchmark that I'm mostly um mentioning um if you already are familiar with SWEBench itself. Uh we try to extend this a little bit to what we call bbench multimodel where we have front- end issues.
So, not only like, you know, like here we have like a little bit of an issue text and we have the code base, but then when you're doing front end um design, oftentimes your your bug reports might be more like screenshots, you know, like someone takes a screenshot of a button and the button looks wrong or something like that. Um and so we wrote the second benchmark um where every single issue um actually has a screenshot or has some kind of image associated to it which you know might be like complicated diagrams uh it might be maps it might be you know like here this is a syntax highlighting um uh project and you know like the syntax highlighting is wrong and so on.
Um so this is even tougher than the original benchmark but still um the best systems right now get 25% u correct. Um and if you just look at the development um in the last year so here a little bit of a complicated diagram but basically we're going to look at gray versus orange where gray was May 24. So that's basically one year ago and this is a few months ago. So within one year uh language models went from for example here from 23% to 58% of all the issues in this benchmark solved.
So we more than doubled it. And these 23% were also you know up from like just a few percent points um a couple of months earlier. Um, and so also this plot is, you know, like now a few months old. And ever since uh Claude came out with Sonnet 4, I think we're looking at uh like the high 60s to 70% here. Meaning that from all the GitHub issues and pull requests that we crawled um one and a half years ago, the AI can now in almost 70% um of the cases um solve them. as in like it produces a solution that passes all the unit tests.
Um so this was an insane amount of progress and I think it absolutely demonstrates that in the future um autonomous agents will solve a lot of our um issues and bugs. Um I should also caution that this number is a little bit hard to interpret because of course it's a curated data set right I mean we just crawl all the issues and pull requests but you know just by the fact that uh they need to be self-contained and they need to have these unit tests it's still a somewhat biased selection because all selection you know necessarily induces some bias.
Um also it corresponds to you know it's a mix of different repositories and the solution rate between these different repositories um you know like they they are quite different. So um even though like this is kind of like an average if you apply to your own project depending on what you're doing exactly you might see different numbers you know like they might be half or they might be higher um it really depends what you're doing for example I think if you're doing something that has a lot of mathematics in it probably AI is not so great at that it really depends what you're doing but still you know like I would say like if you take an average GitHub repository um with a large user base that we get like very um well formatted very complete issues.
Um it might very well be that right now we solve like 50 to 60 um% of the of the issues. All right, any questions to benchmarks because I think um I'll I'll jump to ancient basics. We also will have time uh for questions later. Good. Um so let's talk about how to solve this thing. Um and so maybe first of all start with the with the stupidest way of doing that. you could attempt to just take the entire repository together with a problem and basically just copy paste it all to Jet GPT.
So you would just do like this oneshot thing. Um so certainly one and a half years ago this was entirely impossible because it would literally just not fit into the context windows um of the language model. Uh nowadays where we're looking at you know language models with huge amount of context that you can put into them, it it's probably possible to do that. Um but still language models gets worse the more context you you put into them.
So so probably you're not going to get and it would be very very very expensive. So you would not get a realistic system and probably would also not work well. Um so instead what you can do is trying to essentially mimic um how humans approach this problems and humans don't just read the entire you know repository from first file to last file and then they directly write the solution. Uh but rather we we kind of navigate around in it and perform actions.
Um so the simplest way you could do that um would be to essentially put the language model into the terminal. Um so you know you start with a little prompt you know maybe find all text files in this directory and do whatever with that. Um and then the uh the AI agent it basically takes this prompt and the output of it um might be some some thoughts some some reasoning and then an action like here for example the agent runs pwd to determine um which um path it's on um in the terminal um and then we give back the um the um whatever the terminal would show you like the output of this command and we basically pass it into the name system.
So what you get here, you know, it it looks very similar to a chat, right? And this is great because language models have been trained to be chatbots. That was one of their first um you know, like the most popular uh use case, right? So so basically we mimic the structure of being a chatbot except the you know the agent uh chats it an action and then the response it gets is the environment output. Um however and you know like this is you know we have to talk about how how dated or not this is but bash commands are hard right so which is also why humans don't really like to work in pure bash like or like something like like vim is is you know already something that's built um on on top of it um and it was the same um when we initially um tried this if you just put um the language model into the shell um it's just not going to do too greatly Because already like something like editing a file when you can't use you know interactive things like whim uh it gets quite complicated.
You can like you know do complicated search and replace things with set and so on. Um but one of the biggest failure modes also when when we tried that was that the agent would just cat like display in an entire file and if that file is really really huge then you just um fill most of the context window um and performance degrades. Um so the premise of this uh paper which was called uh sui agent for the agent that we built um was around building an agent computer interface just how humans have specific software specific interfaces with which we um edit code and it looked something like this.
You have the uh language model agent um we provide it with some special commands which are also just bash commands but specifically engineered for the language model. Uh so it can run any bash commands but also some specific commands for uh editing files in particular. Um then these uh commands they just run on they run in a docker session. Um and for example you know give back all all the output that you would usually see.
Um except that we also try to make it as friendly as possible for the language model which for example means uh looking to that never exceeds a certain amount of output lines or something like that. Right. Um so so back then so this is April 24 um we saw a huge gain with giving it basically a lot of specific tools. So, so for example, one of our baselines there was like 10% when we only gave it um um the shell like like just let it execute bash commands and you know like one of the frequent failure modes as I as I said would be like it would just you know like to look at the file it would use cat which is perfectly normal for a human but then if you get like 10k lines back this is really bad for a language model.
Um whereas these specific um commands that we engineered, you know, like they they would um for example make sure we don't overrun that and when the when the agent wanted to edit a file, we did some linting on it and we didn't let through any edits that were obviously wrong just because we're recovering from bad edits is harder than not making these bad edits in the first place. And we basically doubled performance with that.
Um however, since then, you know, things have changed a lot actually. So first of all, you know, like this is pretty specific, but like um we we use a very different way of editing. We basically edit it based on line number ranges. Um and now like a lot of these tools that people give to their AI agents are more like search and replace functions that run under the hood. Um and back then, you know, like language models were still pretty stupid.
They were doing like a lot of I don't know indentation errors and obvious syntax errors. So we had this whole thing about linting and blocking edits if they were obviously destroying stuff. Um but now you know like set for doesn't you know like doesn't do syntax mistakes or very rarely. So so you don't need to you can be even simpler than before. Um I would argue that still like editing if you're you know developing a software engineering agent like getting the edit function right is still um the biggest part.
However, and you know like this is you know going only in the last two months. We actually tried to remove all tools and basically did this thing which I just called the you know put the language model in the terminal recently. And you know what it works just fine nowadays. So uh you know you're going to sacrifice a few percent points in in your final scores. But this actually works. Um but it also works um because people you know especially people of enthropic um have worked really hard to to basically train for exactly this use case.
And we're going to get a little bit uh into that later how how you train language models to be good at that. Um but yeah nowadays you can literally just do that. So so the big takeaway lesson is that um software engineering agents nowadays can be super super simple. um it it really um you know it's different if you want to want to be state-of-the-art or something but this is only a five for a few percent points and basically language models are just that good and have been um optimized for these things.
Um you know let's look a little bit into um how an agent works in a bit more detail. Um, so you're going to have an initial prompt uh where you explain the mission, the strategy, you might give tips. Um, it can be super short though. So back more than one year ago, we had like uh maybe like it was one letter sized page of prompt because we had to make sure the agent didn't do stupid things. Um, so we were like giving a lot of like important, you know, like if a command doesn't work, don't try to run it a second time and you know, like stupid stuff like that. uh no longer really necessary.
Uh can be super brief. Um you know, if you're using function calling, which is one of the things that language models nowadays offer, which is directly optimized for HTML workflows, you don't even really need to explain what the function does and it gets it itself. Um the second thing that we did back then, um which you might still sometimes do, uh is demonstrations. So basically, in addition to this initial prompt, you give it what we call the trajectory of a successful run.
So you show it basically one run where an agent did something really useful and it was really successful and you just basically paste that in with the um initial um prompt. Um I think I wouldn't nowadays do it usually unless you have to because it also means that you know it might lock the agent into very specific ways of thinking and try to solve the problem exactly the same way as in the demonstration which is why it can be harmful. uh but it can be super useful if all of your you know if all of the problems that you do want to solve you know fall into one pattern then giving one um successful example of solving something like this um is super super useful um or if you're using an open source model so most of this is from the point of view of using like the current big models like um an anthropic anthropics cla or or GPT40 even though it's doing not so great or like these kind of models u if you're using um a open source model um maybe adding a demonstration in these things um can can still help a lot especially like if the language models um struggle with some formatting of the actions and stuff like that.
Um but basically what happens after we have these initial prompts there's going to be a system prompt there might be a demonstration then we paste in um the problem that we're solving and then basically the language model replies with a thought and an action where thought might just be any you know motivation for what it's um doing. uh we give it back the response. Um you know like sometimes we strip out things if if things are definitely wrong but mostly it's just this pattern of like um action response action response all at every step are just concatenated to the prompt.
Um which also means that you know like the first step is relatively cheap because you just you know like this is your input. Then um once we start adding more and more actions and output to it, you know, like basically what we paste into the language model at every single step gets longer and longer, right? So at first you just have the initial prompt and one action and output. Then you have the initial prompt and two actions and output.
So with every step um you know this gets slightly more uh expensive and there's some ways um around that but but usually you know agent runs like the naive way would be to say that they kind of scale quadratically in in costs um so not cheap and in the end the agent basically calls the submit command and uh quits and says it's done. Um and this is an actual I think this is pretty much our um current prompt. Um, so, so it's really not long.
It's literally just can you help me implement these changes? Uh, here's how you should proceed. You know, find and read code. Uh, create a script to reproduce. Uh, edit source code. Uh, try and run your reproduction script again. Uh, think about edge cases. Um, and that's it, right? Uh, doesn't take more than that. Um, I think one of the most important things here is honestly this um, create a reproduction script. I think honestly this is why where a lot of the performance of these um agent runs comes from.
It's that the agent has some way of checking if the solution is correct. Which also means that if you have an issue that doesn't really allow it to get to a reproduction script. Um an agent maybe just like a human is having a hard time fixing it, right? Because you just can't check if you really fixed, you know, whatever was wrong. Um I don't know I already mentioned that you know the history gets longer and longer. Um you can try to mitigate that.
Uh we call this history processing. Basically at every step you could you know like take a selection of all the all the things you pass um into the language model. Um, for example, what we did for the initial release is that, you know, we we add all of these um actions and outputs and we would just strip most of the outputs except for the for the last five. Um, which really helped us to conserve context back then. Um, it nowadays we don't do it anymore because sonnet 4 can just handle the amount of context and it doesn't help and it just makes it slightly more complicated.
Uh, and also I mean if you're looking at our newer language models, they have caching. So every time you know you send in more or less the same history plus something else. This will be cheap because this part that stays the same you don't you have you pay less for it. So if you continue to fiddle with your history you're just going to break the cash all the time which makes it more expensive. Um so yeah it depends. Um you know like some people you know like if you use for example cursor or something I think if the history gets very long they summarize the history at some point.
So there's like one point where history is compressed. Um I think that's a that's a useful thing to do um if you if you're dealing with very very long trajectories. Um and then you know the only other ingredient that we have is you know some some requering. If there's like format errors or like the language model doesn't even give you an action then you just ask again or or stop stuff like that like you know stopping after too much money.
But yeah basically this is a software engineering agent. um you know we bas basically now just developed one. It's pretty simple probably you know like for a very small baseline you can probably write one in an afternoon if you want. Um okay I think this is mostly going to skip over this. Um but um another thing you should you can um see is um if you run the same system multiple times on the same problem uh chances are you know it might get it right in one of these cases right so if you try a 100 times the chance is just much much higher than you just try try once.
Um however it's not super easy to determine which you know which submission was correct. So um I think here on on the left is from the original paper. This was the scaling where we just basically did kind of best of K. So we maybe tried six times and then checked if any of them solved it. And you see that back then this was almost double as high. However, for most real world um systems, you know, just trying multiple times also requires you to actually pick the solution that was correct.
Um because you don't have an oracle to tell you which one was was right. Uh and this is not easy uh at all. It might be just as hard as solving the problem if if your problem solutions are as good as they are nowadays. And you know it's just super expensive to run you know agents are not cheap. Running them 100 times you know gets prohibitively expensive. Um so there's some people working on that um on you know how to classify which solutions especially to something like this um are correct and get slightly better scaling.
Um honestly my bet would be that this is not going to be a big thing uh just because systems are getting better and better. So also what you gain from trying again I think like it's getting smaller and it just you know like your price increases linearly um what what you're paying but like your solution rate just you know it doesn't increase so much anymore I think. Um right. So um running out of time here, but like just a few tips um uh on on how to build agents.
You know, we went through this, you know, you take a prompt, maybe demonstration, probably not. Uh you add some tools for it. Uh you know, like the one thing you need for a software engineering agent is like this remote co like the um the sandbox code execution, right? the agent wants to execute code to test it and you should make sure that if it does something stupid it doesn't wreck your computer so this is actually where most of the engineering um goes in the rest is pretty simple then you handle exceptions u maybe manage context um and that's it um so tips I would say stay super simple um just because also language models are still evolving right so if you if you tune it really really really well to a language model that exists right now and then two months later there's a new one, none of your prompt engineering might apply anymore.
So, um or or none of your your very complex scaffold that you're building might might apply anymore. So, I would say better error on on just making it very stable and then just wait for the next best language model to arrive. Um the other thing is uh you know like it's mostly my experience is like making experiments for these things uh is not super easy. Um, and you should especially be aware that, you know, if you just try something with 50 instances, like 50 test cases, and maybe, you know, like you do some changes and gets like two or three more, it's just not statistically meaningful.
Um, and you kind of spend a lot of time just chasing these specters because everyone kind of, you know, like there's a big confirmation bias in all of us. So even if you know what statistical uncertainties are and you know like there might be like a 10% range at 50 instances you know we still want to believe right. So so I would this is another reason why why I would say keep it simple. It's just because optimizing these things uh it's expensive it takes a lot of time.
So, I I would rather place my bet on a few simple um things that I'm trying and then test them really well. Like test them on as many um test cases as I can to make sure they really generalize and aren't overfitit to anything. Um and the other thing is um I think to optimize it reading the trajectories that means reading you know like this you know long sequence of actions and outputs. I think that's the best way to optimize it because you'll just you just need to know what the agent what the language model is doing, what it's struggling with and the best way to do is you know like write yourself like whatever you you need like as a small software to quickly browse through through these trajectories and read as many as you can.
All right. Um only having se seven minutes left. Um so mostly going to flash this. We might have to skip the demo so we have time for questions. Um so we agent that was our system that we built originally um more than a year ago. Um it was the first open source software engineering agent that really did something as it solved like 20% um of issues. Uh we released it I think two weeks after made a big splash with its demo.
So Devon is is a company that's uh you know somewhere close close to here. Uh and they were basically building the same thing in secret. Um we had been working on it in secret for half a year. Uh they released two weeks before of us uh before us but we are open source so I think we can claim we are like the first open source system and then we had almost the same performance as they did um when we launched. Uh and this is still going on.
Um so it's now one year later. Um it still stayed roughly the same as in like the you know how what it really does. Um, even though I more or less rewrote the whole thing from scratch, um, you can check it out. Um, it's open source. Uh, it's very easy to configure and to hack. So, maybe this is the one thing, um, I can very briefly flash. Um, you know, oh, it's a little bit small, but basically remember when I showed you, uh, sorry, this the default when I showed you the prompt, you know, like this is the entire prompt.
Now we're giving into the uh language model. This up here is a little your helpful assistant. That's a system prompt. Um you know here we configure which tools we give it. Um each of these tools is just a simple you know it could be a bash executable. It could be a Python executable really nothing special. Uh and this is already like kind of esoteric. We're just like you know like when the agent submits we can show it one more message or many more messages like and ask it to do a little bit more more um double-checking or something.
But so so in reality like those first 30 lines are really all all you need um in in addition to a problem statement. And so when you run it uh it looks something like oops that's wrong looks something like this. Um it's going to start up uh it's basically spinning up a docker container. Here it says environment initialized and it's you know uploading basically our tools which are single executables. It's going to copy them into the um container.
I'm not sure if I have internet. Oh yeah. Okay. And now you see we basically paste it in this initial prompt. And now the language model proposes actions like looking at this directory. Uh and then we just continue you know like waiting for it to give us actions. We execute it then uh show them the output and so on and it's going to continue to do that. Um, and you know, like there's very very uh there's a lot of things you can can do.
Um, for example, you could try to just remove all of the tools, see if it still, you know, gets it solved, which it probably will. You can even get rid of most of these instructions. Um, and it's probably still going to do relatively well. Um, partially, I think, because of this is with Sonnet 4, and it's just uh just tuned to this kind of task. All right. So, I'm going to run through a couple of these things, uh, including the implementation, which is sad.
Uh, there's also a batch interface, which I didn't just, um, show you. Um, if you have like a thousand issues or you're evaluating on a benchmark, you can just spin them all up in parallel. So the last thing and this touches our most recent project is so far we've only talked about like basically we had models as a black box and then we added some prompts to them and we added some like scaffold around them to basically make them iterate.
Um the other question is like training language models and so there's um usually three different stages to training. Uh we have pre-training which is super expensive and you just have this huge amount of unccurated data where the language model is just trained to predict the next word and we imbue the model with knowledge. And there's fine-tuning which is the same thing except that we have like a super small amount of data but it's very highly curated and we can shape the output and maybe like have it long thoughts and these kind of things.
And then there's reinforcement learning which is dark magic and this is all what I'm going to say about it. Um and so historically, you know, like a lot of people when they're thinking about ancient performance, they were thinking a lot about reinforcement learning where you basically give rewards for, you know, completing a task. Um however, um fine-tuning actually works remarkably well for enforcing uh you know, for shaping models.
Um, and um, you can relatively easily fine-tune, let's say, a 32 billion model. Uh, if you have, you know, a couple of H100s and a day to spare or like a day of runtime, this is enough to fine-tune a 32 million parameter model uh to be a good agent. The only thing you need uh to have uh is good data. Um, which means basically uh kind of similar to the demonstrations that we were talking about like you need examples for the model to see how to successfully solve something.
And this will be the hard and expensive um part. Um, you know, reinforcement learning is maybe even more expensive, but like fine-tuning basically getting the data is is is what's difficult. Fine-tuning um is pretty simple. So I only have one minute left so I'm going to skip through this but basically we had this um project called Swmith where we generated a lot of synthetic problems to solve then had like thousands and thousands of agent trajectories and then we fine-tuned a small 32 billion model uh called Quen 2.
5 um which you know by itself you know at the beginning has like 10% performance on SWEBench um and we fine-tuned it and it got to 40% which um at the time where we release was the best um open- source model for SWEBench outperforming actually even GPD40. Um so you can you know like basically bottom line is um fine-tuning works super well uh you know it's still going to be a lot of effort in getting the data. All right so summary running out of time this bomb will soon explode and switch to red I guess.
Um so Sweet Venge um challenging benchmark even though you know like it's like progress has been so crazy that it's you know we're looking at 70% success rates there uh already. Uh if you want to build an agent uh keep it simple, you can stay simple. Uh read lots of trajectories to find out what doesn't work yet. Um if you run experiments like you know be aware of fluctuations and everything just make them big and meaningful.
Um, you know, if you want to check out our project, uh, they are all on GitHub. Um, you can go to, unfortunately, all of the URLs are cut off, but if you Google SVEN or SV, you're going to find them. Um, open source easy to hack. Uh, maybe not the most user friendly because we are an academic project, but they are easy to run, I think, and they're the easiest ones to experiment with, I think. And finally, the last thing I talked about was this uh project about generating large scale fine-tuning um data, which I I guess you know this will get you take you some time to get into, but if you do or you have a very specific company thing that you want to train agents on, um I I think this is a is a cool way to do it.
All right, so the time is at zero. I don't know if that means we have no time. Oh. Oh, I I see like Okay, no questions. You can you can find me afterwards. Uh, and I'm happy to answer anything here in the hallway or something. Thank you so much.

Learn more about the Databricks Data and AI platform.