Skip to main content

SWE bench & SWE agent | Data Brew | Episode 44

Summary

  • AI evaluation benchmark: SWE-bench assesses AI models on real-world software engineering tasks, addressing data leakage and scope concerns.
  • Automated code generation: SWE-agent demonstrates breakthrough tools for enhancing AI capabilities in software engineering applications.
  • Performance assessment: The tools address benchmarking challenges to measure how well AI agents solve real coding problems at scale.

SWE bench & SWE agent | Data Brew | Episode 44

Watch: SWE bench & SWE agent | Data Brew | Episode 44
In this episode, Kilian Lieret, Research Software Engineer, and Carlos Jimenez, Computer Science PhD Candidate at Princeton University, discuss SWE-bench and SWE-agent, two groundbreaking tools for evaluating and enhancing AI in software engineering.
Highlights include:

  • SWE-bench: A benchmark for assessing AI models on real-world coding tasks.
  • Addressing data leakage concerns in GitHub-sourced benchmarks.
  • SWE-agent: An AI-driven system for navigating and solving coding challenges.
  • Overcoming agent limitations, such as getting stuck in loops.
  • The future of AI-powered code reviews and automation in software engineering.

Connect with Kilian Lieret & Carlos Jimenez

Chapters

FAQs

What is SWE-bench?

SWE-bench is a benchmark for assessing AI models on real-world software engineering tasks. It provides standardized evaluation of AI agents solving actual coding problems at scale.

What is SWE-agent?

SWE-agent is a tool for enhancing AI capabilities in software engineering. It demonstrates how AI agents can be improved through specialized tooling and agent design.

Why is data leakage a concern in benchmarking?

Data leakage occurs when training or benchmark data includes information that shouldn't be available. SWE-bench addresses this concern with proper data isolation and scope management.

How does this advance AI in software engineering?

These tools enable rigorous evaluation of AI coding capabilities and provide frameworks for improving AI agents in real-world software development tasks.

Full transcript

0:00 Introduction to Data Brew

Welcome to data Brew by D bricks with and the series allows to explore various topics in the data and AI community and whether we're talking about data engineering or data science we interview subject B experts to dive deeper into these topics while we're enjoying our morning brute I'm a principal developer advocate here at datab bricks and one half of data bur and hello I'm W director the machine learning practice of datab bricks and the other half of datab brew and today we are thrilled to have the co-authors of s agent on datab Brew today we have Killian larett research software engineer at Princeton University and Carlos Jimenez as computer science PhD candidate Princeton welcome both hi before we dive into s agent perhaps let's take a step back I know you're the first author on S bench perhaps we could start off with what is sbench and what was the motivation behind creating sbench yeah so sbench is a EV to evaluate language models and in particular we're trying to evaluate language models on their ability to write code in a more production like setting prior to sbench a lot of the coding evaluations around language models focused on settings that were more like interview question type problems or maybe like coding competition type problems where the problem was maybe self-contained and the language model was supposed to write maybe one function that did something like finding some optimal solution to a particular thing in a algorithmically efficient kind of way and but in the real world a lot of people's actual time spent software engineering is less on this algorithmic isolated solution problem solving and more on maintaining big software systems making sure your systems are bug free they're they can be large on large complicated code bases and so with sbench the idea is that we want to find a way to have some of the same types of problems where you can provide a language model with a specific isolated problem but within the context of a large code base and the way we did that is essentially is scraped issues from GitHub and these are these are types of problems that a lot of real world software Engineers spend their time spend their actual time in G jobs fixing and essentially packaging it to present to a language model so that they can we can evaluate their ability to those problems in a production like type setting and how do you avoid the issue of data leakage if you're scraping all these GitHub issu sometimes people will include useful snippits in the comments how do you avoid data leakage when you're trying to evaluate these models yeah so that's a good question I would say in some sense it's very hard to avoid contamination or or leakage in a very strong kind of strict way everything on the internet is up for grabs for these for these company or for people training language models and so it's almost I would say it's near impossible to get really clean uncontaminated data sources now with something like sweet bench however we are using some logical structure in how these issues are are organized within GitHub such that often times the solutions don't necessarily show up right next to the problems and unless you're scraping in such a way to really pair these two things together it might not be it's not necessarily that easy I think to inadvertently cheat on sweep bench problems that's not to say that contamination doesn't exist it probably does to some degree but I don't think that the way that these models are trained I think unless you're explicitly training models to solve s bench type problems it's most of the performance is not directly from contamination we also have this breakdown right like over time right there's the number of resolved instances over time there's various effects that can be found there I think for some of our benchmarks I think for multimod we actually found that the newer the issues were the more we would resolve like the opposite of what you would expect from contamination so certainly there's like a lot of other effect it might just be that issues get in big repositories get easier over time I think so far we've never seen this trend that you would expect from a clear contamination thing where you know like suddenly for some especially like all the years you suddenly see like a dramatically Rising the resol rates then one thing I'm curious about because you noted your description about sweet bench is to be able to look at in totality right to look at the problem not just an isolated function but there are plenty of GitHub repos for example that within the repo you it represents multiple modalities or multiple problems right or and or in some cases it's not just the individual repo but within the org or for that matter some of are tightly coupled with other repos I'm just curious is there something about sweet bench you specifically resolve for that or in fact the reality of this particular situation because of all of these potential connection points both either improve or decrease the likelihood of this type of contamination that you're referring to yeah hm I think maybe it's a little bit hard to think about it at that level but generally speaking like the scope of a single problem in sweep bench is just going to be like G A GitHub issue and so it's usually going to be some specific bug and that bug might be isolated in some way or It Maybe not maybe it's something that like T actual resolve that you have to change multiple different functions across different files or change the structure of of your code base in some way so the complexity can really range from very simple isolated problems which still happens within the context of the big codebase even to much more structural bigger refactors necessary to fix those types of problems and I think it's a little bit problem specific but generally speaking I what I would say in terms of contamination I just think that the way that the way that language companies are maybe scraping data from GitHub in order to train their language models are not really optimized to solve swe bench types of problems and so while it definitely might have seen like the literal text in most swe issues for instance it's probably not unless you're training to solve swe bench problems it's probably not like a humongous Advantage because the PO requests and issues aren't necessarily paired right so it might also see a lot of PO requests but doesn't necessarily see often the PO just linked to the issue but unless it's you know specifically follows the links it doesn't see like this problem solution pairing right yeah exactly you're implying that people in GitHub aren't super organized and Ure everything clarified no obviously I say that in just because of course my codee's perfect but not even remotely close to true but I'm what I'm curious about is then related specifically s bench before we make that switch to S agent is there like what is the makeup of the data sets IE the GitHub repost that typically comprise of it is there a particular language set that you're targeting is there or is it just more the entirety of all of GitHub I'm curious as to how the data set quote unquote is built up for sbench so for sbench it's we it's mostly python repositories the reason for that I would say is mostly because of our famili one it's a very popular language and also our familiarity with python is like much higher for them than for most other like languages at least recently and when we were building swed bench one of the big parts to make swed bench work is essentially making sure that you can build like the execution environments for these repositories and because we're better at python than other languages it's easier for us to build those effectively so yeah in a way sbench is is biased in that sense that there's the emphasis is on python repositories but I think if you have a system if you have a system that's trying to solve Street bench problems and it's sufficiently General that doesn't rely necessarily very extensively on python specific features then I think the types of problems that you'll enounter in Python are not radically different from those that you encounter in other languages so you should also mention that there's sweet bench mul model which we mostly yeah developed to see how agents can incorporate images in issues to solve problems but this is all JavaScript many ways that's those are both different problems because of they are more like frontend oriented because that's where the images come in but it's also a different language and we did definitely s see when evaluating Frameworks there that some Frameworks are overly specific to python which for example s is not because it's a very you know I think Common Sense agent scaffold which gives the agent a lot of freedom to explore and it's really only Guided by you know the initial prompt that tells that it has to solve um an issue and gives a few tips but other than that the agent goes around and explores files and does whatever a human would do to solve the issue so there's other approaches to solving sweet bench style problems that a little bit more rag style they they try to accumulate all the context they basically um have a lot more Tools in place to accumulate the content that context that they need to solve the problem but of course these tools will be very specific to the language like in Python you can like you can check where things are imported from you can try to pull in these sources and so on but you're going to end up being very specific to the language so that's something how you can end up being overly tuned to a specific language and so sbench is an open source project you mentioned it expanding from mainly a python Focus to having some JavaScript Focus what's next for sweet bench yeah so I think right now a lot of our emphasis is maybe more on agent side I don't know how much like we I don't know how much I can I I want to stay on record about like future ofal kind of projects so there's may be some things happening but I think that right now our emphasis in terms of what we're working on is a little bit more like enabling language models to tackle these problems in an gentic way that's a fantastic transition to S agent uh perhaps kilan you could give us the pitch for what is s agent very simply put it's it's our way of solving sweet bench style problems and I I think I already hinted at some parts about the nature there it's it's a very basic loop I you start off the asan with an initial prompt you you give it the problem statement you tell it about the tools that has at its it has at its availability and then it proposes the first action uh we execute the action for the agent we we tell it what the result of this um action was and then just proposes the next action it does so until it believes that has solved the issue or it has run out of um out of money that we that we give it and that's it basically so the magic here is of course there this initial prompt but then it also comes down to the tools for example like how can we make the agent interact with your computer or with a code base I think that was the big focus of the original Three agent um paper it was this notion of an ancient computer interface or ACI like just how we humans for example we use maybe rim or vs code or however you want to interact with our code base but we don't we don't just use the Linux comment line and like file manipulation with Echo and set and I think take that because that's the context in which Z agent was built was that before that like the simplest thing you can do is give tri4 or or whatever we we use at that time access to the common line and it's going to find all these crazy ways of editing files it's going to spend a lot of its thinking and energy just about how to make edits and obviously this is not where your thinking power should go similar to a human we want to not think about typing that's why we like to touch type we just want to you know think on on the higher level questions so that was the emphasis and still one of the emphasis of s agent with the new version that's up since maybe two months or so now it's even easier to exchange the tools so we've organized tools and Tool bundles it's very easy to give it different tools and we have various spin-offs of 3 and for example with cyber security where we equipped it now with cyber security specific tools and can just run with that I think that's the big strength of it like it's not really overly optimized to one specific task that's really interesting is there an implication that the agent itself is actually leveraging the intelligence of a tool in order to get access to the properties and methods of a particular language I fully understand you mean that it uses tools to actively explore the code base as well and if you give it for example with the cyber security things we did we gave it GDB the debugger for C++ and was able to use that even though that's like more an interactive Style thing but it can use it to really to a human run experiments or investigate the code Bas whatever way give it access to oh no actually ironically that's exactly what I was referring to I was just I was just pretending the agent was almost like a human and they're literally typing it then it pops up with the property methods but exactly to your point it's actually using the debugger to actually figure it out which Y is precisely the point but probably not as slow as me typing in nvs code something like that but then I'm just curious like why can't you implied it already but why can't you just say no let me just take a Frontier Model gbd4 or Claude or whatever and just toss it there and be done for the day why can't why specifically this idea of an agent instead of actually utilizing a Frontier Model so I think that question is all about context and in or to solve a problem you need the context you need to know the relevant parts of the code that are respons respons for The Bu you're currently trying to solve and there's different ways you can go about that so it's not completely impossible to just use for example C GPT and a human could do that too if you're sick at a certain problem you've identified the five um areas um of your code base that are responsible or likely have a connection to the problem you could copy paste all of these five different things in the chat TPT and then ask if your your fin question where does the where does the problem come from and you'll maybe you you'll get your result however you first have to get that context and there are projects out there some of them doing very well also on the spe bench leader board that use static analysis of the code base so without a language model in the loop or maybe with a language model they might just you know very stupid way of doing this would be to First give a cheap language model every single file and ask whether it has a connection to the problem but anyway you accumulate this context and in the end you're going to hand over the context and the problem to the language model and ask it to propose an edit you can do that I think you're going to invariably be much more specific to the kind of problems you're solving that's one thing the other thing is you can't iterate on something so with s agan after it proposes after it thinks it solved the problem it tries to verify it just like a human would have a reproduced at pie it tries to create a reproduced at pie and it's going to re-evaluate it to see if it really solved the issue and if it hasn't it's going to mess around with things a little bit more so if you don't do this then to make up for that what you can do is you ask jbt 100 times and try to pick the best so there are there are ways to do this in a non authentic way it's going to be different kinds of tools and I think we can talk about that later not all tools are right for the it depends on really your and there might be like a multitude of different tools of different approaches coming in the future to help programmers and so one thing that I know that can be a challenge for agents is sometimes they can get stuck in a loop I know things might be changing with some of the new reasoning models that are coming out but just curious how do you avoid issues where the agent get stuck in a loop this is still a problem it's even a problem like I think the naive way to F or like when you you first try your first fine tuning steps that's even something that can happen even more that Amal will will just try the same thing over and over again maybe because it just has a very predefined notion of what works best in a certain situation and it doesn't really incorporate all all the history of that this doesn't actually work so with the first version of that came out together with the paper we tried to do a lot with prompts and certainly models at that time were more prone to these kind of mistakes and we told it you know very strongly that if you see that a command is not working don't try to run the same command again it's not going to work the second time around are very small silly things like some commands don't have an output if you're an empty folder and you do LS there's no output it doesn't mean your command has not run it just meant the output is empty so then we actually explicitly told it like hey your command run fine there was no output small things like that I think those are like the first order things you can try and then you can go to more sophisticated method in principle you could sample a bunch of times from language model and we could have can take what what's best but it's it's I think that's one of the problems that really limits the length also of how long you can how many step a language model can take and still do reasonable work I think what everyone sees is that if these trajectories if the agent takes many steps the quality of its work will really be great similar to if a human gets stuck at a problem I I think it's a very human problem actually like most people stick to the first idea that they have at solving a problem and they have a really hard time completely abandoning it or like coming up with a completely orthogonal second idea mostly they'll try variations of the first idea and that's exactly what we see with agents I can definitely say I'm guilty of that sometimes it's like the code is right and then it'll be some environment setting somewhere else the code was fine but it was some cash environment variable and just keep running and then end up just restarting the environment and all all is good exactly python 2 Python 3 that switch as well right when you're messing around with that oh absolutely guilty of that so relatedly what are some limitations with agents uh where do they not perform well where just curious to get your insights on yeah just limitations of Agents I think language model agents Excel when problems are still somewhat isolated if there's something obvious like maybe not obvious but something like maybe something easy for a human to Overlook make colon in the wrong place or a sign problem or stuff like that I feel like agents can come sometimes pick up on that like great quite effectively maybe more so than human sometimes I think things that yeah think that have to deal with like bigger systems thinking anything that requires like looking Beyond specific functions so more bigger structural changes refactors all of those things are definitely quite hard for language models still I think that also relates a little bit to like long term memory so the maintainers of repositories are invariably the fastest most effective at fixing bugs or like making changes to that repository and somebody who's coming at coming at it fresh is going to have a lot more like a lot more of a of a learning curve to get up to speed with understanding what the best changes to propose are going to be and so I think the same thing with the language model is code that's extremely popular maybe it has a good understanding of like maybe how it would work for what how it can make edits to that but at some point I think language models that are yeah maybe more specialist in particular repos or like more specialists in particular types of programming would be a lot better at solving a lot of problems just the way that humans like when they specialize on something are much better at doing that so yeah I think it it's a little bit related to this like how many steps can an agent take and still do useful or like still be top not at its reasoning and if you have very large problems I think agents like language more in general like they are like they want to fix everything in one go and I think they're not very good at a human to make small have a plan and slowly iterate on it for a very long time until they they achieve it and I think that's also something we we already hinted at that once things go wrong in an anent wanted a bad edit the like hood of recovering from that I I'd say it's much smaller than a human I think all all of the this wisdom of how how do I deal with a very large problem that's it's not a weakness of agents and it's not like you can solve it in other ways it's just something where the current state of AI is not at to par with a human certainly cool you've got me thinking here and about how we can leverage sweet bench and leverage definitely agents right but then automatically I would want to ask that question how should an Enterprise access it and I'll probably also frame it in one of your recent comments about the fact that okay as time progresses maybe we need a specialist right to go ahead and actually help set agent right does that imply that maybe for Enterprises when they have this is the golden code base that in essence that would be the one they would train first to any new be develop better code as they're actually working with s to S agent yeah I guess thinking from like an Enterprise perspective yeah it depends maybe a bit on on your your resources or like the size of your code bases and I would even say like the history and size of like your repositories with respect to the data available to say specialize build a specialist model on on particular repositories if you have those resources I do think that you could probably essentially build like or fine-tune models for particular use cases and repos that would probably be quite strong or a lot stronger than maybe something that's just for publicly available things I think a very simple thing like if you're a company so I think you have you know with swe bench as a benchmark also there are certain constraints on what you're allowed to do in order to submit and things we consider cheating for example something that would help the agent a lot is that you could have like test you could make testing very simple like with normal submissions to sbench the agent has to find out itself how to run all the unit tests and with something like Chango it's probably not going to work the first time and there's ways you know around that but like I think the whole environment setup and like just filling with a few parameters to really tune it a little bit to your code base can get you quite far and generally if you have especially in we run it on Open Source repositories of maybe various I guess Jango is very well tested but generally if you have a lot of tests and you can run them very easily this is great for an agent because first the agent could iterate for Forever Until It fix something and second you can test if it breaks anything else or or you can just run five different five different agents five times and check based on the test reports if it did well and also what you could do maybe for some type of problems it's very easy to write up a test that's currently failing and you give that to the Asian as well you basically instead of the agent having to come up with its own reproduction script like in C bench where Tak the GitHub issue and maybe it still can also run the tester or already pre-existing at that date in the code base if you're really thinking of something that solves problems the more you do in terms of testability the E will be to iterate on that until I think if your testing is perfect and your reproduction script is perfect and maybe you still have like a small filter that like whatever changes the agent makes aren't too long because you probably only look for like three line fix I'm sure you can get a perfect uh Solution by just trying long enough there's I think a lot of very simple things you can do that involve no no training it's a lot more about like how do I pose the problems and how do I make sure that the agent has access to the best to the best feedback um from the environment possible and before this recording and I were chatting about how much this will transform how people contribute to open source projects for example at data bricks we've hired a number of people just based off of their contributions to open source repos for example ml flow and Spark and now it might be shifting towards who can build the best agent to solve these issues to just really accelerate productivity so that's just been something that I've been thinking about and curious to get both of your takes as to how you think an agentic approach to working on GitHub issues will transform software engineering yeah I think the like stream straightforward thing like you mentioned that I think we've actually had some submissions from s agent or other types of agentic Frameworks to even our own like s agent and related repositories I think the most obvious thing about that is that yeah like as the cost of generating say a poll request or generating some new contribution inevitably goes down with the popularity and proliferation and of agentic language model Frameworks there will be a major increase in the labor required System review all of these changes and have the overide of ensuring that their new contributions to open source software is safe and reliable and makes sense and so on so I think this is generally probably going to be a trend across the board not even just in open source repositories but probably software Engineers are going to spend a lot more time reviewing code and a lot more time in QA in a lot more time writing tests and the historic framework of early software Engineers writing a lot of code and Senior software Engineers we doing a lot of that and ensuring that like Integrations are merged like that quality is like QA the kind of QA component I think of software engineering is going to become a bigger part of everybody's job in a way yeah so I think it's right now most empowering for the maintainers themselves maybe I think if you're someone else who discovers a bug and you just run an agent and you sub just blly submit the solution to the repository I'm not sure how much you're really helping because basically you should be able to run a you have to have a sanity check on your pro request and as a maintainer you have a lot of intuition about what feels right even not even talking about tests but you you probably just have a hunch of where problems come from and what kind of changes to make and you will immediately have a action to seeing that progress I would advise everyone like to try to at least understand what the agent was doing where it's coming from if that really makes sense don't flut poor maintainers with like hundreds of poor requests that makes sense it's like same thing just copy and pasting output from chat gbt actually that recently just happened with a couple of repos there was a there was an external company I I won't mention them but that decided to actually hit a few very popular open source repos oh actually both of them I'm M with so that's why I'm aware of them and they just flooded them with issues and literally we're working on as a community we're working on okay do we need to actually literally add an additional check box no I'm not an AI robot because they're just to your point they're just sending lots of issues right on a more lighter note this reminds me of the meme my original job is software developer my new job is AI code fixer or code reviewer but while I say that in one thing I'm curious about actually related to this is do you feel that the while we're saying hey let's use chat gbd or o4 om mini whatever else do you feel that maybe some of the reasoning models will be more helpful they'll be slower because the reasoning models but perhaps then that will allow us to better understand exactly to your point what the LM is thinking and ultimately helping us understand what the agent was thinking yeah I think that's I think that's totally valid we'll see think that probably these resing models will differ in the particular quality and interpretability of their reasoning traces just the way that RL the kind of like training process that these reasoning models go to I don't know if they necessarily they're helpful for the state of the model but I don't know how much that translates necessarily to the ability for humans to follow along so I don't know I've seen some kind of crazy residing traces out there as well in terms of these where it's not I don't know how helpful it really is to thinking about like the actual solution at some point but yeah we'll see we'll see some I maybe hopefully some analysis and as we models we'll get that or I'm sure those reasoning will um those reasoning like deps will become more transparent as well so I think it's also important to keep in mind that also if human solving problems it's not just sheer intelligence that like decides on how good you are solving problems there's always like a very big kind of Behavioral aspect to it like how do I iterate problems and how persistent am I and do I learn from things that go wrong I bet that with a lot of jobs it's not the most intelligent humans but the most kind of longterm efficiency that really are your best employees and similar with language models just good reasoning is not necessarily what an authentic approach needs and maybe you can even get away with I think there's very particular places where reasoning is very important okay and it also depends on your approach right for example when editing a file and you really the moment you really saying I have identified the cause of the problem I'm going to now make an edit to fix this problem obviously that's very reasoning heavy everything around it is not so reasoning heavy it's okay I'm going to rerun my reproduction script I'm going to check if it's fixed and I'm going to look around see if I find context so maybe we'll see a little bit more of a trend to have agents that employ different language modes and the other place where I think reasoning is very strong maybe is the sanity check or maybe you even run five agents and you want to pick the best solution and you could for example try to do that based on static analysis and OB that's very reasoning heavy a little bit of a little bit of a a trade-off here I think no I actually can empathize it's we fight this problem in a Pache Spark all the time where basically our logs become so long and try to decipher actual information out of it becomes its own problem on its own which is and so exactly I think I think that if we end up over reasoning then you're just going to be going through a massive amount of information trying to figure out what's actually useful here great you went ahead and spun found the bug right also like thinking about now I I think we talked a lot about the load of like how humans become reviewers of maybe AI Pro request in the future of course you could also try to start summarizing these ancient trajectories and you could try really building like currently coding environments and editors are emphasized humans editing code and like this kind of like editability maybe there's going to be a trend to like very specific tools that are only about um reviewing and poking around the code base trying to make sense of it you know there might be very specialized tools to e that load on on human developers no that's really cool and then okay so we're definitely running a little long but this is super exciting so I I want to at least make sure I ask the question like yeah how do folks get started with sweet agent and sweet bench like how do they get started because I'm it's certainly exciting it's certainly really cool so how do we get people uh on this podcast to get started on this yeah so sweet bench is mostly a benchmark for evaluating these AGM tech systems and I think it's most helpful so when like when you have something that you want to test out and you can you can use like sweet benches GitHub repository to say test your kind of test your system and that's definitely something that's helpful and you can compare that to maybe what's on offer from other companies or other other open source repositories for S agent it's more SW agent is more of an agent scaold I think I'll let Killian explain since he's been working a lot on the recent release yeah so to get started you can check out our uh GitHub repository or go to our website it's s minus aan so swe minus agent.

32:58 Getting started with tools

Com you can look at the documentation of how to get started you can also read code base I think our goal with three agent is really to make it simple and hackable in a lot of different ways I think there's a big span and it's probably going this distinction will get even larger in the future between like how much is it a tool versus how much is it something to test new ideas I think we're somewhere in the middle you can certainly use it as a tool but you can also it's very easy to hack or to configure adding tools is now easier than ever before basically you can just write a python or shell script you put it in a certain folder you add one yo file and boom it's a tool that he provides to the to the as you have to just tell it to use it I think that's the easiest way to get started only let us know if you have questions I think the emphasis is a little bit on if you have new ideas you want to play with this technology and maybe you want to modify some of their code I think that's just a really nice way of doing that but if for example you just want to write your own agent completely from scratch and you have some completely different ideas we also in our effort to make C simpler we took a large chunk of very annoying codes namely how to boot up these environments like basically these ster environments and execute code in there and all of this environment management and how do we communicate with the Stocker container stuff is now its own project called serex like T- F fun anyway it'swe minus rex. com uh it's a very small code base so it's very easy to hack and essentially um you can just in in five lines you can just boot up a Docker environment and then start communicating with it so it's once it small web server and you talk to that web server and for example you can tell that web server to execute your your command and get the output so with that I think it's also very easy to write your own agent if you want that but of course is very nice very configurable very hackable please try that awwesome I love that call to action to help close out our episode test out s bench play around withu agent perhaps become a contributor do not flood it with issues generated by agents so yeah thank you so much for both your contributions to sbench and S agent and thank you for joining us today on data Brew

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.