Self-healing AI agents: automating bug triage and fixes

Summary

Factory builds Droid, a multi-model coding agent, and this video shows how the classic engineering runbook — triage logs, reproduce, write regression tests, fix, and ship — can be automated end-to-end using agents with skills as procedural memory.
Agents are given access to the same production logs, observability systems, and terminals that human engineers use, enabling them to query evidence, reproduce failures in sandboxed environments, write regression tests, and create pull requests autonomously.
Production safety best practices include read-only access to observability systems, execution sandboxing, least-privilege permissions, and clear security boundaries between agent components.

Self-healing AI agents: automating bug triage and fixes

AI agents can automatically debug production systems, from triaging logs to shipping fixes. Rather than requiring manual reproduction and analysis, agents leverage skills, procedural memory that encodes repeatable workflows, to access observability systems, reproduce failures in sandboxed terminals, and create pull requests autonomously. This approach turns the engineering runbook into code.
Learn how to build agents that triage bug reports by querying production logs, reproduce failures using terminal automation, write regression tests, and integrate pull request creation and review. You'll see practical patterns for composable skills, end-to-end demonstrations of autonomous debugging, and best practices for production safety: read-only access, sandboxing, least privilege, and security boundaries.
🤝

Chapters

FAQs

What are skills in the context of AI agents for software engineering?

Skills are procedural memory that encode repeatable engineering workflows as reusable agent capabilities. Factory uses skills so their Droid coding agent can execute multi-step processes like querying production logs, reproducing failures, and creating pull requests without rediscovering the steps each time.

How does an AI agent automatically triage and fix bugs end-to-end?

The agent accesses the same observability systems, logs, and terminals a human engineer would use, then follows the engineering runbook: query logs for evidence, reproduce the failure in a sandboxed terminal, write a regression test to prevent recurrence, implement a fix, and open a pull request for review. This workflow is encoded in composable skills that the agent executes autonomously.

What are the production safety best practices for self-healing agents?

Key practices include giving agents read-only access to production observability systems, using sandboxed execution environments for failure reproduction to contain any side effects, applying least-privilege permissions so agents cannot access more than they need, and maintaining clear security boundaries between agent components.

What is the vision of a dark factory and how do self-healing agents relate to it?

A dark factory is an autonomous manufacturing or engineering environment that operates without human intervention — metaphorically so automated it does not need lights because no humans are present. Self-healing agents represent a step toward this future by automating the mechanical, repetitive engineering workflows that currently require constant human attention.

Full transcript

[00:04] Hello everyone. Good night. For the past decade we have all been building software with this ethos in mind, which you all probably are aware of. Maybe in the last year we have been building software this other way. However, I wish from the last year, or at least this talk is going to change your mind to
[00:20] start building software in this way. To let your agents move fast, break things, and fix it by themselves. So, I'm going to talk about how we build Droid, and how at Factory we have built these engineering systems to enable our coding agent to self-heal.
[00:37] So, at Factory we make Droid, which is a multi-model coding agent that lives on your terminal, also in your IDE, in your Slack, in your remote machines, everywhere, anywhere. If you haven't used Droid, this is how it looks on the terminal,
[00:52] which is the team that I work on on that specific surface. So, and as every piece of software out there, we have bugs. So, in order to fix a bug, this is like the classic engineering runbook. First of all, the user files a bug report, then an engineer triages logs, gathers
[01:08] metrics, stack traces, as much as evidence as we can. And then, ideally, we try to reproduce the bug to find like how is this bug manifesting in our software. And then, again, ideally, we write a regression test that reproduces the bug,
[01:23] not manually, but automatically. So, we not only are able to fix it this time, but also in the future we prevent this bug from ever happening again. And then at the end, we fix the bug, submit a PR, and ship it. And of course, the the goal of a good
[01:40] engineering system is to reduce every single piece of friction in this runbook to make it efficient, to be able to iterate faster, to build things faster, to fix bugs faster. And of course, this runbook is very mechanical, as many runbooks. It's just
[01:57] a mechanical set of steps and surprise surprise agents are really good at that. At automating those things. So, how can we navigate through these run book a lot faster? So, first of all, make sure that your agent is able to access the same systems that a human would access, the same
[02:14] logs. If you're using Axiom or DataDog or whatever, make sure that your agent can also access those systems. It's really powerful for the agent to, let's say, just look at the schema of the data, take a peek of like maybe the first 10 records, and then it can go wild with the aggregation
[02:31] queries and stuff like that. Agents can, of course, write the code, but that's half of the problem. The other half is, of course, finding the bug, reproducing the bug, writing tests, validating that the bug has been wiped out of the system. I assume that a lot of people here are already aware of what skills
[02:47] are, but I'm just going to give a quick primer on what they are. So, skills are procedural memory for agents. It's just a markdown file with some front matter metadata. It's just like a specific format within your markdown that may have a name, a description, and like allowed tools, etc. The skill may also
[03:04] have plugins that it will invoke, so kind of like tools for the skill. And a really core and important aspect of skills is that they are composable. So, internally we have a skill called bug report that internally calls the Axiom query skill that calls the Droid
[03:20] control skill, and so on and so forth. I'm going to give more details and examples of what these skills look like in our code base. And of course, something important for skills that is useful for agents, and that's why they have been successful, is that they work in in progressive disclosure way. So,
[03:36] the the way the context is injected in the context window is lazy, so as the agent needs things, it fetches in into the context instead of bloating the contest the context. And as I mentioned, it's a run book, so that's an important thing to keep in
[03:51] mind. This is an example of a of a skill. This is just a bare-bones skill that lists the same series of steps that I have shown you right now. To give you some details on this specific use case, I'm going to talk a little bit more about each of these skills. How do they
[04:06] look like? So, first of all, bug report, it's a a skill that just receives a bug ID and downloads the bug report from S3. I will talk more about what does a bug report means uh and what does it contain. The Axiom
[04:23] query makes sure that you are able to access Axiom in our case through the CLI, but you can also access it through the through an MCP server, and provides some hints on what the schema of the data is like, what are the useful uh data sets that you're going to query
[04:38] data from depending on For example, for the CLI, we have a specific subset of the streams of the log stream um in order to make the queries more efficient. And then, we have a skill called Droid Control, which This is, personally, one of the ones I like the most because it's basically Droid
[04:55] spinning Droids to test them. So, it's not the human reproducing the bug, it's the agent reproducing the bug uh through a library I will also show you. It's called TUI Story. And then, at the very end of the reproduction uh life cycle of the bug,
[05:11] it's the CLI end-to-end testing, which makes sure that we can reproduce this bug deterministically in a in a test suite. Um and at the end, of course, uh a couple of useful skills that are creating the PR and following up on the
[05:27] PR, making sure that all the comments that were left on the PR are addressed, all the checks that are failing that are also addressed. Um and to give you some idea, we have like 40 skills in our repo, but of course, there are many more out there in marketplaces, in different places uh
[05:44] on the internet. So, first of all, in order to make this system efficient, we first have to make bug reporting easy in terms of the product side of things. So, uh in our CLI, we have a custom {{}slash} command {{}slash} bug, and then the user
[06:00] can describe the bug. And what that will do is will create a bundle of what the session basically the session trace, all the logs that are from that session, if there are stack traces, the stack traces, and any context that might be
[06:15] useful. Uh of course, like what is the platform that the user is in, which that is also in the logs, but it's also useful to have that bundled in. Um and an important part of this part of this aspect is returning some kind of identifier for the user to hold and hold
[06:32] those accountable to be able to even though we may be able to automatically fix all the bugs, it's always useful for users to be able to ask, "Hey, has this bug been fixed?" or "Do you know about this bug?" So, we give them a URL ID where they can ping us on Discord, uh
[06:48] Twitter, or what or email us with the bug ID to follow up on that specific issue. So, this is a quick example of how does it look like in our CLI. So, in this case, I am as I told you, Droid is multi-model, so we have uh
[07:03] a {{}slash} command to be able to pick which model you want to use. Uh so, in this case, you run the command, and then when you're in the model picker, you can type to search within the the models. And if you, for example, type Opus you want to pick Opus
[07:19] 4.6, and you type Opus 4 without the space, right now it doesn't it like filters out the the result, so the bug is like the the string is currently matching by like substring, and it should be doing like some sort of fuzzy match. Um so,
[07:36] let's say the user stumbles upon that, so then the user just types in the same interface that it he's or she's using just describe the bug. So, it gives us a hint of what to look for. Um and then once the the bug is
[07:52] submitted, we give them a UUID that again they can take hold of and hold us accountable. So, after having the the the bug reported, we start with a triage phase. So, first of all, as I mentioned, we download the or when the user submits
[08:07] the bug, that UUID is just the a pointer to a a blob of objects that are stored in S3. So, our bug report skill, first of all, downloads that bundle from S3. And it then delegates to a secondary skill which is specialized in querying
[08:25] Axiom, not specifically for bugs, but like any kind of question that we may ask from our logs, uh that skill is able to answer. And in this case, we invoke it with additional instructions that we are trying to fix a bug. So, giving them giving the the
[08:40] skill additional hints to maybe look at how like which versions of the CLI has this bug appeared appeared in or things like that that might help the subsequent steps to be able to fix the bug faster. So, here is an example of uh how
[08:57] does the Axiom query or Axiom skill works. So, basically, you invoke the skill with some description. So, again, skills can invoke other skills with descriptions as well. You invoke the skill and the agent just hits your observability uh data stores and tries
[09:15] different things until it finds some clues. This is kind of like something that I uh I wouldn't be surprised if you all have already done. After the triaging has been done, this is the part that I'm most most excited about because CLIs are kind of in a renaissance right
[09:31] now. They were big in the '90s, maybe in the early 2000s, and then for 20 years the browser took the the first place in like all podiums. Now the CLI is like coming back to life and turns out that the problems in building CLI interfaces
[09:47] are very similar to how you build interfaces for the browser and the same challenges in terms of compatibility of capabilities of different browsers equivalent to compatibility of different features different terminal emulators
[10:02] have. Um So we use TUI Story which is if you're familiar with Playwright or any of these tools to automate browsers or computer use or whatever you may think of. The benefit of using this particular tool is that it's specialized for TUIs
[10:19] for CLIs. So you can launch a a program in a pseudo terminal that is hidden from the user and then you can send key presses, you can send text input in order to automate certain flows. So we basically give Droid the skill to
[10:35] spin up inner Droids in order to first of all try to reproduce the bug that the user stumbled upon and once it found the bug it will be very useful because then we can kind of freeze that, put it in a test. So here is an example. I'm going to do a quick
[10:51] live demo of how TUI Story works so you get a an idea. So here is I'm like creating a a new session and then I'm just going to take a snapshot of what is happening down like under the hood. So as you can see I am running
[11:06] the CLI in some demon or like background process that I am able to then query and send commands asynchronously. So if you can see this is the initial screen. So I'm going to try to manually reproduce the bug so you get an idea. So
[11:21] I'm going to send the model command and then take another snapshot and basically if you can see here in the chat uh I like enter the model input and then I'm going to press enter
[11:37] and then take another snapshot. And then I'm in the model picker right now. So, now I'm going to send Opus 4 to reproduce the bug and if you take a photo of the TUI, basically there are no models being shown. So, we can reproduce
[11:53] the bug. Now, just to make sure I'm going to uh type a backspace and then take another snapshot and the models are there. So, we can definitely reproduce the bug. And so, the idea is for not a human to do what I just showed you, but for Droid to
[12:09] do it by itself. So, now that we have found a way to reproduce the bug, we are now in the task of how can we make sure that we don't make this mistake ever again. So, for that we use another separate, but very similar project called TUI Test, which is some sort of uh testing
[12:27] framework for TUIs where we have a bunch of end-to-end tests where we isolate each of them in a separate project, in a separate uh folder where it has unique set of uh settings, logs, etc. So, we can observe
[12:42] what happened in that particular test and we can reproduce bugs uh and like we have hundreds of them and the idea is to be able to cover as many relevant use cases for the product to make sure that when we try to ship
[12:58] things and we do it as fast as we can, we don't break things. Um a a really cool thing about this is that uh when you run a test, if for some reason we were to introduce the bug again, the test would fail and part of the results of the failure of the tests
[13:15] are is a recording of the test that failed. So, we can not only know that oh, we have reintroduced the bug, but we have a recording of how does the the TUI flow look like when the bug reproduce so we can then feed that back into the
[13:30] agent and say okay whatever feature you're working on broke these 10 other tests. So these are the recordings go and look at them by yourself. So this is more or less how the syntax of this uh framework look like. It's very It is TypeScript by the way. Very similar to
[13:46] uh like how any testing framework works in in TypeScript. As you can see there are these like expectations. Um you can send input into the terminal. And in here at the very top you get access to the like PTY. So now that we have um
[14:03] reproduced the bug, written a test for it. We I say we but it's just Droid who did that. Um we are now in the part of like shipping this. So this is just like a cherry on the cake. Uh creating a PR, making sure that all the
[14:18] static checks pass like linting, uh formatting, all the commit rules we use conventional commits, uh etc. Everything is set up properly. Um and then create the PR and then follow up on the PR. So once Droid
[14:34] submits a PR we have other Droids that review PRs and humans may also review PRs. So this make sure it's like running in a loop on and like checks whenever there are comments, fixes them, addresses them, challenges them in some
[14:49] cases to make sure that all the checks are green and the implementation still works. So here is I'm going to show you uh kind of like a quick demo of what this process looks like from the point of view of Droid. Now that I have kind of primed you of how a human would do it.
[15:05] Now I'm going to show you how Droid did it by itself. So first of all again uh I invoked the bug report skill with the bug ID. And that uh is going to spin up basically our the same run book that I
[15:21] showed you before. And then Droid will start cooking going through each of the steps. So first, this is the S3 bundle. It It's gathering gathering all the evidence it can. And then it's querying Axiom for
[15:36] relevant logs. In this case, there was no stack trace because it's not like a crash or something like that. It was more like a product feature that was not implemented correctly. And in here, you can see in this part of the process, Droid is spinning up to UI
[15:51] story running sending sending messages to the inner Droid. And then eventually it reproduces the bug. And it explains like whenever it the same conditions that we have seen already.
[16:07] So if we continue after Droid is able to reproduce the bug, we go into the end-to-end test uh phase, which something really important for end-to-end test, especially working
[16:23] with agents, is making sure your test fail first before you fix the code. I'll I'll talk more about this in in a few slides, but just keep that in mind. And then run the last like type checking, linting, etc.
[16:41] Everything looks good and now it invokes the create PR. So it In a moment, it's going to prompt me to allow this tool because it's uh a destructive tool or more like a it's going to create a mutation. The side effect is going to mutate state
[16:57] somewhere else, so it I'm I'm running it in medium auto mode here. So in the end, it did exactly the same that I described and what any engineer would do when they're fixing a bug. But the cool thing about this is that it did it all by itself, everything
[17:12] end-to-end. So you can imagine how this extrapolates into how to build a product faster. So, maybe to keep showing this is kind of the the result of the PR.
[17:28] And something cool about some of the tools that we have made, what I was telling you that the end-to-end test record all the PTY all the terminal visualization of the terminal. It's recorded by our test suite. So, we
[17:44] also surface that as GitHub comments whenever things are expected to be checked. So, in this case given that I added a few tests end-to-end tests that check different things. For example, it checked that if Opus 4.6 without space was provided, the
[17:59] model should be there. Also, I added like 4. GPT-4. 4 and also like gibberish shouldn't uh return any any models. So, it also shows me screenshots within the GitHub PR. So, this is really useful
[18:17] whenever I am not the one who is making the PR, but I'm reviewing a PR. I I want to make sure that whoever coded this made it properly and the assertions that the end-to-end tests were made are correct. So, this is also tools to help the reviewer make sure
[18:32] that everything is correct. Also, another cool thing that we have built is this tiny tool that lets you see side-by-side how it was before and how it was afterwards. So, we can This is kind of like other tools that have been made for browsers, but we're doing it
[18:48] for the TUI again because the technology is coming back from the grave and we have to adapt the tools that we already have for the browsers, but for the TUI. Now, a couple of extra notes. Skills, as
[19:03] you may or may not have used, they're not always picked up automatically. So, as you see as you saw in my demos, I was invoking the skill explicitly or hinting the model explicitly to use the skill. I would recommend you to do so. Don't expect the model to magically
[19:19] pick skills up yet. I think they will get better in the future, but for now if just one word unlocks this whole runbook, just say the word. Say the magic word. Um also keep in mind skills are invocable by the agent and nowadays the concept of
[19:36] commands and skills has merged, so also they are invocable by humans. MCPs work as well. They're not dead in case you all may think that. Uh just be careful of context bloating with what I said about progressive disclosure and
[19:53] etc. There are tools to make MCPs compatible with that philosophy of progressive disclosure. We're not there yet the whole ecosystem, but we will be. And uh if I don't know if you all attended the the the talk from Century that was very
[20:09] interesting in terms of security, but start building skills locally and like trying things out yourself, then share them with your team and then we'll talk a little bit more about other stuff. Um also make sure to check out skills.sh. There are a lot of useful
[20:24] skills to get some ideas on what you could build. And uh beyond box fixes, these are some other specialized skills that we have that are uh unlocking a lot of useful things. So, for example, we have a skill memory leak analysis, which is very similar to the droid control one
[20:41] where we droid spins up droid. In this case, it does spin it but it with a memory monitoring harness around it, so we can uh leverage all the V8 heap analytics to be able to track specifically what parts of the code are
[20:57] taking a lot of memory and also doing that in a time series analysis. Uh to to find memory leaks, you have to analyze the temporal dimension as well to be able to see kind of a consistent growth. If it drops at certain points, that's normal. So,
[21:12] uh this skill helps us uh fix memory leaks or identify any memory bloat in terms of uh use RAM usage. We also have on-call alert triage, which this is really useful to analyze like your Sentry alerts or whatever
[21:28] production alerts you may have. This skill, it's kind of like an agentic way of clustering errors together, finding what is a lowest hanging fruit that you should be working on to improve the quality of your users. Another one is automated QA, which is uh
[21:47] similar to the droid control that I showed you, but for the browser. In this one, we have a lot more uh built up because again, there's there are more tools for the browser. So, in this case, we have already introduced the concept of email inboxes for agents. So, agents
[22:03] register to our platform, go through the onboarding flow, and like test out a bunch of things. Try to do it yourself in your systems. I like try to put your agent in your user shoes. That's what I would say. How can you do this? Make sure that whatever access you
[22:18] give to your agents in terms of the observability data is read-only. Um try to make a sandbox reproducible environment. So, being with TUI control, Playwright, computer use, etc. Now, the closer you get to your interface, so for
[22:35] example, to test the TUI, we could be using computer use and opening up the terminal emulator, but then we would be dealing with pixels instead of ASCII characters. So, try to get as close as the most valuable and simple interface with your product as you can.
[22:51] Um and of course, make regression test the success criterion. Again, red first, green next, TDD, that whole thing that uh has been there forever. Uh I have seen so many cases of agents adding like expect true to be true in
[23:07] test or like variations of this thing just to make the test pass. So, make sure that your tests are actually testing what you're building. Um, and of course instrument your code as much as possible. Add logs, add metrics. It is now a lot more useful because your
[23:23] agents can just comb through a lot of information really quickly. Even without context below being because you leverage of course the queries, uh, the query syntax of whatever tool you use. Be careful of how you inject dynamic context into your skills.
[23:39] So, for example, there was a recent incident earlier this month where a GitHub issue had a prompt injection that was fed into a an agent within the repo that caused, I think like open code to be installed in like a bunch of machines.
[23:55] Uh, so like we're capable of making that same mistake. Just be careful of, uh, what you do. Um, make sure you sanitize external inputs as much as you can. And of course give agents, uh, that are doing these kind of automated things the principle of the
[24:11] least privilege. Read-only, limited scope, as few tools as it needs. And if you can, try to run it try to run your agents in a sandbox environment. So, what's next? It's automating the process and becoming a dark factory. What is a dark factory?
[24:27] Is one of these car manufacturing companies or, uh, factories that don't even need to have the lights on because there are no humans in them. Robots don't need light. So, in software we should also be thinking about that. How can we make our software
[24:43] factories dark so humans can be there and we can turn the lights on whenever we need to but we shouldn't need to. And thank you so much for, uh, your time and, uh, I hope this was useful. I hope you can incorporate some of these ideas into your own products.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.