Agentic Admins: Managing Databricks @ Databricks

Summary

As the Databricks platform grew to serve more users, teams, and eventually hundreds of thousands of agents, the manual admin work of stamping pull requests, provisioning resources, and approving requests became unsustainable — leading the team to adopt a no-clicky-ops philosophy.
The team built agentic admin systems grounded in System Tables and live platform data, with explicitly taught skills that encode consistent processes and tight scoping to prevent hallucination and over-reach.
This video covers how a small human admin team can govern an exponentially growing agent workforce by ensuring every admin decision is backed by data, offboarding is automated, and experimental agents operate within sandboxed scopes.

Agentic Admins: Managing Databricks @ Databricks

If we are building a future with 100,000 agents accessing our platform, relying on human administrators to click buttons and approve tickets is a non-starter. The math simply doesn’t work; human admin teams cannot scale linearly with an exponential agent workforce. this video explores how the Data Platform team is solving this by building “agentic admins” — autonomous systems capable of managing the platform for us. We will discuss the shift from manual provisioning to agent-led resource management, permissions granting and configuration tuning. Learn how we are architecting the control plane that allows a small team of humans to govern an army of 100,000 digital users without burning out.
Talk By: James Burns, Staff Software Engineer, Databricks ;

Chapters

FAQs

What is no clicky ops in Databricks platform administration?

No clicky ops is a philosophy adopted by the Databricks Data Platform team that treats any manual click, approval stamp, or repetitive query as a bug rather than a legitimate job. The idea is that if a human is performing the same administrative task repeatedly, that task should be automated — especially as the agent workforce grows exponentially.

How are agentic admin systems built on Databricks?

Agentic admins are built by teaching agents specific skills — encoded as consistent processes — that are grounded in System Tables and real platform data rather than assumptions or hallucinations. Agents are also tightly scoped to prevent over-reach, so they only act on the data and systems relevant to their specific task.

How does Databricks handle automated offboarding and resource cleanup?

The agentic admin system handles offboarding by automatically identifying resources associated with departing employees or deprecated workloads and triggering cleanup workflows without requiring a human to initiate or track each task. This prevents orphaned resources from accumulating and consuming budget on the platform.

What does agent sandboxing mean for Databricks platform management?

Sandboxed experiment scope means that agents performing new or experimental actions operate within carefully bounded boundaries — such as a test workspace or a small subset of users — before any changes are applied at full scale. This allows the team to validate agent behavior safely before granting broader operational permissions.

Full transcript

[00:07] So, let's start with a quick show of hands. Who's here for the agent part? Who's here for the admin part? Okay, interesting balance. Who's here
[00:19] because agents just feel like a lot most of the time? Yeah. So this story starts off with our journey two years ago. Two years ago I ran a lot of this flexar
[00:33] limit one. Let me look up again what the values are for the columns in this table. Let me look up again in the docs what the things are that I need to be doing and let me handwrite a query.
[00:47] Usually again and again and again the same query. That was my Tuesday. That was every Tuesday two years ago. And we weren't yet drowning in agents,
[00:59] we were drowning success. So as data bricks grew and is growing, we had more users, more data, more teams. And all that work landed on my
[01:11] team as time time that we were investigating support issues, times that we were stamping pull requests, time that we were doing something else, everybody's, can I just do another
[01:24] thing? And it was killing us. Not dramatically, quietly. Every hour we spent working on these requests were time that we weren't spending building the things we needed
[01:36] to build. So I started pushing something I called no clicky ops. Half a slogan, half a dare. If a human was clicking it or stamping it or typing it for the hundth
[01:48] time, that was a bug, not a job. And then LLM tool calling came out and it looked like agents could all start doing work on the platform. And that might seem like a good thing. There are more
[02:00] hands, more more ability to do work. But we did the math. Every one of those hands was going to want a table, a permission, an approval.
[02:12] The exact bottleneck that was already we were already losing to was about to accelerate. And we could never staff against that. So, we had a choice. Hire a stamping army that could never keep up or figure
[02:26] out what human intention was for. So, that's what the story is about. how we figured that out, how we grounded it, how we learned to ask questions and how we built based on those questions.
[02:38] You see, the choice we faced was a simple one. We could have said, how do we keep doing this? How do we click faster? How do we approve? How do we do
[02:50] more efficient uh processes here? But and we did try some of that, but we realized that that that curve was just going to beat us every time.
[03:02] So, we learned that clicking faster doesn't work. There isn't some change that you can make to human processes that can help you keep up. We had to figure out how we could be more efficient, how we could do less
[03:15] work, how we could consume human conserve human attention to the things that we actually need. So, we ran everything through this one test. If a human stopped doing this tomorrow, would something break? If a human stopped
[03:29] approving the same thing that they've always approved because someone asked them to stamp it, would something break? If a human stopped typing the same thing for the hundth time, would something break?
[03:41] If nothing broke, they were just being a checkbox. They weren't bringing their skills, their knowledge, their judgment to this thing. because that's what we kept coming back to. You can scale compute, you can scale
[03:54] storage, you can scale agents. That's both the challenge and the opportunity, but you can't scale human attention. So, it wasn't what we weren't trying to do was get more of it. We were trying to
[04:06] figure out how we could use it for the right things. And that was what was behind no clicky ops. It wasn't about clicking. It was about conserving human attention. So, here's our first takeaway. Human intention is the
[04:20] critical resource. It's the thing that you can't scale it exponentially. It's the thing that you can't just buy more of from your provider. So, I'm going to tell you how we figured out how to use it. And the first part of
[04:33] that is figuring out how to ask good questions, new questions, and reasking old questions. Making sure those answers are grounded. So, this is over the last two years, two
[04:46] and a half years. It changes as time goes on and two years in agent time is actually a long time. It starts with this first story and it comes from when tool calling just came out and this was
[04:58] something that data bricks was already using and there are some examples and my team also manages ETL that's spread across hundreds of regions and sometimes jobs fail and you're like why did they
[05:10] fail? Let me go read a whole bunch of logs. Let me look at a whole bunch of history for deployment. I'm like, hm, maybe I could just take some Python, hook it up to the tool calling and see what happens. And I found out it was
[05:24] very good. Even two years ago, even GPT4 error models were very good at looking through a whole bunch of text, pulling out patterns and coming up with theories.
[05:36] And that led me to first realization here that grounding beats cleverness every single time. Even with an older model, even with something that isn't coveting the edge, if you give it good
[05:48] data, if you give it tools, if you give it information, it can help come up with a theory, help you process a whole bunch of information quickly. But the thing was, I hadn't applied this
[06:00] to the rest of the data problem. I'd hook it up to APIs, but I was still handwriting SQL. I was still doing select star limit one. And then when I figured out an answer to a question, I'd copy paste it across to a different
[06:13] workspace in a different region or someone else would copy paste my answer and make a few changes to answer their question. And so the query, the ability to ask these questions travel, but the grounding, the thing that made that an
[06:27] actual answer to the question that was intended stayed in my head. So we talked about the early days with tool calling. Let's talk about data grounding now. So about a year later now uh now about a
[06:42] year and a half ago MCPS came out and they were the rage. I don't know if you remember but everybody was going to build an MCP for everything. I thought what if we took an MCP and hooked it up to data brick SQL. We could start asking
[06:55] questions about the data and that worked. You could hook it up to uh the data bick SQL. It knew what data bricks SQL was. It could run the same things. But what I found out was that it was
[07:08] actually not that great. It could write correct SQL, but it would make a whole bunch of assumptions. It would look at the titles of columns and say, "Oh, this column must have a user email in this particular format, so I'm going to just
[07:20] run the query." And sometimes that query would scan all your data and not come back with the answer. Sometimes it just wouldn't work at all because it was structured wrong. And so I went through this process of like figuring out, well,
[07:32] how can I still make this useful? And that was uh what I ended up putting typing by hand. But this this ritual of grounding I did. I would say use this specific table. I would copy paste the
[07:45] docs tab from my thing into the agent and say read this documentation for what the structure of this table should be. I would tell it to check the schema and I would tell it to sample roles to make
[07:57] sure that the actual data results reflected what was in the documentation because sometimes they don't. The other thing that I want to offer you here as a pattern is iterative sampling. So
[08:09] everybody knows that that agents it's expensive to deal with a lot of data directly in agents. Luckily data bricks gives you an engine to ask questions against data that doesn't require you to
[08:21] send everything to the to the LM. So this pattern that I've learned where you can have it select and then have it write a negative query that's that's in your wear. you say and not and you it's
[08:34] also good at writing regular expressions writes a regular expression to uh exclude the things it's already seen and for some very complex data sets this was a tool that I was able to teach the agent um to go through and find the long
[08:49] query of of data sets with you know hundreds of billions of rows. So just offering that as a tip. Um, and so I put this all together and I was getting
[09:01] great answers out of the the agents. I was like, "This is amazing. Everyone should use it and it worked for me." But then people tried to do the same thing and it didn't work for them. And that was because they didn't have this ritual
[09:14] of grounding, which I hate to admit I was still hand typing every time I went in to ask a qu a question. So grounding beats cleverness, but data isn't self- grounding. There's
[09:28] a whole bunch of things that are around your data that will tell an intelligence, a person or or an LLM how to interact with that data. And if you don't have those things, the answers,
[09:41] and you've heard this also already today, probably multiple times, aren't going to be right. So early days with tool calling, data grounded. Let's talk about skills. So
[09:53] now we're up to about four months ago and skills had been propagating. We finally got an internal skills repo and that internal skills repo allowed people to very easily contribute new skills.
[10:05] And so I took this process that I built and I made it into a skill. And so that skill was able to be reused by other people and they started getting better results. Um, just really quickly because
[10:18] there's not a lot of people talking about this process, let me walk you through my skill building process. Um, you can try and and say, "Hey, write a
[10:30] skill to do X, but if there isn't an example already available for the agent, it's going to have a tough time." You probably already do have an example available because you've probably tried to do the thing that you're trying to
[10:42] get the agent to do. Once you have an example, you can just ask the agent to write a skill. And there are skills that teach the agent to write skills better. Um, you can go look for those. I I have
[10:55] some preferred ones, but I don't want to push push that out here. Um, and then the whole point of writing skills is to get consistency of results. So I've had
[11:07] chance I've had particular things where I wrote a skill and it worked the first time and I was like this is amazing and and like everyone should just use this and I used a second time it didn't work. Uh so this is one of the things I I
[11:20] learned through this process of writing skills that you need to test your skills and there's a process for that. One of the processes is called red green refactor. And what that does is it it takes the question you're trying to ask,
[11:32] the process you're trying to get the agent to go through and runs it without skill and puts it under pressure. It can be like management pressure like my boss needs me to do this. It can be social pressure. Help me out. It can be time
[11:45] pressure. You need to give me an answer right now. And there are certain patterns that the agent will will take shortcuts. It'll do assumptions. It'll make oh the column says this. it must mean this. And then it will take the
[11:57] skill that it just wrote for you and do another run and say, does the skill keep the the agent on the path that you have specified? If it doesn't, it refactors
[12:09] it until it does. One of the other things that you can do that wasn't apparent to me immediately either is you can actually ask the agent to generate more things that are similar to this. So if you have a something where you're
[12:21] like, I want to ask questions about audit logs and here's a question I asked. Make five more questions. It's pretty good at making more cases and then you can run those and you can really build consistency in the skills
[12:34] that you're generating. Two additional bonuses in this section. Um one is that agents are really good at writing Python. Um that's what they do and they'll do it every time if you let
[12:48] them. But when you're looking for this consistency of process, one of the things you want to do is have that consistency. And so you have it generate the code that fits the thing you want to do and then save that as part of the
[13:01] skill. So you're not having it try and write it from scratch every single time. And that can be all sorts of things. If you're trying to do screenshots and rate the quality of the screenshot, which is what what one of my other things does,
[13:13] it was good to have that be consistent. The other part of this that is again not totally apparent if you haven't messed with this stuff a lot is that there's a certain attention and focus
[13:27] that a particular part of the agent has. And so you it helps if you only have each particular agent wear one hat. And that's what sub agents are for because they can wear different hats and talk to
[13:39] each other. But just like humans, they're bad at doing multiple things at the same time. So if you have an agent who is a code code writer, a developer, and you expect it to do a review at the
[13:51] same time, it's either not going to write the code well or it's not going to do the review well, or it's not going to do either well. I it it it's and like that's again this outgrowth of how this reflects human intention. So when you
[14:05] break down these roles, when you have them wear different hats depending on what they're trying to do, they become much more efficient. and particularly the librarian one where at the end of a pass of trying to do something you have something that just updates all your
[14:17] documentation as a separate thing focus specifically on correctness of that documentation you can get a lot of leverage out of that so skills ground process and you can
[14:29] write them all of you can write them right now uh and so this was this this came out we were using this in plugins u but about four weeks ago uh we released released
[14:41] skills for genie and that allowed me to move this grounding this grounding of process into the platform. So before the plug-in marketplace was mostly for engineers and it was optin and everybody
[14:55] installed what they wanted when these skills moved into the platform with Genie then I could make these skills available to everyone and specific to the context of the data that they were using and so you didn't have have to ask
[15:09] me anymore for for answers to these things. I could embed my entire process directly in the platform and anyone can answer questions with it. So again, fact number one, grounding beats cleverness.
[15:22] Fact number two, data isn't self- grounding. You need to include this context. But leverage comes from shared grounding, from having everyone see this. And you saw a lot with the announcements of how that's coming even
[15:34] more into the platform, but you can start building on this today. So this was the evolution of grounding here. The point of all this, what we are trying to do is figure out how to conserve human
[15:46] attention. How do we make it so that you can effectively answer questions consistently? So what do you do with that that attention? Well, you start asking
[15:58] questions and you can ask lots of questions. Lots of people think about questions to ask. There are questions like how much are my failing jobs costing me? Or how many instances generations do I have deployed in my
[16:10] workspace? or how many people are using Photon. But I want to give you a sample of questions that I think are broader and that might be not be as apparent that you could ask.
[16:22] So first question who actually uses your most critical table this question is answerable right now in your platform
[16:35] system tables has all this information about who is querying what what identities where they are and you can ask this question so who's consuming it from what surface how often are they
[16:48] using it directly or are they using it through a SQL warehouse And and I thought I knew the answer to this. I thought well most people are using dashboards. Some people are using notebooks. But I ran this I asked this
[17:01] question of Genie and it gave me back a report and the the interesting thing that showed was that within data bricks agents are becoming the majority of consumption. Most people are using agents either through chat or through um
[17:15] the the SQL interface. So based on that, then I can think about, well, what's the next thing I need to do? Maybe I need to make sure that all these tables that they're using are grounded. Maybe I need to see which
[17:27] tables agents are using most and and how to make them get better answers. It opens up all these follow-up questions, but you have to ask that first question. Second question, what AI features are
[17:41] actually catching on. And this isn't a survey. This is a row count. You can look at the users. you can use see which features they're using. You can see how that's growing over time. So I thought
[17:53] that we we had had a big adoption push for AIBI dashboards and I expected to see that and I did see that. But what I didn't expect to see was chat. So the same thing that we've been talking about
[18:06] here about chat being the the new interface, we started organically seeing that internally and there's no chat for part of it and then there's chat and it's just the line that goes up because
[18:19] people want to ask the questions that that's at the top of their mind. They don't want to have to search through dashboards. They don't want to have to write a notebook. They want an answer. And so again, that raised more
[18:32] questions, more opportunities to figure out how to improve your platform. If people are using chat first, are we teaching them the right way to ask questions? Are we looking at the quality of the questions and the answers that
[18:45] are coming out of that? Third question, how fast does the platform make a new person productive? So from the time they're onboarded into the platform to the time they run their
[18:57] first query or they use their first notebook or they make their first job. What does that look like? I had assumptions. Again I had gone through on boarding three and a bit years ago and
[19:11] it was a week-long process and I have no idea when I started using data bricks but I didn't think it was right away. So I asked this question again to Jeie and Jeie told me that the vast majority of people using our internal workspace are
[19:25] active within the first hour that they are provisioned on the workspace. Very surprising uh probably part of the onboarding now and the first thing that they used was a dashboard which is
[19:37] interesting because from what we just learned in the last one chat is actually the quickest growing thing. So, should we be adapting our onboarding to teach people these questions to point them at
[19:49] chat first instead of dashboards? All things that I wouldn't even known to have thought to ask unless I'd asked this first question. Fourth question is a little bit weirder.
[20:02] How many people does your platform depend on? Usually, you think about adoption. How many people depend on your platform? But really, it's the people on your platform who are making things that
[20:14] other people want to consume there. They're bringing in data. They're running transformations. They're presenting that data in ways that makes other people more effective. I asked this question again. And one of
[20:27] the things I found was that um there was one of the people uh on my team who was one of the top question askers not just in our workspace but across all workspaces everywhere. He was just very
[20:40] effective at asking questions and getting answers and using those to drive change. Uh his name's Alan Jay. He's actually in the back. Uh but he's also speaking here uh later which I will show
[20:52] you on a slide at the end. Um and so I I worked with him and I said, "Hey, how can we take the questions you're asking and turn them into skills and make sure those skills are in the platform so that
[21:04] more people have more context to ask good questions? But remember what the point of this was again. How do we conserve human attention? And this is the fifth
[21:16] question. It's the question of like what do we need to build? Where are we spending our time that is not adding value? And the thing is that this is a
[21:29] Q, the PR stamping, it's all data. It's all data that can be on your platform. It's data that we put into our platform. So where to spend human attention is a
[21:42] data question and it's a question that you need to ask. So that was asking questions. Let's talk about building. We had asked that question. We found a surprising result
[21:54] that I'll tell you what we built for it. Our goal is to claim back attention. not just have free time, not just to sit around, but build the right things, the things that people need to be more effective on the platform.
[22:07] So, the first thing we built was asset transfer. You might think that's odd, but what we're seeing was people were out sick. People were on leave, people
[22:19] left, and their data, their jobs, their process, these things that they created on the platform were locked behind them. And so they needed someone to reassign ownership so someone else could keep working on that. And this process was
[22:34] taking a week, sometimes longer because there were all these manual steps that people were doing. People were opening in a request and then the person who was
[22:46] dealing with the request had to look up who the person is, who the person who is being transferred from, who that person had reported to. they need to document approval for the transfer from that manager who might be in a different time
[23:00] zone. So like there were all these turnarounds that was creating a whole bunch of pain not just for us but for everyone who was trying to actually solve a problem on the platform and then they had to actually do the
[23:12] change and sometimes that change uh didn't work because the last time it it had been done the way it was written up had changed. So what we built was zero trust touch
[23:24] transfer. It was our first pay path. We built it on data bricks apps and when you build on data bricks apps you get the context of who's who's making the request who's the owner that's already
[23:36] in UC with the permissions that are on the objects. We had our org directory in there so we could join and figure out who the manager of that person was and then we could make a decision based just
[23:48] on that context. If the manager was the person, the manager of that person who is requesting the transfer to them, that's easy. It automatically transfers to them. They're the people who have the authority to transfer this. If it was
[24:01] being asked to transfer to someone else, that request was routed to the manager. So, it didn't hit us. We could federate the approval the parts that were still uh human-based.
[24:13] And then as part of that we had information about about which places we needed to approve this that taught us this rule. There are two things that don't work. You can either have human
[24:27] approving every request or you can have an agent making a choice based on things that you don't know what it's going to do. We just we learned that you have to put human intention up a level, human
[24:39] judgment up a level. We have to design the space where the decision happens. So the third takeaway, humans govern the workflow and agents act within it. You create this space
[24:52] where you know what's safe, you know where the risk is and you assign that risk and you get it out of a ticket based process that has to have you involved the whole time. But as we did this re we realized the
[25:06] next thing that we are treating the symptom because agents don't just ask for things agent make things and these agents were making things as the users. So again this curve that we're seeing as
[25:19] as uh we were having to transfer assets when everyone was making 10 or 100 or a thousand times more resources as their agent. we weren't
[25:31] going to be able to have this this part scale either. So, we had to figure out what to build to address that. And there was something we could have done. We could have said, "Let's just make creation expensive." You know, it was nice. Agents could do a lot, but it's
[25:45] too risky. We we we we have to shut it all down. Um it's it's too much. But we thought that that would be the wrong trade-off, right? Like the whole point of this is that you can do experimentation and you can find out
[25:59] what solutions fit the problem that you're having. So we made this thing that I call project the box unofficially. Um and it allows an agent to have uh to get a
[26:13] particular scope of resources to do the experiment. And that scope of resources includes team ownership, a UC schema, workspace folder, service principle, lakebased app and budget. And with these
[26:27] things that agent can then independent of the person start building and that person can start sharing that out and see like does this meet people's needs. A lot of things aren't a lot of things
[26:40] are going to be okay ideas. Some of them are going to be bad ideas and part of when we provision this as as a thing that goes to the agent to the people we
[26:52] can set a time limit on it. So what happens is if it's not if it's not catching on, it expires. We reap it. We delete all those resources. We say that was that was interesting, but it didn't
[27:05] find product market fit. And so instead of that being something that someone has to go through and do another SOP, another set of tickets to clean up everything, it's something that just happens automatically.
[27:18] At the same time, success scales. So when something does work and it needs to be handed off to another team or someone else to have it grow into a full grown full-blown product on the platform, that
[27:32] works too because it it's built for agents first. It's available. It's able to be handed off with ownership. We discovered these three types of judgment. So we've already talked about two of them. Um auto approve. So if you
[27:46] know that something is safe, if you've created a a situation where you're like, I can judge that this is safe, you can just autoprove it. There's this in between where there's judgment required,
[27:59] but it doesn't need to be human judgment. An agent which is grounded in the table which can look at, hey, are other people on similar teams making similar requests? Probably. Okay. But it's not just make a
[28:13] decision. It's it's looking at the data that you're providing the platform and making a judgment but a grounded judgment. And then there's still human review as the exceptional case but that human review is again federated.
[28:31] And so this again like you want to start moving it down the layers. So if if you're seeing a whole bunch of human review, even if it's federated, you want to figure out should we be giving a judgment agent more grounding so that it can answer these questions. Is there
[28:44] something that we're using agents for that should actually be automatically approved? The thing that when we built this, we started getting our attention back. It's great. The thing that we didn't realize
[28:57] would happen was how this accelerated. So as we built paths for agents, we started using agents. The first time we built those, we built them by hand as well. And as we so we we freed up our
[29:11] attention by building paths that conserved it, we then used agents to build more paths. Those paths freed up more attention so that we continued to accelerate having more time to spend our
[29:23] judgment figuring out what the right thing was, figuring out how to ask the right next question from building five paths in a quarter to building five paths in two weeks. That's
[29:36] something that you can do and that's what will free up this attention and keep you from trying to chase the stamping army. That's your other option. Let's review the takeaways. The first
[29:48] takeaway, human intention is the critical resource. You can scale compute, you can scale storage, you can scale agents, you can't scale human attention.
[30:00] Takeaway two, where you spend human attention is a data question. Takeaway three, the way you build effectiveness is by humans governing the
[30:13] workflow and letting agents act within it. The final takeaway, this attention compounds. When you follow this pattern, you can build things that free up more
[30:25] attention over time and give you back your time to do the things you need to do. Jo, you're at the beginning of this road. We were too. Two years ago.
[30:37] Two years ago, I was writing select star. Two years ago, I had my docs tabs all the way across. You're at the start. So are we. But it begins with one question. Not a project,
[30:50] not a platform initiative, not a budget request, one question about a table you care about. Pick one table. Ask it. It may seem too small,
[31:02] but that's where the change happens. When you start exploring the data that's already in your platform, when you start leveraging it, you figure out not just what's happening, but where you should
[31:14] spend your attention and what you should build next. But it starts with the first one. You can just ask your platform. Thank you.
[31:38] and what we've scaled out to build with it. And that's it. Thank you all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.