The Coding Agent Multiverse of Madness

Summary

Coding agents account for more than ninety percent of enterprise LLM inference volume, and their infinitely parallelizable nature is rapidly commoditizing engineering time—but without centralized governance, organizations face vendor sprawl, rate-limit workarounds, and enterprise data privacy risks.
Unity AI Gateway provides a centralized governance layer for all models and coding agents, centralizing user metrics, rate limits, MCP authentication, and data privacy controls into a secure data lake without forcing developers to standardize on a single tool.
The right answer to managing coding agent sprawl is an open ecosystem with centralized governance, supported by native partnerships with Codex and Claude Code and MCP tool governance policies that control which enterprise resources agents can access.

The Coding Agent Multiverse of Madness

As coding agents and AI models scale across enterprise environments, managing the resulting vendor sprawl and security risks creates a critical friction point between developer velocity and IT administration. Shifting from rigid tool standardization to an open ecosystem allows developers to utilize specialized models for distinct engineering tasks, but it necessitates a centralized governance framework to control costs and prevent data leaks. In this video, the lead of Databricks' inference infrastructure introduces an AI gateway framework designed to bridge this operational gap. This approach details how organizations can navigate the complexities of multi-model observability, cost-control, and automated testing, enabling teams to safely leverage high-volume reasoning agents without compromising enterprise data privacy.
𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀
• Coding agents overwhelmingly drive the highest volume of AI inference, thriving on an infinitely parallelable nature that can commoditize traditional engineering constraints.
• Managing high-level AI automation requires an architectural layer that centralizes user metrics, rate limits, and Model Context Protocol (MCP) authentication into a secure data lake.
• The ultimate win-condition for engineering organizations relies on "shifting left" by allocating heavy infrastructure resources to development and staging, allowing autonomous agents to continuously test and optimize performance kernels.
00:00 — Developer Freedom vs. Administrative Governance
01:28 — Product-Market Fit of Coding Agents
02:19 — Shift to the IT Admin Perspective
03:28 — The Pitfalls of Tool Standardization
04:46 — The Reality of Vendor Sprawl and Cost Management
06:40 — Introducing the AI Gateway Solution
07:20 — Observability, Cost Control, and Data Privacy
10:12 — Developer Workflow and Open UI/Isaac Multi-AI Harness Integration
11:42 — Advanced Features (Computer Use and Model Context Protocol)
12:37 — Case Study: Implementing Governance and Scaling at Databricks
Check out future Agentic + AI Observability events: https://luma.com/AgenticAIObservability

Chapters

FAQs

Why do coding agents drive so much of enterprise LLM inference volume?

According to the lead of Databricks' inference infrastructure, more than ninety percent of enterprise LLM inference volume comes from coding use cases. The reason is that coding agents are infinitely parallelizable—unlike other agent types that peak at a few queries per second, coding agents can be scaled to very high volumes simultaneously.

What is wrong with forcing tool standardization for coding agents?

Forcing engineers to use a single approved coding tool creates resentment, productivity loss, and security workarounds where developers bypass enterprise controls to access better models. This video argues that tool standardization is the wrong answer because developers need access to the best models on the day they launch.

What does Unity AI Gateway provide for coding agent governance?

Unity AI Gateway is a governance layer that centralizes user metrics, rate limits, MCP authentication, and data privacy controls for all models and coding agents. It supports native integrations with Codex and Claude Code, allows organizations to set MCP tool policies controlling which enterprise resources agents can access, and ties into source control as governance infrastructure.

What security risks arise when coding agents operate without a gateway?

Without a centralized gateway, rate limits cause engineers to bypass enterprise security controls and use personal accounts or consumer-tier plans that lack enterprise data protection agreements. This video identifies this as a significant security risk, because enterprise code and data then flows through services without the organization's contractual data privacy protections.

Full transcript

[00:02] We want the access to the best models on the day they launch with the best coding agents, but admins need a blader to actually do governance, prescribe policies, and ensure access is controlled. We want freedom for developers to use any coding tool that they want, build any agent using any
[00:15] model, and not have to worry about going and dealing with security review and admins and all this stuff again. We're all here because everyone is building agents, right? Like there's all
[00:28] sorts of them out there. There's open frameworks like LangChain, Llama Index that you can build with. There's some agent platforms out there that you can build these agents on. Uh but a lot of more agents are appearing, too, right? Like there's the
[00:41] coding tools which actually overwhelmingly drive most of the volume uh for all the inference that's being done in the world. Uh and then each of the individual SaaS providers that many of us use in enterprises are also
[00:53] building their own agents. So, when you take a look at this landscape, um for some background, I I've I've been at Databricks for 7 years. I lead our inference uh infrastructure teams. When I look at the amount of volume that we
[01:05] see uh in our inference platform, I would say more than 90% of it is coming from coding use cases. And I think the reason for that is that there's a tremendous amount of product market fit that exists with these with
[01:17] these particular agents. A lot of other agents are cool, they exist, but you know, they maybe peak at like two queries per second. People sometimes talk in queries per minute on some of these other agents when we when we work
[01:29] with these other companies. I think the really exciting part about these coding agents is that they're basically, you know, infinitely parallelizable. And so, here is where we have really see a lot of people use like very very high volume.
[01:42] So, I think just taking an even bigger step back as a software engineer, I think it's an incredibly exciting time to be a software engineer. Every week it feels like, you know, a new model drops. There's a new 10x engineer. There's
[01:54] eight or nine coding tools out there, and every day it feels like there's a new one that comes out that people tell you to use. Um one of the things that, you know, Rohan mentioned when when we were up here before,
[02:06] uh is is that like the really one of the resources that used to be the most constrained, which is engineering time, is really really quickly become commoditized. Um and so I think that's going to really necessitate a very big change in how we
[02:18] build software. But in this talk we're actually going to take a little bit of a step back and we're going to approach this not as developers. I know many of us are developers here in this room. But we're going to approach it from the perspective of an IT admin who's
[02:31] actually rolling out a coding tool at their company, right? In in that in that panel that we had, we talked about how it's important to get these really amazing technologies into the hands of people who are doing real work so that the value we've already
[02:44] created in technology is able to be realized. A lot of the time, things like this are what hold thing, you know, people back from having the best tools at their disposal. A coding agent can really be the most
[02:56] privileged user in your company. And in fact, if it's doing its job, it probably is like this, right? It has access to all the context that you need to do your job. And if you think about the different tools that you use, you
[03:08] probably have access to all of these, too. And you might imagine that coding agents, you want them to have even more access, right? And so for us as developers, it's like very cool when you can connect every tool to your coding agent, but for an IT admin, this really
[03:20] brings up a new set of concerns that they haven't had to deal with before. And I think the way that developers use coding tools also makes it more tricky, right? So I think in almost every company that I talk to, there's like, you know, some
[03:33] person who is in charge of rolling coding tools out and you know, their job is kind of to really push on people to start using these things and they have to make a decision. What do I actually recommend
[03:45] people do? Uh well, they could say something like, "Hey, we bought this license of tool A and like everyone should just use tool A. I think that's an outdated mindset and doesn't really match how developers
[03:57] think about using these models and these coding tools today. Uh I personally use a a different tool for refactoring code, uh a different tool for uh writing tests and a
[04:09] different one for code review, right? And again, like in traditional enterprise software, maybe this is something that you're okay with not allowing. Maybe you leave some productivity on the table for the sake of standardization. But with the amount of leverage that
[04:22] coding tools are buying on the most expensive and important resources in a company, this is a bad strategy. If this strategy allows developers to be 20% more productive, that's a huge amount of value. And so I think this is really
[04:34] what enterprises should be pushing for. You should use the best tool and the best model for the job. And the answer to what the best tool is or what the best model is, frankly, is shifting every week as a new model comes out. So
[04:46] what then happens? Like let's say you let developers do this. What you end up with is a massive amount of sprawl. You have a bunch of different vendors that you're working with on coding tools. You might have OpenAI, you might have Anthropic. Many of those folks have
[04:59] spoken here, right? You have Cursor. Uh As a result of that, if you're practically managing that deployment, you also have several admin panels because none of those tools really care to work with each other. Uh and so now
[05:11] you're actually thinking about rate limits for all of your users across all of these different platforms. Uh a lot of this pain has been masked because, frankly, coding tools have been very cheap and priced per seat. Uh I think it's safe to say that that
[05:24] era is very likely over. And so almost every large company today is having the conversation at the executive level, "Wait, how much are we spending on coding tools?" And the question that someone then who
[05:36] is in charge of this has to go answer is like, "Well, okay, I have to pull this up from three different data sources and uh you know, maybe I don't have per model visibility." These are really big challenges for people. So it's kind of a nightmare, right? If you actually wanted
[05:48] to be successful at this, you now have to manage multiple vendor contracts per tool, multiple admin consoles, deal with billing and uh being in my many different places. You have no idea which users are using how much
[06:00] where, and you then also have this problem that you actually want to make this the most privileged user in your company. You want it to have access to every single thing that you the developer does. Uh so, you have to manage security in 10
[06:12] different places, too. Obviously, this doesn't happen. And the way this shows up for us developers is it is actually more uh silent than we think. It shows up in a lack of velocity. It shows up as, "Hey, we're
[06:24] still waiting on procurement. We're still waiting on security review." Uh and so, no, you're not allowed to use this tool. So, we spend more time as developers blocked on our IT policies and our admin policies or our security policies, and
[06:37] that's actually holding us back from being more productive and using these tools to their maximum leverage. One thing that we're working on that helps with this is the thing called the AI gateway. Right? Many of you may have heard of other AI gateways like Lite LLM. Uh we have one, but like I think
[06:50] like the real kind of ethos that we have behind this gateway is that we want freedom for developers to use any coding tool that they want, build any agent using any model, and not have to worry about going and dealing with security
[07:03] review and admins and all this stuff again. But at the same time, we think it's unrealistic to build something or push for that motion if admins have to deal with the risk of security being compromised. That is an unacceptable
[07:16] risk for almost any company, and I think it will continue to be that way for the next few years at least. So, what does this thing do? Well, the first thing it gives you observability. Uh you can see which users hit rate limits, track where your power users Uh I think this is
[07:29] actually a misconception that a lot of people have. They think that uh the people who are hitting rate limits are doing something wrong. I think it's the opposite. Those are the people who are actually pushing the boundaries of what's possible, and they're the people within your organization you may want to
[07:42] actually empower the most. Um conversely you might want to figure out who's not using coding tools. Uh it could be that uh they just are like, you know, a Luddite or something or it could be that they actually don't
[07:54] have access to the context they need to do their job. So they don't see value. And this is all an important signal for someone who's actually rolling these tools out. You do want to set some cost controls. You want to know how much people are
[08:06] spending so that you're not dealing with a bill of $200,000 on one given day, right? Uh at Databricks we actually have rolled out coding tools very widely to a lot of our developers and even broadly
[08:18] to marketing, finance, and basically everyone. Uh and we encourage people to use those tools aggressively. And today we pay per seat for those tools. Uh but tomorrow that cost can balloon by more than 15x
[08:30] and someone has to pay that bill, right? That could be up to hundreds of millions of dollars with how expensive these models are. And finally there's a really important piece which is privacy. The reality is that most of these coding tools by default retain your data even
[08:44] on enterprise plans and this is a huge security risk for you. Uh and it's something that you really don't want to have to deal with. You want full ownership of the data that you pass in because otherwise again you won't be
[08:56] allowed to give that context to the model. A real problem that we have actually at Databricks until we until we started using this thing is that we really want to automate incident management. We want to be able
[09:08] to when any you know, incident happens first have some kind of a debugging co-pilot that reviews all the logs and all the services and tries to figure out why is this happening? However, our logs, right, can't you
[09:22] know, we try our best but they may have some PII in them and so there's a legitimate question of if is it okay to send them to the models? If we didn't have the guarantee that there was no data retention and that our data couldn't be used for for training,
[09:34] we would be a lot more worried about this. But today because we use this we don't have to do that. So, this is powered by our inference platform, which has both real-time and batch inference. It has every model available inside of it. So, it's Anthropic, OpenAI, Gemini,
[09:47] open source models like Kimmy, which are really, really powerful. Uh and so, we're really excited about this platform. It's something that we started building well before coding agents were successful. And it's, you know, picked up a lot of usage. So, there's a lot of people that
[09:59] use it. We serve 15 trillion tokens per day. We have tens of thousands of GPUs. Uh and so, we're really, you know, feel very comfortable asking people to build their whole engineering teams on top of this platform, which is basically what
[10:11] they're doing when they use coding agents. So, if our admins are happy, I think developers can be happy, too, right? We want the access to the best models on the day they launch with the best coding agents, but admins need a
[10:24] layer to actually do governance, prescribe policies, and ensure access is controlled. So, I'm going to show you a little bit about what it looks like for me when I do my work. The nice thing about this gateway is that it actually connects to any model. And and I wanted to call out
[10:37] this really awesome piece of software, OpenUI. It's open source, not built by Databricks or anything, but it's really cool. You can actually launch like many parallel agents using this thing. And so, we've hooked it up to our our
[10:50] our like remote cloud instances. And so, I actually have like totally different ways of doing different workflows. I have Codex review my PRs. On some tasks, I use Cloud Code when I'm trying to do some kind of a migration. Or for other
[11:04] tasks, I might use, you know, Codex. All of this is managed by this like kind of harness that we have called Isaac, which is a multi-AI harness. And it just all it does is it points the coding tool at our gateway.
[11:17] And then you can use any model and any coding tool that you want. So, you might wonder, okay, does this only work with CLIs? And the answer is actually no. Like, one of the things we've been working on with uh
[11:29] um with Rohan and the OpenAI team making sure that the Codex app natively supports the Databricks AI gateway. Uh, and so here you can see like the beta version of the Codex app. Um, actually one of the really, really cool
[11:41] things that Open AI has built is computer use. I highly recommend every single person try out this thing. It is actually crazy. Like I can, you know, have my, uh, Codex like launch Spotify playlist. It
[11:53] can like file expense reports for me. Uh, it's it's actually incredibly powerful. Um, so all of this kind of still works through that one gateway. You're not compromising on innovation because you want to use this thing. So, what does it actually do behind the
[12:05] scenes? One of the things that I think we're not talking about yet, right, is MCPs. With many things in software engineering, is a layer of indirection. So, with this lay with this gateway, what we have is we actually allow you to
[12:18] configure your endpoints to third-party APIs, and then you have tokens vended for you that give you the right amount of access that your admin is comfortable with. Your admin is able to declaratively specify policies about
[12:30] what kind of MCP tools your coding agent should have access to. And all of this kind of works with a very simple UX for you. So, I'm going to walk you through a little bit of how Databricks uses our gateway and talk a bit about, um, how we just in general
[12:43] are thinking about coding tools. We have tons of code being committed. GitHub breaks every day. We feel really upset about it. Um, and we have tens of millions of lines of code being written. What we felt was important in rolling this out is that you have to measure
[12:56] everything. You have to have every single person leading it, and you absolutely cannot compromise on moving fast. So, let's talk about this Isaac wrapper. Uh, we actually provide a like open CLI
[13:08] wrapper as part of MLflow for this as well. Uh, but basically this thing like it is able to declaratively configure and then launch whichever coding tool you want. So, it works with a bunch of coding tools, and it handles things like
[13:21] reporting metrics into the gateway. So, how many lines of code are you writing? Uh, what how much is each individual developer using in terms of tokens, and this can be used for a bunch of different things like per user
[13:33] authentication, as well as um MCP authentication and things like that. Then you also have that MCP login and the cool thing is that all that usage data is centralized into the same place that your existing data goes, which is
[13:46] your data lake. Uh your capacity is all powered by an inference platform that you can trust and which is actually private and scalable. I just wanted to demo a little bit like what the login flow looks like. So you specify your MCPs and this thing
[13:59] I think is a little small unfortunately here. Uh but the cool thing is that basically this CLI will authenticate you automatically against your all of your UC connections. I think a lot of CEOs are starting writing code. I think with mixed success honestly. I've heard some
[14:11] crazy stories from friends. But our our CEO is actually writing code into this like multi AI coding tool so that we can actually use it at scale. So that's I think pretty cool and I think that this line of reasoning is important to be able to
[14:24] have these things roll out successfully. But it's not clear that they're necessarily what will happen in the future, right? Perhaps automation is really leaning into AI. Um scaling CI is going to be important,
[14:36] too. Um things like and not just CI, but also source control, which I think is a super valuable technology that is struggling to scale. Um these are very hard problems because the code that you write does need to be
[14:48] tested and actually like the this relates to the next bit, which is shifting left your testing strategies is super important. So for example, some of the teams I work with uh build, you know, really optimized inference uh code, right? So running the
[15:02] models. This is incredibly expensive because you actually need very expensive machines that actually are able to like test that your the code you wrote is more performant. But there's a legitimate point that AI is probably
[15:14] doing a better job optimizing the software to run AI than humans are at this point. And so the The thing we have been investing in is actually shifting left by enough GPUs for dev and staging so that we can actually have AI auto
[15:27] optimizing all of our kernels all the time. This has generated massive massive performance wins for us. And I think that's going to that's going to play out in almost every software. We kind of need to rethink how we're building tests
[15:39] and software so that we're actually giving the agents the ability to with very very low latency iterate. So all of these things are shifting. Uh you know, all of us are learning. I think that's the very fun part about a technology revolution is like there's no
[15:51] right answers. Uh there's only exploration. But we do think that hopefully we can help people roll these things out more aggressively, more quickly by solving the hard problems around security and compliance.
[16:04] So yeah, if you're excited about coding tools, like please reach out. Tell us uh talk to us. Actually this QR code probably doesn't work anymore, but um I will I will send a blog. Uh and uh you know, we have a bunch of built-in governance, observability,
[16:16] privacy for admins.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.