Fireside Chat with Rohan Varma (OpenAI) and Ankit Mathur (Databricks) on Coding and Agents

Summary

This video explores the evolution from simple code autocomplete to full autonomous coding agents, drawing on OpenAI's Codex reaching 97% adoption across every role and Databricks' experience serving 125 trillion tokens per month.
Rohan Varma and Ankit Mathur discuss why enterprises need programmatic governance, verification controls, and multi-agent orchestration architecture to safely deploy coding agents beyond developer roles.
The conversation covers outcome-driven evaluation, the analogy between AI adoption and the cloud shift to an opex model, and the challenge of resolving context debt when organizations scale agent use.

Fireside Chat with Rohan Varma (OpenAI) and Ankit Mathur (Databricks) on Coding and Agents

Hear from Rohan Varma from OpenAI and Ankit Mathur from Databricks, moderated by Denny Lee (Databricks) on what it really takes to build and ship autonomous coding agents at scale. Rohan will dig into the shift from simple autocomplete to full agent development environments, and how teams can balance developer productivity with the governance, verification and controls required for enterprise use.
Talk By: Ankit Mathur, Sr. Staff Software Engineer, Databricks ; Denny Lee, Director, Developer Relations, Databricks ; Rohan Varma, OpenAI ;

Chapters

FAQs

What is OpenAI Codex and how widely is it used inside OpenAI?

Codex is OpenAI's autonomous coding agent product, and according to Rohan Varma in this video, it has reached 97% adoption across every role at OpenAI, including non-engineering roles like product management and finance. Varma describes this as evidence that code is becoming the core unit of automation well beyond software development.

How does Databricks approach LLM serving infrastructure at scale?

Ankit Mathur explains that Databricks rebuilt its LLM serving infrastructure from first principles to handle the demands of large-scale production workloads, now serving approximately 125 trillion tokens per month. This required rethinking everything from scheduling to token efficiency rather than adapting existing systems.

What are the key governance challenges for enterprise coding agents?

This video identifies three main challenges: agent security and access control, resolving context debt when agents lack the institutional knowledge employees carry, and designing programmatic governance that allows autonomy without catastrophic mistakes. The speakers discuss how enterprise adoption requires organizational change, not just technology deployment.

How should teams evaluate AI coding agent quality beyond token count?

The speakers advocate for outcome-driven evaluation—measuring whether the agent actually achieves the desired result—rather than optimizing for token usage or response length. This shift in evaluation philosophy is described as critical to deploying agents that deliver real productivity gains.

Full transcript

[00:09] Hello, hello. How's it going everybody? All right, good to hear. All right, well, my name is Danny. I'm a PM director or something like that. Yeah, here at Databricks. I'm going to be the host for today. But the important people are actually here. So I'm going to have them introduce themselves. Tell
[00:22] a little bit about ourselves and then we're going to do a bit of a Q&A. We're going to ask them some questions and then we're going to leave a little bit of time for you to also ask some questions as well. Okay? Cool with that? Excellent. All right. >> Awesome. Yeah, I'm Ankit. I'm the tech
[00:35] lead for some of our AI infrastructure products. I've been working at Databricks for a while and excited to work with OpenAI. So Rohan. >> And he just came back from vacation, you know, from like you know, so.
[00:47] >> Yeah.
[01:00] uh high school together kind of, so. And we look exactly the same right now clothing
[01:13] questions, which is actually yeah, how did you even get to this journey to be over at OpenAI and helping to build Codex? I'm just curious like like how did you get even get here on this one? >> Yeah, that's a unexpected journey
[01:25] >>
[01:37] and then the most recent one was we started working on AI coding. And this was back in like 2024 or something like that. And then we also started using Cursor and we were like, this is amazing. And so then we ended up getting acquired
[01:50] by Cursor and joining them, spent a lot of time there building coding agents and other stuff. And then earlier this year I joined Open AI kind of cuz I think when I was at Cursor it was pretty clear that the like
[02:03] AI was really useful for coding but it was starting to become really useful for other things and I think at Open AI our ambition is essentially to you know apply AI to every single possible thing and make it very useful for people
[02:16] and so that was a really exciting mission to be a part of so then I joined and we're starting to do that so it's exciting. >> That's pretty awesome and then I'm going to ask you the same same question for you. How did what's your journey that got you into where you are right now in Databricks and actually working on
[02:29] building some of the major infrastructure that we do have in Databricks actually for AI? >> Yeah so I was always I actually was doing research in the AI infrastructure at Berkeley in Ion Stoica's lab
[02:42] and then I I thought I wanted to do a PhD so then I you know he he said hey like I have a student Matei he's at Stanford so you should go work with him and then I started there I was working on computer vision inference
[02:54] and basically I didn't have like a grant over the summer. >>
[03:07] it was it was fun so I dropped out joined here and now we've you know four or five years ago we started building a serving stack and and starting serving models and just kind of has taken off from there because it turns out that you know AI
[03:20] and using it in production is extremely hard but extremely valuable. And so then you know a couple of years ago we expanded that to LLMs now we've you know got all these models available on the platform including the ones from Open AI so I think it's it's it makes it possible
[03:33] for enterprises to actually build these apps without having to think about infrastructure. >> Perfect and actually let me go back to Rohan your call out which is basically this idea that you know we're all
[03:45] familiar with the idea of building within coding agents right everybody now and their brother sister that we're using coding agents to generate code that probably don't necessarily understand, but that's a whole other discussion. But let's let's talk a little bit about what you had called
[03:57] out, which is that is you joined OpenAI because it wasn't just about coding agents, right? It was about basically applying this to the broader ecosystem, to what you're doing in humanity, to for open source. So just
[04:10] you know, broadly talk a little bit about like what what you're seeing, what what are the other cool agents that you're seeing happen like in in what you're doing at OpenAI. >> Yeah, totally. So I mean so at OpenAI Codex is essentially our agents.
[04:24] Um we have like 100% adoption internally at OpenAI with Codex across every role. Actually, it's like something like 97% and then I looked in and I was like, who's not using Codex? And it's like our physical security team, so like I think it's okay. They
[04:37] don't need
[04:51] incremental benefit to AI where it just like completes the next few lines of code and like that was kind of like how useful AI was. And then for engineering, that was still super useful, right? Like if you're writing code, you're using Tab, it's like this is amazing. Um but I think that if you apply that to
[05:03] like finance or like, you know, I don't know, material science or something, it's like that incremental benefit felt less useful. Um and then I think like fast forward now it's like we have agents that you can fully delegate entire chunks of work
[05:15] that would take, you know, dozens of hours. Um that is now like extremely useful for other roles like finance and uh HR, literally anything. And so I mean even like although the consumer use cases are fun, like one of my friends was texting
[05:27] me that he set up his dad with Codex. We have this feature like {{}slash} goal, uh and they had a goal to basically like they do some real estate business and there was a goal to just like find a good investment property or something and it like Codex was like emailing people, setting up automations to like
[05:40] respond to the emails and then finally like he literally got on a call with somebody from the Codex that scheduled. So um I think that like the the use cases are pretty vast. >> That was pretty cool. So, in other words, what I should do is I should go ahead and do when I want to buy a house,
[05:52] I should do the same thing. Is that what >> I mean, honestly, you should just do like like someone I think posted on Twitter. It's like they just had slash goal like make me money. And like I don't know that it was profitable, but you know, >>
[06:04] >> Okay, cool. Good to know. Okay, I'll have to try that out next time. And then from your perspective, Ankit, I mean, I guess, you know, because you noted the fact that building out this infrastructure is really hard, right? And so, I want you to discuss that
[06:17] journey. Like, you know, first of all, how you built from the standpoint of machine learning, and then how actually serving LLMs is actually that much harder and that much more complex because I don't think that's obvious to folks. Yeah. >> Yeah, no, I mean, look, I think everyone
[06:29] who knows Databricks, you know, knows that we started our world in offline infrastructure, which is extremely different than online, right? And so, the the first part of the journey was just like, how do you build great online infrastructure while having a great
[06:41] product that's, you know, simpler than other ones like SageMaker or something like that. >> Right, right. >> But then you come to serving LLMs, and it's like completely breaks all the rules that you otherwise would have had for infrastructure, and particular serving, cuz each request is actually
[06:54] super different and weirdly shaped, and you know, the harnesses are starting to send more and more input tokens, right? So, it it kind of has you have to almost relearn from first principles, and so, we've done that and now, you know,
[07:06] scaled it up a lot, serving 125 trillion tokens per month, which is great. Um but I mean, I I, you know, going back to your thing, I'm I'm also just I think the scale just keeps coming because of
[07:18] that, right? Like, just the amount of people for whom this is applicable continues to grow, and even like 6 months ago, you you were in this place where it was like mostly engineers with a single coding session.
[07:30] At least, that's what we were seeing. And and it quickly became, okay, how can I have many of these in parallel cuz I like, you know, I'm just waiting for the agent to do most of the work. >> Right. >> Uh and then it became, okay, well, now everyone's got one, and this harness
[07:42] plus model thing, it turns out the context doesn't need to be code. Like, a lot of the principles that you need to write good code also apply to solving other problems. >> Right. >> So, it feels like that's what's, you know, part of the, you know, solution
[07:54] here. >> I think also like an interesting insight is like the code is basically just like a really good interface to interact with the computer. >> Right. >> And it's like essentially the core unit of automation. And so, both of those things make it really useful for
[08:07] non-coding tasks. >> Right. >> Some other interesting thing is like, for me at least, about like we track, you know, how many tokens or like, you know, the amount of value that people are getting from coding agents internally at OpenAI.
[08:19] Um, and about like when I joined like me or like even just like a couple months ago, I wasn't using that, like I was probably using like 10% of the amount of tokens that like an engineer was using. >> Okay. >> Um, but in the last month, I'm now using as many as a and I mostly do product slash, you know, just
[08:32] stuff. And like not coding primarily, and it's like now I'm using about the same number of tokens as like kind of like the P50 engineer at OpenAI. >> Um. >> And I think, you know, that's like generally what we're seeing across the board, which I think is interesting because like I think the
[08:45] a lot of people think, okay, cool, I can use this automation technology to automate stuff I'm currently doing. And that's I think actually like a small percentage of things. I actually think more interesting is like, what are all the things you're not doing that we can actually make possible to do now? Like,
[08:58] um, and so there's actually a ton of things that uh, like we're just doing more. It's not necessarily the same amount of stuff faster. Um, like there's an interesting graph that we have where it's uh, the number of messages that people were sending to ChatGPT
[09:11] internally, kind of over time. >> Mhm. >> And then the number of messages people were sending to Codex kind of like stacked on top of that. Uh, and around when Codex launched, maybe like, you know, the app like four or five months ago, like
[09:23] most people use Codex as their daily driver, but it's like the same number of messages are going to ChatGPT, but there's like 10x more messages going to Codex. >> Mhm. >> Like so, the just the volume of things that people are doing is increasing, it's not just like offset. >> I I I think those are related, right?
[09:36] Because like basically now that I'm no longer hands-on with the agent, I can actually start thinking about other work. >> Right. >> And it to me it's actually almost like sometimes I need to pull myself back cuz it it yeah, then I end up with like 12
[09:48] threads of things that I'm juggling and like okay, I maybe I should cool off, but uh I as the models get better and better, it feels like it's been uh almost like an inflection point with the with the quality. >> Yeah, totally. >> What are you guys seeing in terms of
[10:01] uh you know, next generation of models? How what's what do you think is going to change next? >> Yeah, I think I mean, there's a lot of exciting stuff. Um you know, >>
[10:13] Um I think that a lot of the things that we're thinking about, um you know, there's like different ways that the models can be better for different tasks. One is like we care a lot about token efficiency. I think at this point like uh it's interesting. I think in the last
[10:25] like I'd say like 6 weeks there's been like kind of this like inflection of also people realizing that these things cost a lot. Um and so at OpenAI, you know, like we've always had the most token efficient models, meaning like you can spend like a dollar spent with us is
[10:37] going to go farther um about like twice as far. And so I think like there's even more interesting things we can do there. I think also, you know, different offering different models at different intelligence levels, different speeds. Uh you know, there's all these different variables for different workloads that
[10:50] end up being really valuable. And so um I think that like that's some of the stuff that we're thinking about. Um but I think that in general like to your point like the models ideally all of this is actually not stuff that as a user you have to think about. Um
[11:03] you know, as you offload more and more work like our ambition is that ideally the tool itself is just good at managing itself to the desire that you have. Like one of the interesting principles that we've made with Codex is basically everything we build as a user interface
[11:16] or like a tool to the user is essentially Codex knows how to use it. Um as an example, Codex can actually manage its own threads. It can like put itself in goal mode. It can um you know, change models, things like this. So
[11:28] uh that's kind of the hope is like as a user you're just kind of like this is what I need, and then, you know, whether it's like work for this long, do this good of a job, you know, you can kind of specify those things. >> Yeah. No, I think we're seeing a similar thing in terms of uh the desire to kind
[11:42] of manage costs and manage efficiency. Uh we talked a little bit about that yesterday as well. Um and in particular, I think what we've seen is that there is a strong demand for kind of things that are simpler, and you want them to complete faster, right?
[11:55] So, we need the smaller model families, I think, to be continued to be investing in. So, glad to hear that you're doing that. Um I I I I think we're also seeing this desire to manage a project and kind of individually roll out the tasks with
[12:08] different level of cost efficiency, uh which is kind of why we also rolled out this Omnigen thing, which which uses Codex, right? So, um it's it's going to be interesting, I think, to see how people use these tools going forward, especially as they become more
[12:20] expensive, uh and people start feeling a little bit more of the, you know, the the pain from that. >> Yeah, I think it's also kind of counterintuitive to understand the value that you're getting from them, um because it's like
[12:32] typically you don't think of software as this thing that you should be spending thousands of dollars per person per month on. >> Yeah. >> Um but then if you look back at the work that it did, it's like it enabled us to do so much more, like operate with
[12:44] different ways, and like but I think also then you like rebaseline on like, "Okay, this is normal now, and we're still spending so much money." So, um yeah, I think it is it's interesting. I think And I I mean, I think working both at OpenAI and uh my previous job at Cursor, it's like I do think
[12:57] fundamentally like both companies kind of being born in more of like an AI native form, it's easier for us to have like always operated with those principles. So, like everything we do is just fully leveraged, requires us to be using AI. Like we couldn't operate the way that we
[13:09] do without it. Versus I think a lot of other companies are like, you know, most companies are large large organizations that have a ton of like organizational debt, ways of operating processes, where it's like then you inject AI into the
[13:21] situation, and it's like some things make sense, some things don't. And so, I think there's is kind of like change management that's also happening. >> So one of the one things I because what you're reminding me of is so I'm old enough where I basically I
[13:33] went through multiple generations of changes including like the idea of going to the cloud, right? And so the idea being that you're switching from a capex model to no opex model, right? In other words, I had a to your point about the software. You had a set amount of money
[13:46] you're going to spend, you bought all the infrastructure, then we just maxed out of the infrastructure assuming people knew what they were doing with it. And then we switched to cloud and then became an opex model because now you're realizing oh, I don't have I only spend as much as I actually need for the
[13:58] from the cloud infrastructure. One thing that I'm wondering about what you're implying with and this is actually directed to both of you is that well then in that case are we almost now seeing in this stage of business not
[14:10] just coding now with the in the age of agents almost like this opex model for how you run your business now, right? So in other words, I'm not buying a specific software or specific set of infrastructure. I'm actually literally
[14:22] using AI to and as an opex model to say, okay, if you're actually efficient with what you do, then in fact, you're going to get more done or and if you're not efficient with what you do, then guess what? You're not spending as much and then there's your problem because you're not your output and your outcomes are
[14:36] not as much. And so I'm just curious from the do you feel that analogy makes sense or do you think it's missing something? >> Yeah, I think I think it makes sense in the sense that like what the models like what these agents are doing like is
[14:49] ideally going tied directly to the bottom line of the business and so like the more you invest in it, the more value you get. But I think uh in a lot of environments, it's not like
[15:02] have been a problem before that the things you were spending money on weren't actually helping your business. You know, so like it kind of like forces you to kind of like figure out, okay, what is actually important? Um and I think that like the so yeah, I I like our like this is something that's kind
[15:14] of almost not like a research problem, but it is like an interesting problem that we're working on which is like how do we help people actually like measure both the like economic productivity of like what they're doing with Codex and also like internally like you know
[15:26] how do you tie it whether it's like okay connect into your other tools and like your finance systems and actually start to like measure the value. I think like our goal is to make it so that you can say hey every dollar I spend is like giving me this much more and like yeah I
[15:38] think like there's intuition that that's true but I think it's harder to like connect the dots. >> Yeah I mean I think there's actually lessons to be learned in how we think about and evaluate software engineers you know even throughout history right? >> Right.
[15:50] >> I I would say that you know it's useful input signal how many tokens you're spending per month but you know companies that have put up dashboards for that are probably you know going to end up having people optimizing for that.
[16:02] >> Oh absolutely. >> been true throughout the history of you know software engineering like you anyone can optimize for being input driven. >> Yeah. >> Um and and if you make decisions on how good so how much value you're getting just based on the inputs you're very likely to reward the wrong behavior.
[16:15] >> Yes. >> So I do think that output like you know outcome driven kind of evaluation is still what's going to drive value because if you're not driving an outcome but you're spending tokens there there's actually that's still a problem right? So then what you're talking about is
[16:27] basically finding ways to tie outcomes to those tokens which I think I do think is important. Um but I think the token maxing kind of stuff you know has to be over because that's purely yeah purely input driven. >> Right.
[16:39] >> It's input driven and also the idea is like I was actually having a recent conversation with Shanku about this in terms of from a product perspective and we we in the end you know that as much as we're trying to be engineers we end up talking a lot about the product as well right? And which is this idea of
[16:51] like you've got your inputs token maxing you've got your output okay I've spent this much amount of tokens but you've never actually figured out what the outcome was right? And you're going like I need KPIs or some outcome to tell me did I actually was the money I spent
[17:04] actually useful or not? >> Totally. I mean I think honestly the only metric that I think matters is like revenue probably. >> Fair. >> At the end of the day and I think like if you look at even if you look at OpenAI like you know like we I think
[17:16] it's like under appreciated how small of a company we are. >> Mhm. >> Um you know, like we're I think we're probably around like, you know, three, four, five, like around 4,000 people. And it's like uh you know, we're serving like basically every company in the world, like tons of
[17:29] like hundreds of thousands of enterprise customers, and like um you know, serving like billions of users. And so, I think like with that, if you do the math, it's like the per engineer value that we're getting is pretty high. And I think also like we generally operate in
[17:43] this way where it's like each project that normally would be like a team of like five or 10 people is typically like one or two engineers. Um and so, it's like I think yeah, that's where the like it makes a lot of sense for us just to see it, and we can kind of just like
[17:55] tell like, "Yeah, this is useful." Um but I do think in other environments, it's like harder to like connect those dots as much. And like but I do think revenue ultimately is probably the most important thing that matters. Uh cuz otherwise, what else I you know, like what what else is like kind of the outcome of
[18:07] >> know, there there are some other metrics like adoption,
[18:19] Omniglot team that built out the the product. Like uh relatively small team, extremely AI-native, uh probably in a previous era of software, you need 6 months and a lot of engineers to build
[18:31] something sophisticated. Uh that team kind of really just started from the ground up in this open-source project, uh and just built this amazing thing very, very quickly. Uh so, I do think that that's, you know, like that's
[18:44] valuable asset that's been created. Uh now, how do we measure the value of that? Uh unclear, uh but uh you know, revenue is one way. >> Yeah, I don't think GitHub stars is a very good way to measure that, right?
[18:56] >> Uh well,
[19:08] Microsoft, one of the things he did was, in order to get the company to switch from on-prem to cloud was that he basically same version of a token maxing which is basically said, okay, I'm going to give you all the cloud credits you want. Like
[19:20] every single business, every single one, you can do as much cloud as you want, but we're now shutting off your capex budget. And so now you have to request for an exception for the capex budget and that has to have a VP level or higher to sign off on it. And within 5
[19:33] years, basically you don't have a capex budget at all or if you do, I I Sacha Nadella or have to sign off on it. And so the idea is that you had this 5-year window that allowed you to basically get
[19:45] your team to switch to the cloud. Now, from your perspective, especially now we're talking about the AI days, I don't think we have 5 years to do that switch, do we? >>
[19:58] asking how how many months do you think we have before companies need to get their AI costs in order? >> Right. >> Uh and I said no longer than six because you have you're stuck with tough a tough option, right? Like you can either try to do, you know, kind of not have AI,
[20:12] right? And then you're just going to fall behind your competitors because all your competitors are using code X, uh they're using these AI tools extremely effectively. Uh or you can try to have AI and then not try to control your costs, in which
[20:25] case your, you know, fundamentals of your business are destroyed. >> Yeah. >> Uh so you got to you have to solve this problem. Right. >> Yeah, I mean I do think I mean one thing I will say is like even though input measurement is like maybe not as
[20:37] valuable, I do think we're probably all like under leveraging AI in general. Like one of the things that, you know, we feel or like, you know, we talk like our goal at OpenAI is to develop artificial general intelligence, um you know, that benefits humanity and
[20:49] I think in general I feel like a like and also what's interesting, we think of ourselves as like a research and a deployment company. >> Right, right. >> Um that's like what we call ourselves. Uh and I think I mean from my perspective, you know, like we did the research, it was awesome.
[21:02] >>
[21:15] person, entity, organization actually like benefit from the intelligence that exists. Um and that's like really nitty-gritty stuff like at the infrastructure level that's kind of blocking a ton of adoption, whether it's like, you know, the ability for these agents to run like autonomously in the
[21:28] cloud, the ability for them to have secure governance and controls and permissions, um you know, observability, like all of these things are have nothing to do with like model intelligence, although, you know, since we control the full stack, we can like, you know, kind of
[21:40] tune in all of these things together, but I think that what's interesting is, yeah, like I I don't think that a company that's trying to just have everyone use more AI is probably going to be in a bad spot. They're probably more likely to be in a better position
[21:52] than a company that's trying to limit the amount of AI that's being used. Um you know, like but also there's like an interesting middle ground where I think like right I'd say like 6 months ago like the average like number of to like the average cost for developer was maybe
[22:05] like like a really good AI user was using like $100 per month. Um I think now like probably your P90 engineer is probably using like, I don't know, somewhere between two to $5,000 per month of
[22:17] uh tokens, which is kind of, you know, that that point you're like pushing towards like salary cost. Um so then you maybe you're like starting to think about it, but still probably the average person at a company is spending only like, you know, somewhere between 50 to $100, um which means they're still, you know,
[22:30] they're probably pretty under-leveraged. Um and so I think that like there's some There's probably not a lot of like loss to actually just trying to get people to use it more. Cuz I think you also get a lot of emergence use cases from like those very curious people that are like
[22:43] pushing the frontier like in any role, right, not just like engineering, so. >> Yeah, I think what we've seen is that one of the things that also holds back, you know, deployment more broadly within enterprises is context, right? Like because it's it's it's significantly harder when you are an established
[22:56] company, uh you have a bunch of data that that's in many, many different places. You know, models' intelligence cannot be leveraged without the context, right? Uh but we're going to kind of we totally agree with you in the sense that I think we think the models are smart enough as
[23:09] it is to solve almost all the tasks that a lot of people are doing. Uh it's really about getting the context into the models so that you can be effective. Uh and then I I do think it it is still about control like, you know, doing that on a reasonable cost because uh just the
[23:23] economics of of most companies are very different than software and so uh I I think that the top 1% uh according to the ramp index is like $90,000 per user per year, right? >> On AI? >> Yeah, on AI. And that's like I don't see
[23:36] any signs of that slowing down. Um so it would not be surprising to me if the top 10% became $90,000 per user per year. Uh and that's a lot of money. >> Yeah. >>
[23:49] slightly different ways, so let me um merge them together. So, what's interesting is okay, starting with what you said, which is like at some point you're running so many terminal windows or so many browser windows or so many CLIs per se, right? That basically
[24:01] you're having a hard time managing like 12 or so because the analogy is like basically each agent is his own junior developer that you're basically managing, right? So, it's and so um the the old uh Amazon two-pizza box means means you can
[24:15] only have five to seven people that you can actually manage. So, in other words, you >>
[24:28] here. Just like do we have a solution or do we Where do Where do we think we're going when it comes to actually trying to as each developer is able to leverage AI better, at least the good ones are able
[24:40] to leverage AI better, like can they actually in fact manage more or are we still always going to be stuck in that paradigm of saying, "No, you can really only manage five to seven just like a manager can only manage five to seven people before basically you yourself has
[24:53] context overload?" >> Yeah, I think um I think that that is a problem because we're operating with these agents the way that we do operate with humans, which is like we delegate and then we
[25:05] like wait. I mean, it is funny like I think that the the most maybe like the people that are using AI the most like maybe like 3 months ago realized if you're like watching your agents it's like pretty unproductive. So like >>
[25:17] do something then you just sit there and watch them do it. But like so you know, that's we moved on from that but I think that it's still you know, it feels like you're managing and I think that actually the optimal solution is completely like rethinking the way that
[25:29] we build software. I mean, if software is the thing that you're using AI for because if you think about it like the whole way that we we have this like software development life cycle where we have all these steps like there's planning and then there's building then there's review and deployments and you
[25:41] know, iteration and it's all basically because of this core fundamental constraint that it's hard to write code. So we have to put all this effort into planning so that our most precious resource engineering time is only used on the most important things and like
[25:54] all this stuff and I think now the the code part's the easy part, you know, like the hard parts are everything else. And so but I think that this process that can kind of be turned over and like so we at OpenAI we've done some interesting experiments like there's this article we have
[26:07] that we posted called like engineering the harness where we actually built like a self-improving product that was actually just agents handing off to agents and then human intervention at key points to kind of provide like product direction
[26:19] and then like user feedback actually being automated processed and like kind of I think a lot of what engineers are going to be doing over the next couple years is probably like setting up these like almost I don't know if factory is the right word but essentially these and also I
[26:31] don't think loops are the right word but >>
[26:44] of monitoring and improving the system over time. And I think like you know, our goal is actually to build platforms that enable building these systems which is not about just like because the other thing is if you think about all of our agent usage so far, it's essentially
[26:57] synchronous like human in the loop, right? >> Right. Right. Right. >> Like agents like an engineer using an agent or even like anyone using Codex, you're like you're there. You know, even if you go do something else, you come back and you're still there. Um, and I think that like async agents is kind of
[27:09] I think where we see the future going and like but to actually delegate a task to autonomous async agents, there's like a ton of there's not a like the trust is actually not there yet, you know, and the ability for it to verify its work. There's a lot of stuff that needs to
[27:21] happen. And so I think building these systems is the way that we kind of close the gap on like getting to that point. >> Yeah. Yeah, we're I think we're seeing the same thing at Databricks, right? Like you first of all, you have to use AI to manage your AI, otherwise you're you're just going to be stuck at that
[27:33] five to seven limit, right? And as much as I think people will try to learn how to parallelize more, there are I think fundamental limits that prevent people from just having that much context. But that's I think we're seeing the same thing, right? Like that
[27:46] you know, that's why you want would want something like a meta harness, right? So that you can actually use that to actually manage the work and then you're actually thinking about the important pieces. The other thing that gets in the way for us, right? Obviously is security and and that's a big problem because I
[28:00] you get asked these questions by you know, your coding agent that's like, "Hey, should I run this script?" And it's like, "I don't know." >>
[28:12] >> And there is something messed up in there sometimes. Like you know, like I I was recently using Codex to edit some slides. >> I'm scared. >>
[28:24] script in Google to edit Google Slides, which was clever but but weird. >>
[28:36] don't want you to edit anyone else's slides. Please don't do that." Um, but I how do I actually make that happen, right? And then on the other side, it's asking me, "Hey, I want to run this app script." I'm like, okay, I are you going
[28:48] to edit someone else's site or not? Uh and I think solving that problem, it actually requires like some core infrastructure thinking as well. Um so we're also spending time on that because we we also think autonomous agents are
[29:00] going to be the future. I do think the models are good enough at a lot of the tasks now. Uh the problem ends up being just do you feel comfortable letting it run that long? >> Right. >> Totally. Like I think I mean so to that
[29:12] point like we made it some investments like we we shipped this thing called >> auto review uh where basically like it'll launch an agent to go and do a a review of the app script and be like, you know, and then like you know, kind of have some confidence there which I
[29:25] think is like kind of the it's funny because uh unfort- unfortunately unfortunately it's like as you lean more into AI like writing and building things, you create all these other bottlenecks and pretty much the only solution to those bottlenecks is more AI. >> Yeah. >> Um which
[29:38] is good for I guess me selling the AI
[29:52] Like I don't know about you guys but I was like not excited to like review thousands of lines of code. I'm not good at it. I'm definitely skipping stuff like um and it's like, you know, AI's pretty diligent. It'll it'll get it done. Um so I mean I think that like that's the hope is like when we do apply
[30:04] it to these other places, those things actually happen better. And I think security is a great example where like, you know, we have like a security product code exchange. We have this awesome team working on it and uh it can actually like allow a security like a CISO to like codify their best
[30:17] practices and actually like programmatically affect like essentially programmatically ensure that those things are happening which was not really possible when everything was being like managed, governed, and written by humans. >> Right cuz there's like there's a
[30:29] security design review process and like you're basically count on counting on that human who reads that design doc to read it correctly and you're counting on the author to actually adhere to the design doc which is almost never true, right? Um
[30:41] >> Forget about implementation in on that as well, right? >> Yeah, we're also seeing something similar, right? With like the ability to just like automate even more pieces. The agents have this tenacity that
[30:54] humans just fundamentally can't have cuz like they they don't go to sleep really and they don't get tired. So I do love that about the review pieces. >> Totally. >> I recently someone sent me a three I I sent a 3,000 line PR which is like
[31:07] terrible. I felt awful but we had to merge it quickly. And I sent it to an engineer who This is this is not that big. What are you talking about? It's it's just like you're in reviewing the intermediate
[31:20] states of the agent, right? You almost if you have an issue with a PR you basically have an issue with a plan that generated the PR and you should go talk to the agent instead. >> No, totally. Like we're working on some interesting interface ideas around like
[31:32] reviewing code. Um, you know, like so we like I mean fundamentally like we have this thing called it's the app the Codex app. Um, and it's kind of like a really great way to work with agents. Like kind of our vision for it is just like an interface for like agentic development
[31:45] but not just for coding for basically everything. Um, and a big part of that is like reviewing the outputs of agents. So like in the Codex app you can like you essentially have like a diff view of code. You can like review slides, PowerPoints. You have a browser to
[31:57] review websites. But the interesting thing on like code review is that kind of yeah, like we're like some of the things that we're thinking about is like oh like instead of reviewing the code like let's review the prompts that the person sends. Let's review like let's have the agent
[32:09] potentially document its decisions as it goes. >> Right. >> show those in the review. You know, like I think that like cuz before like code review was interesting as a like way to help teach someone how to write better code.
[32:21] >> Right. >> But now it's like no longer that relevant because they didn't write the code. >> Right. >> So it's like what did they do? Well, it's they were prompting the agent they were reviewing it. So like can we actually make it easy to see like those human decisions that were made? >> Exactly. So you're actually teaching them how to
[32:33] >> You review the prompts that you're you're teaching them to actually uh deliver better prompts in the first place. >> need to know how to like think about the problem, right? Like if you're going to ask the agent to write insecure code, it will go write insecure code. And so,
[32:45] one of the things that we try to do is review people's plans, right? And so that we can actually share these sessions around and say, "Hey, like you told it to write this API, but you didn't tell it to make sure that it respected this off framework or
[32:57] something like that." So, that can help. I do think there's other bottlenecks, like since we've become fully AI-driven at Databricks, we've seen the amount of CI runs that we have literally double or triple to the point where we actually have now completely had to rebuild and
[33:10] change the way we do PR like authoring and testing because we can't like it's unreasonable to actually run CI on every commit. >> Yeah. So, agents behave very differently than humans. I think we all need to like
[33:22] rebuild our software factories. >> Totally. >> Cool. I just realized we only have 6 minutes left and then we did want to give the audience a chance to ask some questions. So, we I think we got a little too engaged ourselves. So, yes, why don't we want to raise your hand? Uh we've got a couple of microphones or
[33:36] runners around here that would love to let you ask some questions here. >> Hi. Uh this question's for uh B Myden Open AI. Sorry, I forgot your name. So, um you mentioned earlier that
[33:50] um you know, you use as many tokens as a P50 engineer at Open AI. So, just wanted to get a little bit more on that front and understand how your role as a PM has changed with Code
[34:04] X Claude code in the picture. >> Yeah, totally. Um I think it's changed a lot, but like in a ways it's just doing the things that PM should have been doing in the first place, maybe. I mean, also I haven't really been a PM for that long. I'm like
[34:17] >>
[34:29] lot more time with our customers. I think like I spend a lot more time on go to market as well. Like I'm focused more on like the business outcomes and then work with very high agency engineers that can basically just drive most things and it's just like making sure that we make these like key decisions.
[34:42] I think that the way that like in the way that AI like maybe is that's like a lot of downstream of like engineers using AI a lot. I think that like product wise like a lot of it is just every like literally
[34:54] everything I do starts with like me asking Codex to do something whether it's like generating documents, like doing customer synthesis. Like at Open AI like you know we spread ourselves across a bunch of different things. So like I'm working on like probably like
[35:06] seven or eight different like product things that you can imagine and it's like a single PM working on. And often it's like because of Codex I can like do like fundamental research on like every customer call that's happened and like
[35:18] go and look at every Slack message and make sure that I deeply understand something in like an hour that would have taken like you know weeks of like investigation. And so like even yesterday like there's like this product that we're like trying to dog food and I like had Codex find every customer
[35:30] conversation from sales, find the account reps, then send a DM to those account reps to figure out the status, and then set up an automation to like review what they said so that that then updates like a spreadsheet so that
[35:42] I can now figure out who to like alpha test with, which is like I think in the bucket of things I would not do if I didn't have Codex. So like I don't know if that like helps paint a picture a little bit.
[35:55] >> Cool. Anybody else? Oh we got some people down here. I'll keep your hand raised so they can the runners can >> I'll start. Sorry. >> Hi. Great topic. Um I'm in a regulated industry, HIPAA.
[36:07] It's a minefield. Okay? So Codex, we can use it for coding in our dev environment. We can't use it for business uh because I can't take the risk of the
[36:22] agent harness dipping into areas that it shouldn't access. For Databricks, we are inside of HIPAA for the core environment, but preview, public,
[36:35] private, beta, etc. make it almost impossible for us to use the frontier technologies. So, I'm asking for a better path for your regulated
[36:48] customers. Any thoughts on that? >> Yeah, absolutely. Um so, uh the LLM capacity and serving products that we have available are HIPAA compliant. Um and by default, we have uh you know,
[37:01] the most secure and non-data retention rules uh that we have that are there in the industry thanks to our partners at at all the various labs. Um so, we should follow up, but that I think a lot of the capabilities, for
[37:13] example, that we showed yesterday um should be available to folks in regulated industries. Uh but also, I think like, you know, this is one of those things where uh we are pushing really hard to make sure that we get all
[37:25] of our products uh you know, HIPAA compliant by default and things like that. Um obviously, I think it's a journey. In the last few years, we've uh I think hopefully improved on that, but we've also increased the velocity of our products. So, maybe that's a little bit
[37:37] of what you're seeing. Um but, you know, happy to follow up myself. I think you should be able to use uh the AI products in particular, or the coding agents and stuff uh as a as a HIPAA compliant customer.
[37:52] >> Okay, we probably have time for one, possibly two more questions. So, I noticed there's some folks down here as well. >> I got you. I got you. >> Thank you. Um so, you mentioned that um
[38:04] a little bit briefly there about uh downstream of the actual software development. So, the rest of the software development life cycle, like after you write code, like we hear all this talk about how I can write code faster, I can make code faster, but how
[38:18] do we optimize the parts downstream of that? You mentioned code reviews, but there's testing cycles, there's release cycles, there's, you know, PM reviews of stuff. Like, how do we optimize that? Because I feel like that's becoming, at
[38:31] least in my business, the bottleneck more so than the actual software development at the moment. >> Totally. Yeah, 100%. This is like our main focus. Um, so fundamentally, you know, there are like specific things where like
[38:43] uh you can leverage AI very easily. So, let's say like code review, security review, um you know, planning and design and things like this. Like, Codex can do all of these things, but I think when it comes to like other parts of your SDLC, that's where
[38:56] uh it requires a little bit more of like you guys are figuring out like where are those bottlenecks and then finding ways to leverage kind of more asynchronous agents or like processes uh to do things. So, one thing that's interesting is like at OpenAI, we have like two roles I think that are relevant. There's
[39:08] like a Codex deployment engineer, and then we also have a deployed product manager that will actually go and like work with customers to help kind of identify those areas that Codex could be deployed a little bit more asynchronously. Um, I think a lot of things that we see is like often like using Codex headless
[39:21] to like, you know, respond to incidents or do like create automated tests, uh even like QA agents that will go in and like, you know, almost like in a CI type of way make sure that like the front end is like correct. Um, so I think it just
[39:34] takes a lot of like going back to the previous topic of like what our engineers doing. I think actually a lot of what we see is like engineers are working on these systems and less on actually like writing the code for the product. And then that actually is like super leveraged. Um, but it's not easy,
[39:47] and I think it's like actually very company to company like what actually needs to happen, which is why we're investing a lot in like kind of the deployment phase. >> Yeah, we also we work with a bunch of customers who are trying to roll out AI, and often they report similar bottlenecks. I I think uh there's a lot
[40:01] of things that you we can do like to automate, use AI more in those processes as well, and that's what we're trying to do at Databricks. But really fundamentally, I do think it requires senior folks in the company with the with the agency to actually change things uh to lean in and and to really
[40:14] understand the impact of AI, the value of it, and why, you know, some of these processes need to change and evolve cuz I think uh you know, at most organizations, that's what's needed to actually re- really restructure the way things
[40:26] are built, right? Like uh our CEO Ali gave give this example to our engineering team earlier this year about uh how long it took steam engines to actually have an impact in factories. And the reason was because it it you know, you had to actually rebuild the
[40:38] factory uh to be oriented around this new process. I think if you just put in a faster engine uh but don't change the rest of the car, like you're not going to move move that much faster um in this case. So, you we you know, you
[40:51] need senior folks in your companies to be leaning in on this. >> All right. Well, thank you very much. We are definitely out of time. I didn't want them to yank us physically off. So, again, I want to say thank you very much to Rohan and Ankit for their time, and I appreciate you guys coming here today.
[41:04] >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.