Sponsored by: Datadog | Experimentation and Product Analytics in the Agentic World

Summary

In an agentic development world where code is generated continuously, A/B experimentation becomes the critical infrastructure for determining what actually works—studies from Microsoft, Coinbase, and Netflix show even expert teams achieve only ten to twenty percent success rates without data-driven validation.
Datadog's release agents automate the full safe rollout process from build to evaluation, while its Bits Data Analyst tool root-causes revenue drops by combining warehouse data with observability signals across the full stack.
The evaluation infrastructure introduced in this video uses synthetic tests, production telemetry, analytics, and controlled releases as circuit breakers to gate autonomous agent changes and prevent broken data from silently corrupting experimentation metrics.

Chapters

FAQs

Why is A/B experimentation the foundation of agentic development?

A/B experimentation provides the data signal needed to determine whether agent-generated changes actually improve outcomes, since even experienced teams at companies like Microsoft, Coinbase, and Netflix achieve only ten to twenty percent success rates for product changes based on intuition alone. In an agentic world where code is produced continuously, the speed and reliability of these feedback signals determines how fast teams can improve.

What are Datadog's release agents?

Datadog's release agents automate the end-to-end safe rollout process, from build through deployment and evaluation, allowing teams to ship autonomous changes with governance guardrails. This video describes them as a soup-to-nuts solution for integrating experimentation into the agentic build loop.

How does data observability intersect with A/B experimentation?

Broken or stale data can corrupt experiment metric signals, causing teams to draw false conclusions about whether a change is working. This video argues that data observability and experimentation must be treated as a unified system, with data quality failures triggering circuit breakers that pause or invalidate experiments before they produce misleading results.

How should teams evaluate LLM-powered features with A/B tests?

LLM-powered features require evaluation datasets and feature-flagged A/B tests designed for agentic behavior, where outputs are probabilistic rather than deterministic. This video introduces agentic observability—combining eval datasets with feature flags so that autonomous agents can run and interpret their own A/B tests within a governed framework.

Full transcript

[00:08] Cool. Well, excited to have you all here at the Data Brick Summit. My name is Che Sharma. I'm on the product team at Data Dog. I was the former founder CEO of a company called Eppo in the experimentation and feature flagging space. Um okay, one more check. Everyone can
[00:20] hear me all clear? All right, perfect. Perfect. So, we're going to be talking about uh experimentation software life cycle, you know, evaluating whether products work in an era where turning out
[00:33] products is easier than ever. Where suddenly you have this firehose of code going out and how do you know what's actually working? So, what I like to start this off, I don't know who here spends a lot as much time on Twitter as I do. Um but, one of the
[00:46] things you see a lot of is uh that this idea that in a world where coding is basically zero-cost, that suddenly taste is this big factor. That,
[00:58] you know, it's really the people who just kind of know what works who are going to be successful in the agentic age. But, the funny thing about taste and judgment is that we've had this killer app for it, this workflow
[01:10] that uh that really helps determine taste for the longest time and it's called AB experimentation. Who here has run an AB experiment before or knows roughly what I'm talking about? Okay. So, for everyone who doesn't, it's basically a
[01:23] randomized control trial. You give half the people your new experience, half the people the old experience and you see who does better. And when you do this process, this very simple process, you quickly realize that the actual success rate of taste is on the order of 30%,
[01:38] 20%, you know, it's much less than you'd think. You know, it's just it turns out it's hard to predict what products are successful. And there's a bunch of published rates and papers from places like Microsoft and Coinbase and Netflix,
[01:50] you know, places that certainly have smart people with taste and yet they are still finding success rates around 10, 20%. So, you know it turns out you need data to guide taste as well and to work in tandem.
[02:03] And you you can kind of see this evidence in the way the leaders of these companies talk. They're always talking about speed speed speed testing testing testing experiment experiment. You know Bezos literally says the reason Amazon is successful is the volume of
[02:15] experiments. And Zuckerberg literally says that the way you win is by learning velocity by experiments. So it's just been the secret sauce of all of kind of modern tech in the internet era.
[02:28] And the reason why it's so powerful AB experiments it's it can really be boiled down to two things. The first is that you suddenly have infrastructure that can makes it safe to just try out ideas. You know you don't know whether
[02:41] it's good or bad I don't know just try it out gather the data. And so this is kind of a safe releases process a kind of QA process. And then the other side is you can actually get the truth of what
[02:54] customers want which as we just saw is a lot less predictable than what you may think. And to give you a sense of what I mean by unpredictability of product I have a few case studies from my past. So my formative career was at
[03:08] Airbnb on the data science team between 2012 and 2017. And there's just a few examples that are really illustrative of experiments and why they're important. The first one is the search ranking team at Airbnb. So if
[03:20] you've ever used Airbnb you know the first thing you do is you search San Francisco or whatever and see a bunch of list things. Um and there was this team the search ranking team that was churning out really good ideas. You know things like figuring out your location
[03:32] preference or matching you with the host preference and all these like very interesting methodological approaches. But the problem they suffered was that Airbnb is founded by a designer and it's very design oriented company and every
[03:45] time you make a new search ranking model the UX looks exactly the same. It's just a map with listings on it, right? And so, it was not a process that's naturally rewarded in a company
[03:57] that, you know, rewards flashy launches. But, they ran all the experiments, so they had all the receipts, and they could prove that they basically drove the entire booking growth of Airbnb in 2013. So, you know, that's an example of
[04:10] what you get with experiments. The less flashy stuff might be the most important. So, the another one that happened a lot is that um uh at Airbnb, if you've ever done a
[04:22] search and you clicked up clicked on an Airbnb listing, you may notice it opens a new tab. That little change of going from I'm looking at an Airbnb listing and it navigates to it versus opens a new tab,
[04:34] that was actually the most successful experiment in 5 years at Airbnb. So, from 2012-2017, just making it that clicking on a listing opens a new tab. The most successful experiment. And so,
[04:46] that also goes to show you that small changes can lead to huge effects. And then, one last uh example is um in 2012 when I joined, Airbnb's big
[04:58] launch of that year was a thing called Airbnb Neighborhoods. The idea was you were looking to stay in San Francisco, um it might say, "Oh, you should consider staying in the Mission." And
[05:10] when you're in the Mission, go look at Tartine, go stay at Dolores Park, do all these things. It was like 10 pages of flashy content. So, these neighborhood guides and, you know, Brian Chesky was on stage doing press releases. There was a lot of confidence that this was the
[05:23] way forward for Airbnb. Um the third most successful product change in 5 years was to nix that product entirely. Just cut it out entirely. So, despite leadership thinking it was so
[05:36] powerful, it turns out it was a huge distraction. That you're going you're going to go book an Airbnb and suddenly you're reading 10 pages of content, and then, you know, the coffee's finished, your dog starts barking, and you're gone. And this thing that could have been a booking is gone. So, it just
[05:49] again, it turns out it's hard to predict what is good. And so, this is why AB experiments has always been kind of the foundation of fast-moving digital businesses in the internet age. But, the And what are those three
[06:02] examples going to show you? So, again, a search ranking team making model changes kind of in the dark, but very powerful. Small changes like opening listing new tab that were quite impactful, or big obstacle changes that were actually bad.
[06:15] Um this all goes to show you that, you know, in a world where you're not testing, you're probably just doing a lot of committee discussions. So, in a world where uh you're not running experiments, you're basically
[06:27] doing a ton of committee discussions around like, is this good? Is this bad? Is it good to open in a new tab, or is that ugly? You're kind of just discussing your way through it as opposed to, you know, just test the thing. It takes like no effort if the
[06:39] infrastructure is good, just test the thing. Uh another thing that happens is you start just rewarding opti- optically big launches instead of the actual needle movers. So, you're kind of misallocating resources instead of doubling down on
[06:52] what's working. And then, you know, you get to what the agentic world is starting to move towards of people just deciding, you know, I just know what's going to work. You know, I I got taste. Uh when in fact, you got to measure this thing cuz
[07:04] I tell you that neighborhood thing, plenty of people with taste really thought it was going to work. But, now just imagine saying, "Okay, I have all these product changes going out. I don't know what's working or not." And the number of product changes
[07:16] just multiplied by 100. Right? That's basically what we're all dealing with. If you go look at the backlog in your GitHub, all those PRs, there's suddenly 10x more of them, and no one wants to launch them, no one wants to read them. So, how do you
[07:28] handle this in the new agent age? Well, the problem is it with current experiment infrastructure, uh you can't really do it. And the reason is, you know, who here has experiment
[07:40] infrastructure? You have it by a vendor or an in-house tool. Okay, maybe like half the audience here. So, one thing you see a lot with with uh experimentation solutions is they're just highly variable on how much
[07:52] time you're going to sink into that, right? Like you go in and sure, if everything just works and the QA's perfect and everything's happy, it can be fast. You can set it up in a few minutes. But, realistically, what
[08:04] happens is there's a error bar of like 2 weeks. Suddenly, the data's broken, there's some traffic imbalance issue, there's much QA issues, and it just takes a long time. And then, you don't end up trusting the results.
[08:16] Um the other side is let me try to fix myself. Uh when you're actually doing the experiments, uh you're often swivel chairing between tools. You know, you're
[08:28] going between like a planning process in Notion, a uh roll-out process in some point solution flagging thing, then you're probably going to Datadog, where we work now, to actually to see if
[08:40] things are healthy and ultimately, you know, some BI tool that's going to actually uh see whether it's good or not. And what you're really doing is uh just you're putting your entire
[08:52] experimentation practice and making it go at a human pace, right? You're you're requiring human synthesis at every stage to push it through. And that's both annoying, but no one actually wants this
[09:04] error-prone, slow experimentation process, but it's also completely ill-fit for the agent gauge, where suddenly you're having intelligence-driven velocity as opposed to head count, right? So, if you're actually going to get this type of velocity, what can you do?
[09:17] Well, you know, we are as we'll get into the next uh section, we're proposing a world where platforms really are advantaged in the agent gauge, where you can bring all of this stuff together, your flagging process, your measurement process, your
[09:30] planning process, and everything to a level where agents can operate it, not humans. So, if we're talking about agents operating experiments, let's kind of decompose it a little bit and see see what has to be true.
[09:42] So, this is a simple maturity curve. Um I won't talk through everything, but the you know, the basics of it are if you start with no measurement, you're basically flying blind. You just know kind of at a total level how many users
[09:54] do I have at a total level how many transactions do I have. Eventually, you bring on some of the tools, you know, the stuff that defined the 2010s, your product analytics tools, your session replay tools, and you now have a feature
[10:07] product level understanding of your products. Like this product has this much usage, this product has this much usage. But again, humans are synthesizing the thing. And where I'm proposing we're going, and I'm going to talk about a bunch of products that allow you to get there, uh
[10:19] you want to create a world where agents can synthesize and act completely autonomously. And suddenly, your product is sort of self-improving in a sense. So, to get there, what has to be true? Well,
[10:31] let's talk about the software process, the software delivery process. I think there are three moments that matter. The three after Well, four if you include building and coding, but let's say now you've coded it, it's sitting in GitHub. Three moments that matter. The
[10:43] first is when you roll out the code, and you want to make sure you're not taking down the app, right? You're not crashing things. That's kind of a typical software delivery process handled by Canaries and the flaming gates and a lot
[10:55] of things. The second thing, okay, now we've made sure we're not taking down the app. How can we know on a QA level that our product is basically successful? People can actually complete tasks. They don't get stuck. There's not some broken button. It's not like broken on tablets
[11:08] or some old Android version and whatever, right? So, kind of QA process, post-launch health. And then the third moment is when you actually have to decide whether something works or not. Is it actually moving the needle? Is it
[11:20] actually improving business metrics? And each one of these moments consumes a lot of signals. Like at every single stage, you are a humans are synthesizing a bunch of tools, right? They're looking at their flagging tool, their CI
[11:32] systems, their product analytics, and you'd probably their warehouse to actually like data bricks to figure out if it's moving business metrics. So, what does that mean? It means that when you're doing any of these moments,
[11:45] the workflows around these moments require synthesizing signals. It's about what is that diversity of signals you can bring. And to kind of talk about where this is going, I'm now going to bring you into Datadog and our perspective. So, for anyone who doesn't
[11:58] know about Datadog, we have a lot of different products. You know, it's historically been known for application health stuff in, you know, real user monitoring on front end, application monitoring on back end, stuff like that. But, we actually have quite a lot more.
[12:10] Uh we have product analytics capabilities, data observability, uh feature flagging experimentation. And because we have all these things together, we can do these workflows, these sort of holistic moment workflows,
[12:22] that actually allow agents to operate autonomously. So, here's a Here's a simple one. This moment of safe roll out. The sort of I want to make sure I don't take down the app sort of roll out.
[12:35] Uh we just released uh last week at our conference a thing called release agents. So, what this is is our homage to the AI age of safe rollouts in saying like in the process of doing a rollout,
[12:47] you're doing more than just feature flag operations, you're doing more than CI operations, you're doing a lot of things and they all combine, all these signals. So, what this system will do is it's a soup-to-nuts uh release process. It starts from GitHub PRs, where it's going
[13:01] to rerun your CI testing or validate you've run it. If there are any issues, the agent will self-heal, it'll repair the code to pass this unit test. Then, you'll run a bunch of synthetic tests using synthetic traffic, you know, fire
[13:14] into staging, make sure that you can run basic scripts, you can roll go through basic pathways, and that everything works. Then, it will load in feature flag or if you or notice if you've done it already, and do a canary rollout, 1% 5% 25% checking observability metrics,
[13:28] product analytics metrics. Um it will add telemetry. Make sure it's set up to be monitored in the future. Um and then maybe launch an AB experiment. And so, this is the type of thing where, you know, an honest assessment of what a
[13:41] release is includes all of those steps. And the problem is there's never been something an agent can just autonomously go through the entire way. And the reason why you haven't had that is that you it requires a vendor to be both a
[13:53] feature flagging vendor and an experimentation vendor and a synthetics vendor and a CI orchestration vendor. And again, platforms are really well suited to thrive in the agent to AI. And so, Datadog is very fortunate to have all these together and to be able to
[14:05] release tools like this. So, now this hopefully is what's going to happen to all that giant pile of PRs in GitHub. Then, let's go to the post-launch health period. This is where uh you want to
[14:19] make sure your product basically works, even if you're not taking down the app. We just launched a data agent. Uh it's called Bits Data Analyst. So, if any of you have used these sort of data agents like Databricks Genie is obviously here
[14:31] or, you know, uh various other vendors have them. Uh this is a way of giving human language interface queries to all of your co-workers. But, in contrast with many of the other agents out there,
[14:44] Datadog has a way wider diversity of data. So, in addition to having your warehouse data with a semantic layer on top, a context layer on top, you have product analytics data, you know, all the funnels and usage across all of your
[14:57] features. You have uh observability data. How is your application doing? And so, a simple query might weave together all of these data sources in interesting way. So, as give an example, let's suppose every day I wake up and I have a
[15:10] report of my revenue, my business metrics, and I notice that the revenue in Asia is down now for a multiple days, weeks in a row. The agents can start from there, notice that that's a problem. Then, notice
[15:22] that, oh, the problem is there is these funnels, these journeys that have a lot of friction that are not succeeding in East Asia. And the reason why they're not succeeding is that there's an endpoint that is very slow in those countries.
[15:35] And that this is a way you might remediate it, right? So in one single root causing chain, we have gone from warehouse metrics to product analytics funnels to observability. And that's the power of the DataDog bits analyst.
[15:50] And again, it's a This is something an agent can operate for figuring out post-launch health. But you can't really have it unless you are both a product analytics vendor, a RUM vendor, APM vendor, etc. So again, an advantage to the platforms.
[16:04] And this is kind of what the whole picture looks like. So, you know, I I like doing this sort of thing at a Databricks audience cuz you all live data a lot and trying to think through how do you map it all? But what way our system works is we have this data agent,
[16:16] you know, you can ping it in Slack and Cloudera whatever. It's going to hit a contact layer that weaves together all these data sources that sits on top of observability product analytics warehouse native. And underlying all of it is our data observability capability. So, you know,
[16:29] we DataDog acquired a company called Metaplane. It's in data observability category. It's meant to make sure your pipelines are healthy, your data is healthy. So that when agents go and autonomously operate, they're not operating on noise.
[16:46] Then let's get into post-launch health, you know, the process of deep diving and validation. Uh On the experimentation side, we have the idea of protocols. So, think about it. When you're running an AB test, there's actually a lot of high complexity
[16:59] decisions to make. Who should be in the test? What metrics should I track? What should I do if one metric goes up and one metric goes down? Statistically, how long do I need to wait to have signal versus noise? You know, there's a lot of
[17:11] things that are just the average tech worker is not equipped to do. And so what we have done is create a very templateized approach to AB testing so that, you know, companies can set up a world where they say, "Every time we run a homepage experiment, you know, you're
[17:24] going to use these metrics, you're going to wait this long, you know, and if depending on the movement, this is going to happen next." So, you have sort of created these paved paths of how do we make decisions under uncertainty that again an agent can just do, right? And
[17:36] so, this gives, you know, the organization confidence that we're making good decisions. It gives agents the ability to like just run experiments seamlessly without consuming a lot of tokens and making bad decisions. And,
[17:48] you know, you now have something you can observe and say like, "We have a set of policies. Are those policies suiting us? Is a P value of 0.05 the right one or should it be 0.2? You know, should we add these other guardrail metrics in? Are they kind of slowing us down?" You
[18:01] can actually observe everything at the protocol level. But again, it requires you to be an experimentation vendor, a feature flagging vendor, a data observability vendor, all these sorts of things. Um another simpler one uh is data
[18:13] observability. So, I'm going to share you know, around here you've probably seen a few vendors that offer the ability to check for anomalies in your data, check for anomalies in your pipelines. Um but funnily enough, there's never been a vendor with both data
[18:25] observability and experimentation together. And so, now we're linking these so that as you're making experiment decisions, the business metrics underlying them have uh the ability to notice if there is issues. Because the sneaky thing
[18:38] about experiments and and randomization is suppose your revenue metric is completely broken in some device or something like that, right? In experimentation lands, it will be equally broken on control and treatment, and so you might not notice, right? So,
[18:51] here you can actually stamp it and approve it with a really comprehensive anomaly detection regime. And then one last one uh that I'll talk about, which again emphasizes the platform advantage in the agentic world,
[19:03] is uh agentic observability. So, suppose you're making agents and you want to improve their accuracy. Well, uh Datadog has the capability um a very few other vendors out there as well to do eval
[19:15] data sets. You know, you create these big eval data sets and you run agents through them to see if they're accurate if they're improving on the past. Now with one click, you can take some candidate model and put it in production
[19:27] behind the feature flag in an AB test because one of the things you notice the off the test that win on offline simulation only like 30% of them actually win in production. You know, so there's a lot more you have to improve. And so but again, it requires
[19:42] a vendor to be both an agent and LM's or building vendor and an experimentation vendor. So, you know, what do we have in common here? It's you have a lot of workflows that are happening as the process of rolling out code. These workflows currently are involving a lot
[19:55] of synthesis that on humans. And so even as you're spending enormous amount of money on all the coding tools, you are not getting the product velocity you deserve. And to actually get that, you need all these evaluation and delivery workflows
[20:08] to be completely automated to operated by an agent which leads to new products and platform advantages. So, I'll pause there. Thank you for joining for the talk. Our Datadog booth is literally right over here. So, we'd love to talk more about these products
[20:21] or anything else. Thank you. >> Woo!

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.