Databricks on Databricks: Build Your Own Experimentation Platform

Summary

This video shows how Databricks built its own full-stack A/B experimentation platform on the Databricks Data and AI platform to run hundreds of product experiments per month, using Unity Catalog Metric Views, Lakeflow Serverless Jobs, and a Lakehouse App built in two days.
The platform architecture consists of four layers: a foundation on serverless compute with Unity Catalog, a centralized Metric Store with reusable Metric Views, a YAML-driven analysis harness with a statistical library, and a Lakehouse App user interface.
A data scientist—not an expert engineer—built the platform, demonstrating that teams can construct a fully owned, extensible experimentation system on the Databricks Data and AI platform without large engineering investment.

Databricks on Databricks: Build Your Own Experimentation Platform

Commercial experimentation platforms promise everything – feature flags, metrics, and analysis, but often limit flexibility when you want to go beyond the basics. At Databricks, we took a different approach: we built our own full-stack experimentation platform entirely on our internal Databricks and used it to power hundreds of product releases every month. In this talk, we’ll show how you can do the same. You’ll learn how we: Use Unity Catalog to centralize and govern feature enablement data Define reusable, trustworthy metrics with our internal Metrics Platform Run scalable experiment analyses with Lakeflow Serverless Jobs Deliver fast, intuitive user experiences with Lakehouse Apps Beyond the platform itself, we’ll share how we extend it with advanced methodologies that go far beyond what commercial tools support. Whether you’re a data scientist or platform engineer, you’ll walk away with a blueprint for building a reliable, extensible, and fully owned experimentation platform on Databricks.
Talk By: Donghan Zhang, Sr. Software Engineer, Databricks ; Nicholas Stanisha, Data Scientist, Databricks ;

Chapters

FAQs

Why did Databricks build its own experimentation platform instead of using a commercial tool?

This video explains that commercial experimentation tools imposed limits on flexibility, particularly when Databricks needed to go beyond standard metrics and apply advanced statistical methodologies at the scale of hundreds of product releases per month. Building on the Databricks Data and AI platform allowed the team to define fully custom metrics, own the entire analysis pipeline, and extend the platform with methodologies no off-the-shelf product supported.

What is Unity Catalog Metric Views and how are they used for experimentation?

Unity Catalog Metric Views are reusable, governed metric definitions stored in Unity Catalog that provide a centralized, trustworthy source of metrics for experiment analysis. This video shows how Databricks uses Metric Views alongside table value functions to track experiment assignments and compute business metrics consistently across every experiment run.

How does the Databricks experimentation platform use Lakeflow Serverless Jobs?

Lakeflow Serverless Jobs are used to execute the experiment analysis harness at scale, processing experiment assignment data and applying a statistical method library to produce results for each product release. This video explains that the serverless approach means the team never needs to provision or manage compute clusters for experiment execution.

How long does it take to build a Lakehouse App for an experimentation platform?

The Databricks team built the Lakehouse App user interface for their experimentation platform in two days, according to this video. This is presented as evidence that the Lakehouse Apps framework dramatically reduces the engineering effort required to deliver a fast, intuitive user experience on top of Databricks data and AI infrastructure.

Full transcript

[00:09] All right. Hi everyone and welcome to this last talk of Data and AI Summit. We appreciate you guys coming. Uh, thank you. Uh, build your own experimentation platform. Hopefully we can go out on a high note with this one.
[00:21] Um, just to clear the air, not going to be talking about test tubes and beakers today, no kite and thunderstorm type stuff. We're talking about a different type of experiments that are more like a
[00:34] framework for making decisions about your business. So, we're going to tell you how we use experiments at Databricks to make our products a little bit better every day. Uh, we're going to tell you we're going
[00:47] to share our journey, you know, starting from running no experiments at all and growing to running hundreds of experiments every month. And uh, finally we're going to show you how we assembled everything on Databricks. Uh, hopefully we'll convince
[00:59] you not just that this is possible, but also that it's easy to do. But before I begin, let us introduce ourselves. My name is Nick Stanich. I'm a data scientist at Databricks and
[01:12] something that might be surprising is that I'm also the primary contributor to the experimentation platform. Uh, so I'm proof that you don't need to be an expert engineer to build a software platform on Databricks.
[01:24] >> Hi everyone. My name is Dong Han Zhang. I'm a software engineer from data platform team at Databricks. And our team manages one of the largest Databricks workspaces on this planet.
[01:36] We create tools and building blocks for users like Nick to unblock business insights and build their own products like experimentation platform that we will talk about this today.
[01:48] >> All right, so on to the experiments. So, like we said, we're not studying the universe or developing vaccines. Maybe some of you guys are, Uh but we're selling software, right? So, what are we
[02:00] talking about when we say experimentation? So, let's start with the basics. Hopefully, everyone remembers the scientific method from back in your school days. This is how we learn about the world by
[02:13] forming a hypothesis, testing it, and seeing if the data supports our theory. Right? You know it, I know it. Hopefully, we all know it. And it turns out that this process is
[02:25] not just useful for science, it's also useful for business. Companies like Google and Netflix very famously used experiments to drive their development, and it gave them a huge advantage over their competitors who
[02:37] weren't doing that at the time. Today, experimentation is part of the toolkit for modern companies. Your hypothesis might be that a new page layout could drive more sales on your
[02:50] website. Your experiment might be to say, "Okay, for the next 2 weeks, I'm going to take all the users that visit my site, I'm going to show half of them the old version, I'm going to show half of them the new version."
[03:03] Through analysis, you attempt to see if one group did better than the other. And if it works, you ship it, and if not, you try something else. And that's the loop that helps you constantly improve.
[03:19] So, let me ground this with a real example from Databricks. So, first, show of hands, who's actually seen this before? All right, almost everybody. Uh everyone who didn't raise your hand, I guess you've never made a mistake before. So,
[03:31] please visit the Databricks jobs page and uh send your application in. Um but if you're not familiar, this is a Databricks feature called quick fix. And the job of quick fix is basically when you make a mistake in your query,
[03:45] quick fix catches it and tries to fix it for you. And behind the scenes, we try to measure if quick fix is doing a good job so that we don't need to send you all surveys about if you like QuickFix or not.
[03:58] So, behind the scenes we measure if that suggestion is actually accepted. We also measure if the fix actually fixed the query. And we take a look at how long it took for QuickFix to come up with the answer
[04:10] as well. As you can imagine, there's some noise when we're measuring this, right? Sometimes the query is actually fine and it's the data that's kind of messed up. Think about how you can cast a string to a date, but you can't cast every string
[04:23] to a date. Right? The user might not want to actually fix that. They might just want to go fix the data instead. Uh sometimes there's, you know, internet issues, rate limits that make the response the whole overall back and forth slower, right? Lots of things can
[04:36] happen. So, what experimentation gives us is a framework to cut through the noise, focus on the stuff that's within our control, and see if it's actually working or not.
[04:48] So, with experimentation, we try out different things behind the scenes. We try different models, we try new knowledge bases, we try new system prompts. And using the scientific method, we make comparisons to find the best one, which
[05:01] is not always obvious, right? We want QuickFix to be accurate, but we also want it to be fast. And sometimes those two things are actually at odds with each other. So, with experimentation, we can do the comparison and make decisions based off
[05:15] of the entire picture. Another place we use experimentation is in free edition. So, you might have seen this sign-up page. Free edition is the zero-cost offering. You can go to your browser right now and sign up for free
[05:27] today. And just like QuickFix, we want to measure behind the scenes if free edition is doing a good job. So, what does that mean for us? It means, you know, did you complete your sign-up? Did you go on to become active
[05:39] on Databricks? Did you upload some data, run some queries, maybe even take a course, right? So, exactly like QuickFix, we use experimentation to try out different versions of free edition and compare
[05:52] them along these axes. Ultimately, we hope this helps us find the best version for our users. And that's not all. Basically, from top to bottom at Databricks, uh you know,
[06:05] engineers and and product managers and everyone are able to make their area of the product a little bit better every single day using experiments. Today, right now, there is hundreds of changes being tested every month on
[06:17] Databricks. And that number is actually going up. But, we didn't start there. We had to build up to get to this point. So, I'm going to hand things over to Dong Han. He'll walk us through the journey and
[06:29] tell us how it shaped the platform that we have today. >> Thank you, Aniq. Experiments are great. It turns product development from guessing into learning.
[06:41] But, manual experience do not scale. The first experiment can be scrappy. You go to software engineer and ask them to set up the assignments.
[06:54] And then you go to a data engineer to build the data pipelines and generate the metrics. And then you go to data scientist to run some tech statistical analysis, and then you come with a result.
[07:07] That works only once. But, then another team wants to test the sign-up workflow. Another team wants to test the model change, the query performance, and go on and on. So, every time you repeat this process
[07:21] over and over again, setting up assignments, generating metrics and data, and then running analysis. At the point In that point, the problem is not assignments or experiments are
[07:34] useful or not. The problem is that every experiment starts from scratch, and it is very manual. If it's manual, it is hard to become a habit.
[07:47] That is one and the why you need experimentation platform. A good platform can enforce the right thing as a default. Without a platform, every experiment is
[08:00] just a custom project. But the with the platform, experimentation is a reusable workflow with metrics, assignments, and analysis built in.
[08:12] And ultimately, we want to provide a product data science, so that everyone can run analysis, can run experiments without stitching together. That's the application layer we built.
[08:28] Great. Now we know we need experimentation platform. But building a platform is very hard. How about the buying one? Yes, we considered the buying external
[08:40] experimentation platform seriously. And commercial tools provides us lots of great features, like fancy UI, assignments tracking, and analysis templates.
[08:52] However, for us, the center of the gravity was already at the Databricks. Our data was there, our pipelines were there, our governance were there. And increasingly, our AI agents were
[09:05] also there, too. So adding another separate system means that we were adding another place where our metric definitions could drift, where our access control had to be managed separately,
[09:19] and we need to do our governance to in two different spaces, and our users have to adapt their workflows into two different systems. So the question
[09:31] we got ask ourselves is, instead of moving data and logic to external space. Can we bring experiments back to where data already lives?
[09:43] So, that's our bet. We decide to build on Databricks, not from scratch, but by assembling the building blocks that the Databricks already provides.
[09:55] On the foundation layer, we have service compute. With service compute, we don't have to worry about uh Spark configurations, like uh sizing, resource planning, none of
[10:07] that. We also have Lake Flow jobs, which help us schedule, monitor the job runs and data pipelines. And last, we have Unity Catalog, which give us governance, access control, and
[10:20] the lineage. So, the work what left for us was not rebuilding the infrastructure layer, but it was just reassembling the experimentation layer. They are metrics,
[10:33] assignments, analysis, and user experience. So, with Databricks, building experimentation platform is easy.
[10:47] Let's start with metrics and let's see how we really build this experimentation platform layer by layer. Metrics may sound simple, but for experimentation, it is contract between experiments and this contract behind
[11:00] every product decision. So, first, let's imagine this. We have a product manager and goes to the AI agent and ask, "How many active users did we have last week?"
[11:12] And the AI agent search the catalog, finds the table that relevant, write SQL, and it comes up with a confident answer. Active users were 1.61 million, up 8%
[11:24] week over week. And even attached the SQL query here. That sounds very useful. But then someone from the growth team say, "Wait, this is not the number we
[11:36] are looking at." And someone from the finance team says, "We even have a different table for that metric." So, what's wrong?
[11:49] Product team count anyone who did any event. So, that's 1.61 million. But, growth team only count anyone who filed a login event. So, there's a login filter here. So, it's a less than 1.01 million.
[12:02] And finally, finance team, you know, finance people are the most serious people in the company. They're very They're very restricted on these filters. So, they add, "Okay, we need to add to paid action." That's less than 1
[12:14] million. So, three teams have three numbers. And to be clear, none of these numbers or none of these definitions are wrong. Everyone can be valid for the specific
[12:26] purpose. So, the question is not this definition is wrong. The question is we started using the same name for all of them. Both human beings or agents,
[12:38] we cannot tell which one represent the truth. So, two people can run experiments on the same metric and then they got different answers. And both technically, both answers are correct.
[12:52] Our experiment cannot build on top of these unreliable metrics. Otherwise, the decision to launch or roll back a product change may only depend on which
[13:04] definition you are looking at. To solve this, thanks to Databricks, we have a building block called Unity Catalog Metric Views. It is a great building block cuz instead
[13:17] of hiding the metric definition inside the dashboard, a notebook, or copy the SQL query, we define the metric definitions explicitly in a YAML config. And this YAML config is stored in the
[13:30] Unity catalog, where both human beings and agents can read from. But metric views alone does not solve the problem we mentioned earlier. Because you can still end up with three
[13:42] different numbers, three different weird and confusing definitions. For example, this metric view is defined in my own health catalog. Other teams may not have access to this
[13:54] catalog or this table. And other people may do not have the context on this table. So, with that, we built a central and a global metric
[14:06] metric registry. We call it Metric Store. Metric Store is our internal metric registry, where important business metrics are defined, certified,
[14:19] and shared across the company. Defining means the SQL expression of the metric is defined in a YAML config. And this config is checked in as code,
[14:31] so it can be reviewed and versioned. We even provide a change log for this definition. Certified means that those definition is reviewed by
[14:43] a month a group of people. They can be a console meeting for that. And leadership trusts this definition for decision making. And also, high quality is guaranteed for certified metrics. Strict SRA, clear
[14:57] ownership, on-call support for data delays, you name it. And third, shared. Shared means those definitions are shared among notebooks, experiments, AI agents, and so on.
[15:13] So now, when an AI agent asks business on answers the business questions, it does not have to guess which table to use. And when we run experiment, everyone look at the same metrics. This is goal of the semantic layer.
[15:27] Define once, reuse everywhere. This guarantees that everyone in the company speaks the same language. Now that we have a trusted definition
[15:39] for metrics, experiments need one more thing, that's assignments. Metrics only describe what happened in the experiment. The conversion rate went up or down.
[15:54] Well, assignments describe another thing. Who got which experience? Which user in group treatment, which user in group control? So, let's get into how we track
[16:06] assignments with the tools in Databricks. First, the standard data model for assignments is that we have a simple event log at the bottom, and that's raw events,
[16:18] like exposure data. And on top of that, we define assignments. For example, say I went to your webpage and I saw version A, and now I'm considered as a group A.
[16:32] Then I leave your webpage and go get lunch. I'm still considered as group A because that is what my experience was. My assignment group does not flip, does not change, unless I return your webpage
[16:46] and saw a different experience, like version B. Then my group changes to version B. So, an assignment is just exposure over a time window for a given entity.
[16:59] That's essentially semantics. So, here's the move. We don't build another system for tracking assignments. We just the model assignments data as measures and dimensions and entities in
[17:11] the metric views. So, metric views are also what we where we define business metrics. So, both metrics and assignments are defined in the same way. Are governed in
[17:23] the same way. And then more importantly, we can enforce a shared entities. Those entities like user ID, workspace ID can be later used as a join keys so that
[17:36] user can easily join assignments with metrics. So, we just a reuse the layer of the magic. And then let's take a closer look.
[17:54] So, here first we have exposure events as a raw table and every row is a event. On top of that, we have a magic view as a YAML config. We define dimensions, measures, and entities.
[18:06] For the measures part, we have start time and end time of exposure. And with that, we can get the treatment duration, which is a very important metric. And for dimension part, we have very
[18:19] complicated dimension to track the assignment flipping. It's a a very complex time window function. It looks fancy. It looks very complicated, but the question is whether
[18:32] the end users need to know all these implementation details. Probably not. Right? This dimension is very useful for sure, but the end users does not care. They only
[18:45] care please tell me which user, which assignment group this user belongs to. Right? So, with these dimensions, we decide to build on top of it, which is a single API.
[18:58] This single API is a table value function called assignments. A table value function is essentially a UDF that returns a table. You hand it in an entity, a version, a flag ID, and you get it back at user
[19:11] level, assignments history with the same format every time. It hides messy grouping logic and enforces right filters. So, nobody can scan a billion rows by accident.
[19:24] And it also limits the way that we can query the metric views and online underlying tables. We make doing the right thing easy and it makes doing the wrong thing even harder.
[19:38] So, for example here, they can just select from assignments from with entity equals user, flight ID equals my flight, and the version equals two. And then you can get this table
[19:50] with the same format every time. It is the assignment history for your specified filters. So, now we have successfully built assignment tracking with Databricks using metric views and the raw events.
[20:05] Let's zoom out a little bit. This is high-level architecture for assignments tracking. We have different services populate raw events to a raw events table. And we can onboard a new service easily
[20:17] cuz we provide a shared interface. Every new service onboards the same way. We also have metric views built on top of it cuz there the assignments can be measured as a semantics.
[20:30] On top of that, we have unified API to abstract away all those complexity. And finally, we provide assignments to our end users. And more importantly, with those dimensions,
[20:42] we can it is also a a health metric that we can track for the overall system health. For example, if the number of the flips is unexpectedly high, then it means probably means that the bucketing
[20:54] service may be wrong or the setup for the assignments is broken. So, if you just remember one thing about this architecture, it that the end users do not have to know the implementation
[21:07] details like what the raw table looks like, what column should I look at for the duration of the treatment. How to handle the assignment flipping? They don't do They don't need to know that.
[21:19] So, as a platform provider, we service exactly what they need which are the final table we have.
[21:31] Now, we complete the full picture of the semantics layer. It is the data foundation of experimentation platform. Metrics answer who Metrics answer what happened and assignments answer who got which experience.
[21:43] So, both having um trusted, reusable data. But, having trusted with data is not the same as having trustable decisions. We still need to choose the right method for analysis and make the
[21:56] workflow simple enough for users to use it safely and over and over again. So, now I will hand it over to Nick to talk about the next layers, analysis and
[22:10] user experience, and how we transform this data foundation into a product experience for experimentation. Nick, back to you. >> Thank you.
[22:24] Okay. So, now we have metrics and assignments. We're not quite done. We're close, but we're not done. Just one more thing to solve in the analysis layer with one sticky problem remaining.
[22:36] And that problem is you can't apply any analysis to any data. Kind of a bummer, right? Cuz we just spent all that time building a nice data foundation and now you're telling me that there's still more stuff to figure out.
[22:48] Yes. And if you notice we didn't put any of this stuff in the metrics and assignments layer. So, now we need to figure it out here in the analysis layer. So, what matters to an analyst? Well,
[23:03] your data collection matters. Are you observing a snapshot like you would if you took a survey? Or are you observing a group of users over time like a time series? The assignment process also matters.
[23:15] Were you able to do a fully randomized assignment for your experiment? Did you have to do a phased rollout? Or did you have to do something like an opt-in private preview? And obviously the method you choose
[23:27] matters, and not every method is compatible with every other type of data. And the real tricky bit about this part is that you can get this wrong in pretty bad ways. So, you can make a mistake
[23:40] that maybe makes your confidence intervals too wide, right? Then you're not being as decisive as you could be, but maybe it's okay cuz I guess you're being cautious. But on the other side, you could make a
[23:52] mistake that makes your confidence intervals artificially small. And that's how you make wrong decisions and waste millions of dollars. And the really scary part is if you're not an expert, you probably can't tell
[24:04] what mistake you might be making and which direction that mistake is going to send you. So, what we really need to figure out this layer is some sort of bridge, right? The problem fundamentally is that the people
[24:18] who can change your product are not the same people who understand the math. Right? So, here on the left side, you have, you know, engineers and product managers and other people who say, "We produce all the data
[24:31] and we can make changes to the product, right?" And on the right, you have data scientists and analysts who say, "We understand all the math." And we need to allow these two people to work together in a scalable way that's
[24:44] not just the people on the left asking the people on the right to do analysis for them. So, really the core question of not just this layer, but the whole platform is how do we allow non-experts to access
[24:56] these tools safely. And we broke this down into three steps. The first thing we did was we created a really precise specification for what an
[25:08] analysis is. So, in our platform, we said an analysis is an object that accepts simple arguments like strings, numbers, and lists.
[25:20] It also declares what data it's compatible with. So, you see this uh example analysis here would accept cross-sectional data like from a survey, but reject time series data as an example.
[25:32] And then the third thing, obviously, it has to return some structured data that we can understand. Under the hood, this is a software library that anyone can contribute to, and it's maintained by the data science team.
[25:48] The second thing we did was we created a harness to run the analysis. And the job of the harness is to accept any analysis, run it, and then store the output in this data model. You see here the data model's basically just a relational database that we define in
[26:01] our lakehouse. The harness can also handle any kind of stateful logic like if we want to skip an unnecessary run. Like, we just ran the same analysis 5 minutes ago, and we know the upstream data didn't change.
[26:13] So, we can just skip that analysis run and return a a slightly old result to you. Those are the things that the harness does. And then on top of these two things, we put a
[26:25] plain text interface on top so that anybody could use it, right? So, we created this YAML specification, very similar to a metric view, and presented this as the way that
[26:37] non-experts can actually interact with this stuff. So, it's kind of, you know, drawing a line back to what we talked about before. The metrics and assignments build the foundation. The analysis encodes the rules that, you
[26:49] know, what will apply to what, and the harness handles the execution. This layer is the layer on top that allows you to mix and match. So, you see, in the data section, we have the names of metrics.
[27:04] And then we have also a set of arguments that we can pass to the assignments function that Don Han talked about. And we have all the other information you need to aggregate or filter your metric queries. And then in the analysis section, we
[27:17] basically pick the method that we want to use and point it at all the different fields in the data. And having this plain text representation is awesome. So, one, humans can read it. I'm hoping that, you know, you can kind of grok what this is
[27:29] about just at a first glance. Also, machines can read this. Agents can write this. We can store this in tables. We can send it to jobs as parameters. We can send it over the wire to hit an API. Right?
[27:42] There's a lot of things we can do when we have this nice interface. So, taking all that, we now have a way that people or agents can show up and define an analysis and say, "I want this
[27:55] analysis to be run." Let's get into how we actually run those analyses. The secret is basically Lake Flow Serverless and Unity Catalog. Maybe it's not that much of a secret, but
[28:08] basically what we do is we grant users permissions to interact with different resources, mostly Lake Flow jobs. Lake Flow handles all the different execution types that we care about. On demand for running your analysis right
[28:20] now, scheduled for updating your analysis over a period of time, and event triggered for automating your analysis when external events happens. Regardless of the execution mode, everything writes back through the data
[28:33] model to the same Lake House that all our other data stored it. So, it feels nice and elegant for us. Besides that, there's not too much to talk about, really. We benefit tremendously from the auto scaling compute, the transactional guarantees
[28:46] from the Lake House, and the top-to-bottom governance with Unity Catalog, but in this layer it's really just putting the pieces together. So, one last thing, this
[28:58] does not really look like anything yet. This is just a diagram. And that's where the user interface comes in.
[29:10] That system diagram a few slides ago is too complicated to show to a casual user. So, we put a Lakehouse app on top to smooth out the interaction with those different resources and also give users a gateway to put together the data that they need to run
[29:23] the analysis. The built-in Lakehouse app tools for user authorization make it super easy to allow users to put data together, which is great because that's in a lot of the time like 90% of the job of running an
[29:36] experiment is just putting together a useful view of the data. The app can authenticate either as itself or on behalf of the user to access some other resources from that uh system diagram before. And the user does not know and does not
[29:49] need to know anything about the system under the hood. Everything complicated about app development, that's the governance, the authorization, even the deployment, is handled by the framework and we just get to figure it out and basically build the
[30:02] user interface on top. Let me show you one particular user flow in our real app. So, let's say user comes to the app. They want to put together some data.
[30:14] They want to run the analysis right now and see the results right away. This is how that actually looks on our app. User clicks create an experiment and they're prompted to put in some
[30:26] basic information like the name of their experiment and the entity which is also the measurement level. So, you recall from before, this informs the one of the arguments to the assignments API and also determines what
[30:38] metrics are applicable. Paging forward one, user's given a form to select from some known feature flags, and this fills out basically the rest of the stuff in the assignments API.
[30:54] After that, they're prompted to select their metrics. I see a lot of photos happening. >>
[31:08] that we talked about. One last chance to review. Photo up. >>
[31:21] >> And then redirect to the results card, right? Now, I'm not showing this to you to flex our app development skills, right? Uh this isn't exactly fancy stuff. Um but, this is really simple, and it's very powerful.
[31:34] So, when we did this, we prevented the user from inputting a lot of bad data. We validated the overall configuration before we ran it. We facilitated the user's actual execution of the analysis, and we wrote
[31:48] back to the shared data model. So, that's a lot of stuff that the user doesn't really have to worry about. And one additional surprising thing about this is we built the initial version of this app in 2 days.
[32:01] So, that's like from the diagram that you saw before to the app, less than a full work day for one person to put together something like this, which is amazing.
[32:14] One last thing before we move on is automation, right? When you have a framework that allows you to give users secure access to resources, the sky is really the limit when it comes to extending that with different accessories and automation.
[32:26] Basically, it's the same bag of tricks. The app facilitates the permissions either as itself or on behalf of the user, and then the code does whatever you want it to do. So, we can continue to just kind of tack
[32:38] on to the same sort of system diagram. Here's a list of some of the actual stuff that we've automated on top of our experimentation platform, uh just by stacking on to this basic model. Some highlights, at least for me, are
[32:51] troubleshooting missing data. So, if your data frame shows up empty or full of nulls, send an agent to go figure out why. Uh sending automated reports, Slack messages, and emails, always useful.
[33:03] Uh and maybe the coolest thing, in my opinion, is sending health signals back to our release automation pipeline. So, that we can actually uh roll things forward if they look healthy, and pause or roll them back in the uh in the event
[33:15] of regressions. So, we're coming up at the end here. I just want to ask the obvious question, you know, is it worth it? We did a lot of thinking when we could have opened the the checkbook instead. Um
[33:29] we certainly think it's worth it. Uh but, to demonstrate that, I want to share with you a story like a success story from our organization. And it's about everyone's favorite topic, which is making queries faster.
[33:43] So, little primer. We have a query optimization team that works every day to make your queries fast. And it's really, really hard to do that. It's basically like a research problem. This whole pipeline comes into play
[33:57] anytime you run a workload on Databricks. The early stages check that everything exists. The optimization folds your expressions into more uh efficient forms. Starts pushing things down as far as possible.
[34:10] Cost models take different options from the physical plan around different join types. Try to pick the best one. And basically every single thing here uh can be the the topic of an optimization
[34:22] change that we want to make. And we were making these for years, but we had a really big problem, which is that we didn't have a great way to actually measure our improvements. We had benchmarks, which are great, um
[34:35] but they're not really representative of real life. And we also had real data, which is representative of real life, but it's very hard to analyze. It's unpredictable, it's not in the vacuum like your benchmark is. There's a
[34:49] lot of other variables that factor into the run time, and they factor in in really non-obvious ways. Um something that was especially important for us was to understand all the potentially interesting segments of
[35:02] queries that can be run. That's like reads and writes, Spark SQL versus SDP, different types of joins, and many more. So, we wanted to do like these massive measurements over potentially millions
[35:16] of interesting different segments of the types of queries that users could write. And what we found is that off-the-shelf analysis just didn't scale, and we were kind of stuck.
[35:29] That was until one day a data scientist was able to take the analysis that we want to do and implement a really efficient version in PySpark. The first one had to do two passes, and it completed in, let's say about an hour to do like an optimization analysis on
[35:42] an enormous fleet of sample queries. Um but that was a big improvement in that at least it actually ran. And then later we optimized that even further to do only a single pass. And this unlocked a massive productivity gain for us.
[35:58] You might recall AI Summit a few years ago, we shared this chart of Databricks SQL queries getting faster over time. This uh two-pass, you know, measurement um methodology is actually how we measured
[36:10] this, and it took a data scientist to kind of figure out how to do this, and we shared this at DAIS 2024. A short while after that, when we built our experimentation platform, we were
[36:22] able to take that analysis and apply it to every single feature release. So, something that used to take a dedicated, you know, session with a data scientist is now happening all the time for every single optimization that we
[36:34] ship. And a little while after that, we started running the same analysis for every single major version upgrade. So, now not only do we see individual optimizations, we can also get the net impact of these big version changes that
[36:47] come out when you move from Databricks 18 to Databricks 19. And that's not all. We are able to basically harness innovation from our data science team and scale it up in all
[37:01] sorts of different areas of the company, from query optimization to commercialization and finance. And that wouldn't be possible if we went with something off-the-shelf.
[37:14] Uh so, to wrap it up, building with Databricks is easy cuz all your data's already there, and it allows you to scale your ideas. Hopefully, we've convinced you that experimentation isn't just possible, but it's actually achievable for you given what you already have on Databricks.
[37:28] And more broadly, hopefully we've convinced you that building anything you like is potentially easier than you think since you're already on Databricks. So, our message to you is basically get
[37:40] building, build your ideas. You have all of the tools that you need to do lots of really cool stuff. And that's it for us. Uh feel free to find us after this talk if you like. Um
[37:52] thank you, everyone. We hope you enjoyed Databricks this year. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.