Evolution of the Databricks Commercial Motion: Building Scalable Intelligence on Databricks

Summary

Databricks' commercial data science team shares a four-year journey from gut-instinct decision-making to a data-driven commercial engine serving over $5B in revenue, structured around three phases: centralize, understand causation, and act at scale.
The first phase focused on unifying fragmented commercial data sources in Unity Catalog to establish a single source of truth, eliminating time wasted in meetings arguing about whose numbers are correct, and saving approximately 300 hours per year through standardized reporting.
Later phases applied causal modeling to distinguish correlation from causation in commercial analytics, then productionized a contract scoring system built on MLflow and integrated with Salesforce to help sellers structure deals at scale.

Evolution of the Databricks Commercial Motion: Building Scalable Intelligence on Databricks

A truly "data-driven" enterprise requires close collaboration between Business and Data Science. In this video, we share our journey using Databricks to bridge the gap between technical complexity and operational reality.
We detail our journey: We start by centralizing fragmented sources via Unity Catalog, surfacing descriptive trends via AiBi dashboards. We then move from descriptive to prescriptive insights, using Lakeflow orchestration to automate generation and understand causal effects. Then, we evolve from static insights to dynamic optimization, exemplified by a contract scoring system that helps sellers structure deals, built on MLflow and integrated with Salesforce. Finally, we explores how Databricks' AI products amplify the values of the previous stages.
Attendees will learn to unify Databricks’ core capabilities, transitioning from high-level vision to a functional reality where AI and human strategy are woven together.
Talk By: Jason Huang, Data Scientist, Databricks ; Stephen Moss, VP, Bus. Insights and Commercialization, Databricks

Chapters

FAQs

What is the Databricks commercial motion case study about?

This video documents how Databricks' internal data science team, over four years, transformed the company's commercial decision-making from gut instinct to a rigorous data-driven system at over $5B in revenue. The journey covers centralizing fragmented commercial data, applying causal modeling to business analytics, and deploying productionized ML models integrated with Salesforce.

How did Databricks establish a single source of truth for commercial data?

The first phase involved centralizing fragmented commercial data sources into Unity Catalog on the Databricks Data and AI platform, creating standardized reporting that saved approximately 300 hours per year. This eliminated the common problem of teams arguing over conflicting numbers in meetings and allowed the organization to focus analytical effort on solving problems rather than reconciling data.

What is causal modeling and why did Databricks apply it to commercial analytics?

Causal modeling goes beyond correlation to identify what actually drives a business outcome, using techniques like parallel-world counterfactuals to isolate the effect of a specific intervention. Databricks applied it because stakeholders were incorrectly attributing business outcomes to correlated events rather than their true causes, leading to misguided commercial decisions.

What is the contract scoring system described in this video?

The contract scoring system is a productionized ML model built on MLflow and integrated with Salesforce that helps sellers structure deals by scoring contract configurations based on patterns from peer deals. It evolved from scattered notebooks into a scalable system that runs population scoring across open opportunities to surface the highest-potential deals for structured intervention.

Full transcript

[00:07] Um so anyway, we're going to talk about the evolution of the commercial engine. Um Jason and I joined data bricks about four years ago and we have a confession to make. When we joined, even as a data and AI company, we were making a lot of
[00:21] decisions based on gut instinct rather than data itself. So over the last four years, we've taken upon ourselves and our colleagues to to change this, right? And to make many more of our decisions commercially based on data, based on
[00:35] rigorous analytics. And that's what we're going to talk to you about today. Um because at our scale, if you're making a decision based on gut instinct, which you may say there's plenty of decisions in life that are made out of
[00:47] gut instinct, what's wrong with that? But at our scale, these decisions have real consequences. you know, over $5 billion of revenue, double-digit growth, it's massive. You know, tens of thousands of customers and deals.
[00:59] Anything you do, any decision you make could have a very positive outcome or it could have a very negative outcome and really put a dent to these numbers. So, we want to make sure that the decisions are as good as they as they can be and
[01:11] that requires data. So, what we did was we went on a journey. Um, so we're going to talk to you about the journey today. It really had three phases. I won't pretend that at the very beginning of the journey we knew exactly where this
[01:23] was going to go and exactly how each phase was going to play out, but we had a general idea and we always kind of knew what was the next thing to do. And the first phase of the journey was to really centralize and standardize and
[01:35] get people singing off of one song sheet. What I mean by that is looking at the same data to make decisions and not arguing about the data. How many people have been in a meeting? You're there to hopefully solve a problem or advance a
[01:48] problem and you spend more than 50% of the time arguing about whose numbers are right. Yeah. Okay. Pretty common challenge. So that was our first thing. It was a how do we get at least in the commercial realm to one source of truth.
[02:00] So that's chapter one. Chapter two is we realized that people were making the mistake of attributing causes to correlation. So they were
[02:12] noticing that an outcome and something else were happening kind of at the same time. So they assumed oh this is one thing is driving the other. But as you know that's not necessarily true. It's the difference between correlation and causation. And so we wanted to move our
[02:26] analytics, our modeling, everything that we were doing towards really a causal uh type of framework so we could understand what drove outcomes. And the reason why we wanted to do that is actually chapter three, which is if
[02:40] you understand what is driving outcomes in the moment, you can take decisions that help you to act in a different way going forward and improve your forecast, improve your outcome, improve the probability of success.
[02:53] So these are the three chapters that we went on on a journey. And so for chapter one, I'm going to call Jason up and ask him to take us through it. Jason, >> thank you, Stephen. So I'm excited
[03:05] everyone to kind of take you on this uh story and as with me uh with many stories we'll start with chapter one centralize and standardize. So we begin with a CEO
[03:20] asking how much revenue do we book in deals uh last quarter and three teams finance strategy and sales comes back with three different numbers. There
[03:33] should be only one answer. So, and also for the data scientists in the room, maybe you can sympathize. Data science often then gets pulled in to help reconcile. And we spent all this time
[03:45] doing data investigation. And we find that everybody's actually using the same source. The difference is how they uh use the data. Fi uh finance uses close date while uh sales was using start and
[04:00] strategy was annualizing by dividing the total uh deal value by the term length. So
[04:12] on top of it to make everything useful. So our solution was to build all that logic into one govern table called deals 360 managed by UC and this gives us
[04:24] three things. So first the obvious deal anatomy just a unified definition for the all the key uh fields two a certified source of truth with uh full
[04:36] lineage which shows how people are using it how often it's using it and all the downstream dependencies which notebooks which tables etc. And then three, it is the foundation for everything that I want to talk to you about next. And the
[04:50] first thing is self-service. Uh this whole DIS has really been about uh Genie. So I won't go into the details but speaking from our actual experience
[05:02] uh with Genie the data uh after putting the data in discover in place we can use Genie to just talk to it ask questions like are the deals getting longer get an answer with the source sited and ask
[05:16] follow-up questions have this whole conversation without writing a single line of query and this used to take a lot of data science effort and now it's it's self-s serve um the productivity
[05:29] gain has been real and we you know show this in with these numbers while I think I felt as a data scientist I think it's a little secondary in fact Stephen
[05:41] probably has really lived this firsthand so I also want to see if you know Stephen you have anything just to add to these numbers >> yeah well I think as recently as you know six nine months ago I always felt bad if I had a somewhat complex data
[05:54] question I'd like ah I gotta bother Jason or one of his colleagues to help me with this because my SQL skills are not great. I mess up the syntax all the time. It takes me a little while. Uh but now that I can self-s serve, I can stop
[06:07] bothering my colleagues like Jason. Um I and these are actually some of my own productivity numbers which are similar for other you know business users just asking a lot more questions like double the number of questions you know getting
[06:19] a lot faster in terms of like getting those answers because normally if I bother Jason he's like ah I'm working on something else like let me put that down let me understand what you're asking and it's it's a waste of time for both of us
[06:32] so it's almost like 300 hours a year for both of us um in in terms of like uh time that we can use for something else. So the productivity gain for a business user like myself is is just massive.
[06:44] >> Um yeah, thanks. Even though I still enjoy all that conversation, but all that time save does open up um our resource for doing even harder, more complex and higher value work which I
[06:56] want to talk about in the next two uh chapters. So chapter two, measure and prove. Every review cycle and this is probably true with many companies. Each team will
[07:09] claim its effort drove some dollar amount and each team will have some number and that number is probably positive. But if you do proper accounting, actually add up all those
[07:21] numbers, that sum oftent times exceeds the total realized growth of the entire company. uh which is just mathematically impossible. So either some or everybody
[07:33] is kind of inflating. And if we don't have a reliable way to know what has worked and what didn't, how do we make reliable decisions going forward? Just to further illustrate this problem
[07:45] and then you know talk about how we solve it. Let's just take this hypothetical curve of revenue over time and you have a team that launches some feature uh and we mark that time with
[07:59] the black dot and after some time say where that like red dash line is they look at the company revenue and they say hey look at all that growth and revenue that has happened since our launch
[08:12] that's because of us. Well, that's not the full story because that's just correlation, not causation. And maybe a lot of the people in the room, they're like, "Yeah, of course." Um, what's really the question is what the revenue
[08:25] growth would have been had we not launched this feature, but holding everything else constant. And that's really hard to do because that curve is inherently unobservable. We we never get
[08:37] to see the parallel world where we held everything else except for that change. At the same time that is also where data science comes in and add the value. So we will try to model that curve and just
[08:50] to see how this really impacts uh business decisions. Let's I want to talk about an example that happened at data bricks and probably happens in many other places. You have a product team that launches a new product growth and
[09:02] they celebrate all that revenue coming in in this like nicel looking dashboard and you know because of this growth um we're redoing the road map prioritizing
[09:15] features to drive this even further. But what this narrative misses is that this product revenue might be
[09:28] cannibalizing the revenue from the legacy product uh which is shown in purple. See how the purple line was growing nicely right up to the point when the new product launches and
[09:40] then it just starts to decelerate right away. And when we overlay this model for the total revenue of the legacy product, had we not launched this new one, it cuts
[09:56] the new product revenue clearly into two parts. The part that is below the dash line, that's a light green. That's all the revenue of the new product that is just really moving from one pocket to another. That that's the the old product
[10:09] revenue. And it's only the the dark green part which is above the line that is truly net new business that the new product unlocked. And the the dark green
[10:21] is also a a fraction of the total product revenue. And it's this gap that um many team misses this you know what the difference between what seems to be the obvious answer and the more subtle
[10:34] but the true answer that causes this uh overinflation that we were talking about earlier. >>
[10:48] rather than a a series of scatter notebooks which was a lot of the case in the past this could be easily made into a production pipeline that also refreshes automatically so that as new information comes in we can update our
[11:02] decision how aggressively do we invest in this new product depending on the extent of the cannibalization that we see. Okay, also this chapter it's been about measurements and it has uh really
[11:15] influenced a lot of the key decisions but humans are still you know critically in the loop. So in chapter three I want to talk about an example where we really start to take the humans out in uh in
[11:27] the case of applying to deals and actually I I should talk about you know the deal. So data bicks is a B2B company and we will use the term deal as the this um the financial agreement this
[11:40] construct that we tend to have with our customers that really govern just how we price what kind of services they're getting etc.
[11:53] Okay, deals are complex and every customer is unique which is still why we rely on the human judgment of our sellers to tailor our deals accordingly. But data bicks as a company still needs a holistic guideline for how we
[12:07] structure these deals. And you know be really before uh uh Stephen and I joined the the company would rely on a rubric like the one shown to the left where it
[12:19] only used two dimensions the size and the term length to say what a a reasonable discount is. Um but
[12:33] we start to enumerate every possible combination the complexity just explodes. So how do we have a datadriven way to say as a new deal is coming in
[12:45] what a reasonable discount behavior looks like? And we do this by comparing a new deal to a set of historical deals that are quote quote similar which we
[12:58] call peer deals. So a little bit more about how that mean what what does it mean for a deal something so complex and highdimensional to be similar? Well, we will never have
[13:10] two identical deals. Um but we can find a set of peers that sort of surround uh a given target. So let's say you know we have this uh fictional deal with Wayne
[13:23] Enterprise. It's in finance about a thousand in annual recurring revenue two years etc. Uh we have three peers and all of them are pretty similar while
[13:35] they're you know just a little bit off in one or two dimensions yet they all surround the target in the key dimensions and more importantly what are the deals that we kind of leave out the ones that are orders or orders of
[13:48] magnitude different or in completely different regions and industries. So how do we have a model to choose one which dimensions matter and then what sort of variation or what what is close enough
[14:02] uh me and we do this with something very classical just um and many of you may have known but we use a tree uh so tree is a very popular classical machine
[14:14] learning model uh that you could think of as the game 20 questions. we ask a series of yes and no to split the deal so that in every uh leaf the discounts
[14:28] are as similar as possible. So the model learns which questions to ask and in what order so that we're splitting the deals um as much as we can and to see
[14:40] how that works in practice on the la on the right I'm showing you using the actual model that we built you take a specific deal and then you compare the characteristics of its peers against the
[14:53] general population and what you see is okay it's not a perfect fit but the peers are really um it it really surrounds the target much
[15:05] more than the general population. And with this now we have a model to build like a personalized pure set and we can grade them against this curve.
[15:17] So what do we do with these scores right um first is just automation. Um, the ones that are getting the A's, just pre-approve. The ones that are getting Fs pre uh autoreject and we only need to
[15:31] uh have human judgment for the grades that are in the middle, which really speeds up the approval process. Two is recommend. Rather than just saying, hey, this is a B, we could also say for you
[15:43] to make it an A, do these concrete things. Um and three is build. Rather than starting from a blank slate, we could also you know give these AI agents
[15:56] the guidelines uh using this model to then prepopulate based on all the context we already have between the our sellers and and the accounts.
[16:12] So you know those are just some examples but we have applied these uh this this engine basically this these three three chapters along many verticals of our key decisions um things like how we price
[16:24] our new product rebalancing the inservice uh investment amounts go to market headcount planning sellers to customer matching and also even our field compensation structure uh and and
[16:37] all these have had you know three comma business impact. So with that said I want to pass it back to Stephen who will end with talking about the collaboration between the the two of us. Yeah.
[16:50] >> Thank you Jason. >> So Jason took us on the three chapters of the journey but there's a fourth ingredient uh that is important to understand what like why has this
[17:02] worked? Why has it been such a productive collaboration? And so it has to do with working across boundaries. Part of the reason I asked the question at the beginning about which part of the organization or what role you have
[17:28] embed the data science team in the business. Um so we consistently work with Jason and his colleagues on a defined set of topics. Um and we include them in our meetings. We do joint
[17:42] problem solvings together and it allows you know both both teams to ask each other questions. Like sometimes the business will say oh we have this question we want to ask and the data scientists will say wait a minute why are you asking the question that way?
[17:55] Shouldn't it be this that you're asking? And that that helps the teams to refine uh perspectives and and start working on the right question to begin to and it ultimately for us culminates we have you know big commercial decisions. We take
[18:07] them all the way to Ali and in the meeting is the data scientist who did the work. So if we're relying on a simulation or something to make that decision you know Ali will go and say
[18:19] like I don't want to hear Stephen from you about the model like Jason tell me and he'll ask specific questions. Um and so that's very empowering obviously for for a data science team but it also makes as a business leader it makes the
[18:33] the recommendations like quite credible because you can see the rigor that went into the analysis behind them. So number one is embedding. Um number two is understanding that translation goes both ways. You know sometimes there's an
[18:46] expectation for example that you know data science needs to explain the model to the business. Well, equally the business needs to explain all the context to the data science team. Uh and
[18:58] there might be context that's not understood like well this is actually a strategic objective where for a change we're not trying to maximize financial metric XYZ or we're not trying to
[19:11] maximize adoption. we have some other reason or maybe there's some sort of constraint that's not obvious like maybe it has to do with seller compensation that can't be changed until the next year or something like that. And so just
[19:23] being able to translate both directions um really helps each team to learn the other's vocabulary and and to start working on an initiative or a project in the right way. And then the final one,
[19:36] this one's a little bit hard, but we like to take on the challenge of continuously raising the commercial IQ of our business. And sometimes that means getting uncomfortable and showing,
[19:49] you know, seale leaders charts that might take them a moment to understand, you know, the output of a model, a histogram with different things using terms like counterfactual and synthetic control and, you know, those kinds of
[20:02] terms. And it, you know, it takes a minute and you might have let's like kind of, you know, simplify it or dumb it down. Uh, but let let's do that. And we say no, like don't be afraid to show the model. Don't be afraid to use the
[20:14] real terms because over time the the the senior leaders in the company become accustomed to that. They start asking for it and the overall commercial IQ of the of the business increases. So those
[20:26] are a few lessons learned uh from our collaboration model. So kind of bringing it all back together really four things that we talked about today. The journey consisted of the three parts of you know centralizing standardizing on a data set
[20:39] to stop arguing about it. The second is to realize that correlation and causation are not the same thing and it's super important to do the hard work to get to causation. Understand what is driving outcomes. uh use that
[20:53] information then at scale to automate and be able to take actions based on insights and then finally you know that joint team model even if the teams are in different parts of the organization
[21:06] really working across boundaries super important to make this happen. So, as you've heard the last couple of days, the platform and all its capabilities make this possible, make the analytics happen, but it's really the way that we
[21:18] work together as as teams uh that makes it possible. So, with that, I'll say thank you. And we have, I think, a little bit of time for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.