From Raw Data to Real-Time Retention: Powering Customer Health Scores on Databricks
Summary
- Customer Health Scoring at Scale: Coalesce and Databricks built a scalable, governed customer health scoring model transforming raw engagement signals into actionable insights.
- Low-Code Development Integration: Coalesce's low-code platform seamlessly integrates with Databricks' lakehouse architecture, automating data transformations with built-in governance.
- Automated Scoring Pipeline: The solution delivers trusted customer health scores supporting proactive decision-making with full data lineage and compliance transparency.
From Raw Data to Real-Time Retention: Powering Customer Health Scores on Databricks

Talk by Josh Hall, Sr Product Marketing Data Engineer, Coalesce (Sponsor Speaker)
Understanding customer engagement and retention isn’t optional—it’s mission-critical. Join us for a live demo to see how you can build a scalable, governed customer health scoring model by transforming raw signals into actionable insights. Discover how Coalesce’s low-code development platform works seamlessly with Databricks’ lakehouse architecture to unify and operationalize customer data at scale. With built-in governance, automation, and metadata intelligence, you’ll deliver trusted scores that support proactive decision-making across the business. Why Attend? Accelerate time-to-insight with automated, low-code transformations Build repeatable, enterprise-grade scoring models with full data lineage Ensure governance, transparency, and compliance at every step
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
Why is customer health scoring important for SaaS companies?
Customer health scoring enables SaaS companies to identify at-risk customers early, prioritize retention efforts, and forecast renewal likelihood. Accurate health scores inform sales, success, and product teams, enabling proactive interventions that improve retention and customer lifetime value.
How does Coalesce integrate with Databricks for this solution?
Coalesce's low-code development platform automates data transformations within Databricks' lakehouse, building the data model needed for health scoring. The integration combines Coalesce's ease of use with Databricks' governance, performance, and scalability.
What data sources feed the customer health scoring model?
The model ingests raw customer engagement signals—product usage metrics, feature adoption, support tickets, API calls, and other interactions stored in Firestore. Coalesce transforms this messy, unstructured data into structured, governed tables ready for analysis.
What benefits does built-in governance provide?
Built-in governance ensures data lineage is captured automatically, compliance standards are enforced, and stakeholders can trace how health scores are calculated. This transparency builds trust in scores and enables faster, more confident decision-making based on the data.
Full transcript
Super stoked that everyone's here. We're going to be talking about primarily how Coales, which is both a partner and a customer of data bricks, built out our customer health metrics pipeline to effectively evaluate how our own customers are doing from the product that we sell. But we built the entire pipeline itself on data bicks to effectively be able to take advantage of all the uh functionality and also some of the really cool features that coles has available on data bricks outside of just some of the raw data bricks functionality that's there to begin with.
So hopefully a really interesting perspective today to be able to understand a little bit more about how both a partner thinks about how we develop on data bricks but also how we actually use data bicks as a data bricks customer to fuel our own analytics internally. So a couple things to kind of frame why we decided to do this. So the first thing is cos has been around for roughly in the ballpark of five years now. So not the most established company.
Haven't been around for 50. But we have enough customers now, hundreds of customers now to we've gotten to the point where it's really difficult to actually manually go through try to understand exactly how our customers the health of those customers, how it actually looks. At the beginning, it was really easy as we had 10, 20, 30 customers for a salesperson, someone who's working on an account to be able to reach out to the product team and basically say, "Hey, what's going on with this customer?
How do they look? What's the chances of renewal?" Whatever the case might be. But as we've grown, as the team has scaled, as the product has scaled, it's become significantly more difficult to be able to understand what the health of our customers actually looks like. And so we decided and we recognized that in order to be able to continue to do that, we needed to develop a pipeline in a way that was going to provide us that data to be able to make those decisions a lot more quickly.
The data that we're working with is really messy. So the actual underlying database of where our product information, the actual data from our product gets stored is in currently is in fire store. And so it's in a really nonstructured format and to be able to work with that data in a structured way is not necessarily the easiest way to be able to have analytics happen for you. And especially for someone who's oneoff going to our product team or we don't really have a designated analytics team, we're going to get to that here in a second. to be able to go to them and say, "Hey, what's actually happening with this customer?" was relatively difficult to do.
And our team, like I mentioned, starting to expect a lot more turnaround to be able to understand, hey, what is actually going on real time with this customer so we can check in with them. We've spun up a professional services practice over the past year, and that's been really invaluable to them to be able to understand exactly what's going on with our customers in real time all of the time. and myself and our head of finance who are the primary contributors to this product that we're going to be looking at today.
This data product that we're going to be looking at today that we developed on data bricks when we first kind of set out to frame this problem really wanted to start with what we didn't want. We didn't really want to start with a product that was going to have to rely on a lot of manual coding, a lot of complex logic that was embedded deep in different products or scattered across multiple different products. We wanted something that was going to be available to our users that was really easy to access, but also because it was me, myself, and our head of finance, just two people working on this product.
We wanted everyone else who was going to have visibility into this product, this data product, this pipeline that we were developing to be able to understand what was actually happening. And so embedding complex logic, having that in different products really didn't make a lot of sense for us. We wanted this to be in a really holistic view of being able to understand what it is that we were actually developing. We didn't want this to be a manual process.
Like I mentioned before, it already was. We were reaching out or our salespeople or our professional service team were reaching out to our product team to understand exactly what was going on with these customers. So we didn't want that to be a manual process anymore. And so alleviating that to be able to make this a real time pipeline to be able to fuel those specific sales team members who needed that data was really important for us to be able to create that as well.
And then a lack of lineage and documentation is something that we didn't want to have. We wanted quite literally everyone to have visibility into this, which is why it was really important as well because it's just two of us developing it. We did not want this siloed in some big code mess that was not going to allow our team to understand exactly what was going on if they were to look into it. And so we decided that although at first we were trying to figure out the direction we wanted to go, that we were actually going to go ahead and dog food our own product.
And so again, like I mentioned at the top of the session, really getting a unique perspective today and understanding a little bit about how coales actually works as a product, but also how we thought about developing on data bricks. And so we really wanted to we decided that we really wanted to leverage a lot of the functionality that coles had to offer to alleviate those pain points that I just brought up a second ago.
Wanted really modular design, the ability for our team members to understand what was going on even if they weren't involved in the development of those pipelines. had really reusable components so that we could not have to duplicate our working efforts and or run additional data bricks compute for tables that already created. We really wanted to take advantage of the data bricks lakehouse and the way that coles works allows you for you to deploy your data pipelines in a really seamless manner that allows you to take advantage of all the data bricks functionality without necessarily going through some of the deployment headaches that you might in a traditional deployment.
And then finally, mentioned this in the last slide, but we wanted a lot of that built-in lineage and documentation, all that metadata that's available within coal, which we'll see in a brief demo here in a few minutes to be able to take advantage of so that when our team inevitably did go to look and see what we built, is the logic correct? Are those components and those metrics that we care about measuring our customer health, are they correct, that they would understand, they would have the full visibility and transparency into how that actually works.
So I want to give a quick word about what Coales is so that when we jump into the demo portion of today's session, we're actually going to have a little bit of an understanding of what's going on. And actually before we look at the major pipeline that we built to power our entire customer success pipeline, I I want to be able to give a brief demo so that everyone kind of tangibly understands what's happening. But before we do that, I want to give a little bit of an overview.
So Coales is a data development and discovery tool. What that means is effectively we allow you to really seamlessly build data pipelines. We allow you to do that really quickly and then we allow you to understand and discover what you've built not just your engineers but the business users who are involved in your business as well. So you build your data products and then you make those products available for your business users in a really discoverable way.
One of the main concepts that we use within coales is this concept of what we call a node n ode. And a node effectively is a userdefined template of an object that exists in data bricks. Think of a table or a view. But that means that we can templatize a lot more critical or more complex functionality like a type two slowly changing dimension for example or even a delta live table where we can templatize exactly how that should execute and then allows me or you as an engineer as a user to be able to go in and just apply your business logic and allow that template to execute leveraging that single time definition of what that template should actually be doing.
And we'll see that when we jump into the actual demo portion here of today's session. So customer health scoring in action. There are really a couple components here that we're going to see when we jump into the actual pipeline because I'm going to show you all the actual pipeline that we built out within coales that's sitting directly on top of data bricks. And so one of the big criteria here was to actually go out and score exactly what metrics we cared about for our customer health metrics.
And one of the big things that we need to do here is determine the waitings. And when it comes to business logic, waitings are sometimes really difficult to embed into the logic of what you're working with and allow those team members to be able to understand exactly what's happening. And we're going to get to a portion of the discoverability portion of the demo today that's going to allow you to really understand how some of those waitings and some of that logic comes through in a really easy to understand way.
Like I mentioned earlier, we really wanted to automate the pipeline runs in documentation. And so instead of having this be a manual process or a salesperson having to reach out and us have to activate that process ourselves, we wanted this to be a really automated process that allowed them to be able to self-s serve, go out, understand what's happening with a specific customer. If it's our professional services team, to be able to go out and understand this is a relatively new customer.
Here's the usage of what our product looks like for their account right now. Here's what we should be doing moving forward to be able to help them continue to grow utilizing the product. And then we really wanted to be able to again trace that logic, trace everything that we had built really easily using column level lineage, all the other discoverability components of coalles. And so we're going to be jumping into that here in a moment with a demo.
But like I mentioned before, before using coales as our customer health metrics pipeline, a lot of what we were doing is really manual, really broken up in terms of our data was landing in data bricks and we were spending a lot of manual time trying to understand exactly what was happening with our customers doing basically oneoff ad hoc analysis with our data. primarily fell upon myself or someone else within my team to be able to focus on writing those queries, cleaning that data, making it available to be actually answer the questions for our sales team.
After implementing coalles, we have a much more streamlined pipeline. Our cost went down on using data bricks. We have clear visibility and exactly what's happening. and our customers which are effectively our sales team and our professional services team are able to serve themselves in real time on the data that's stored in data bricks every single time that data is executing from our source systems. So with that I think it's most helpful to probably jump into a demo to kind of understand a little bit about how cos is actually working and then we're going to look at the exact pipeline that we actually created to power all of what we just talked about.
So the first thing I want to do is actually jump into coales. What we're looking at here is what we call the build interface which is where we actually build out our data pipelines. It's completely blank at the moment. And I wanted to do that intentionally. I wanted to start from a blank slate intentionally because I wanted to be able to showcase how quickly you can actually develop utilizing coales from basically raw data source your bronze level all the way through to a builtout data mart.
So the first thing we're going to do is actually go up to this little plus button in the upper lefthand corner and we're going to navigate to this add sources button. And what that's going to do is it's going to pull open any of the data sources that are available that I have exposed as a user to coales and I'm actually going to while we're here I'm going to go to my displays and let's see if I can zoom in a little bit.
We'll wait on that for a second. So, one of the things that I've done here is created what we call storage locations here in coalles at a bronze, silver, and gold level. And in this case, I have four objects that exist in my bronze level. They're just raw data sources that I've used an ingestion solution to be able to pull in and now available for my team or myself to be able to work with. I can select the little hierarchy checkbox to select all four at once, or I could select each one individually if I wanted to.
And once I've selected that, can go down to the bottom right hand corner and select add four sources. And that's going to bring in these four sources, these orange tables here, as what we call nodes, like I mentioned a little bit earlier, as the data sources or the raw layer of my pipeline for me to build out. I've also added in this little customer reviews table, which I'll actually show the data that's in here. It's just a single record that has this big JSON string that exists in this src column.
The reason I wanted to do that is because I mentioned earlier that a lot of the data that we were ingesting from our fire store database is in JSON format. It's in really messy format. It was taking us a long time to be able to parse through that and make that data actually usable. And one of our favorite things, myself and our head of finance who are working on this pipeline to take advantage of and using coless is the ability to basically shred that JSON, be able to parse through that JSON really really easily without having to write and manage all that code ourselves.
And so we're going to leverage that here in a moment. So I'm going to pop back out to the builder interface. So I have these five data sources. One is basically a single record that has a huge JSON string in it. And then the other four tabular data that exists in these orange nodes. And what we can do because we're leveraging these templates, what we call nodes within coales, is because it's defined one time, we can reuse them over and over again.
And so if I look at all my nodes here on the lefth hand side of the screen, I can look at these data sources here. I could select all four at once. And then I could right click on any of them, navigate to add nodes. And this is just a sampling of some of the node types that are available. And so you'll see that I'm kind of hovering over a dimension node here at the top at the beginning here. But we could use all kinds of nodes because they're templates.
You can define them yourself. So if you need to define one yourself or you need to go to what we call the coales marketplace to download additional functionality that's already been defined for you, you have the capability of continue to extend the amount of nodes and the available functionality to you here in coalles. And so what I'm going to do here is actually navigate down to kind of a stage node or a staging or working layer to kind of start building out my basic transformations here in coalles.
And that's going to add a table and node to each of those data sources. And because we're just working with a template, that means that we can reuse that same standard over and over again. And we're going to have the exact same experience regardless of who I am as a user. And so, for example, if I jump into one of these nodes, we're going to see an experience that's somewhat similar to kind of what you would see see within data bricks where I have all of my columns that are coming from my data source available to me on the left hand side of what we call the mapping grid here.
And then I've got some other metadata that's pulling through here such as the data types, the actual source of the data where it's coming from that raw data source. And then over here on the right hand side of the screen, I have what are called my configuration options. And the reason we can make these available is because again we're defining these options one time and then passing that through to the user via the interface to be able to work with in the same manner regardless of what user I am over and over again.
And so for example, if I wanted to change the materialization of this object from a table to a view, I could do that. And coales is going to programmatically handle that conversion behind the scenes, which we'll see here in just a moment. I could also add my testing in here as well. So if I want to write specific SQL testing for my data that's existing in this table, I could have that all here within the exact same node.
In the middle of the screen here, we have this transform column, which is currently blank. And next to every single column is basically a blank column within this transform column that allows me to basically leverage a SQL editor. So for example, if I came in here and I doubleclicked into the first name column here, I could write any data bricks supported SQL to start building out my data transformations. So for example, I could write an upper function.
And then I'm going to pass through this handy little function called the source function, which is just going to resolve my column to the actual name of the column I want to transform, which in this case is first name. And assuming I don't want to want to perform any other transformations, the last component of a node is at the bottom of the screen here where these two blue buttons exist. What we call the create button and the run button.
That create button is going to execute the DDL for your table in datab bricks. And underneath the hood, this is just leveraging a datab bricks SQL warehouse. We're not actually having we don't have any compute within coalles. We're not doing anything fancy in coalles. We are pushing all the SQL that's generated down to datab bricks and running it on a datab bricks SQL warehouse. And so when I click create, you'll see that it automatically created an object here.
And I can see the exact SQL that was created in this case. So it ran a create or replace table statement and then it included all of the columns with the data types associated with it, which is exactly what I would expect to see from the DDL of something that I were to create in data bricks on my own. And then when I click the run button, what that's going to do is actually execute the DML for the table. So, it's going to insert all the data into that table.
And we can actually see the output here where that first name column actually is in uppercase. And we're going to expand the execution steps. And we'll see that there's a truncate stage table that actually ran here. That was the first execution step. And the reason that ran is because on the right hand side of the screen in that configuration options pane, there's a toggle called truncate before that was toggled to on.
If we were to toggle that off, obviously programmatically behind the scenes, coales is not going to push that through to data bricks anymore. And then the second step that ran was an insert into statement where we can see the exact statement that was run. It ran an insert into into the actual table that was created in the previous execution. And then there's that upper function that was passed through to it. So it's applying all that SQL that I as a user wrote.
And really the main benefit here is that I as a user and especially as we built out this customer health metrics pipeline, we didn't have to focus on managing the infrastructure of actually creating the tables and managing some of the DML that was created here. We could focus on just the business logic of what we wanted to apply. I mentioned the weightings that we were working through, the metrics that we wanted to define.
That allowed us and freed us up to focus on that and again as a twoperson team to build out the pipeline that we're about to look at in about a month and a half. Last thing I want to talk about real quickly is that JSON stuff that we were going through a second ago. So we have this record in this src column that's in all JSON. And if I were to add another node to this JSON record that we have stored here, maybe a view in this case.
We could add a fact. We'll go ahead and add a dimension node in here. We can look through what a dimension actually does in a second. And what I can do here is without having to write any of that data brick SQL myself, I can rightclick on that src column. I can navigate to derive mappings and then say, hey, I want to derive all of the columns out of that JSON string that's stored in that column. And you'll see here at the bottom of the screen that I had five columns ripped out of that JSON string.
So the key pair values that existed in that case JSON string are now available to me as columns with their associated value at the bottom of the screen here. And if I needed to move them because I don't want them associated with the system columns here, I could just drop them up here. And now I have those columns available for me to be able to work with. And I did that in quite literally two seconds. And if I go to this join tab, this is where you're going to actually see where the dependency is generated for you as a user to build out the DAG that we're working with or the DAG that we saw in that browser tab a second ago.
And you'll also see that the reason we were able to parse that out is because coales automatically generated that flatten function for us to be able to work with that JSON data. And so that is how we're able to work with that data. And one last thing that I wanted to do here is we'll go ahead and create this dimension here. I need to over on the right hand side of the screen, I need to define a what's called a business key.
And this business key is basically going to be a unique identifier for each of the records that exist in my table. And I'll go ahead and get rid of each of those metadata columns that exist here. So we'll create this table and then if I want to configure as a say a type two slowly changing dimension, I could pass through any of these columns as a change tracking column. In this case, there's not a lot to track changes on.
We'll just pass through customer ID as an example here. And when I go to run it this time, it's basically going to work through this data as a type two slowly changing dimension. I got an error here. I'm going to pass through my menu table here. I was working with this table earlier today. And the lateral flatten on the outside of it was not working the way that it was supposed to, which was my fault. I passed through a table or a column that didn't work the way it was supposed to.
And so when we go to run this table, it's going to generate the type two slowly changing dimension code that me as a user would need to have to write on my own that I no longer have to worry about. So we ran a merge into statement and generated all that type two slowly changing dimension code for me. So now that we've gone through a little bit of a demo, you're actually going to be able to understand and see a bit more about what's actually happening when we look at this pipeline here in a second.
So I'm going to switch over to a different coales project. This one's going to be a lot bigger. We're leveraging primarily stage nodes and what we call fact nodes within coales for this customer health metrics pipeline that we've created. The reason we decided to do that is because a for the stage nodes we wanted to effectively prepare all of our data in a way that was going to be useful for the downstream metrics that we were generating.
And then b there really wasn't that much dimension data that we were working with. The only really dimensional data we were working with was our customer information. the c the account name and any of the other potential really minor customer information that was associated with it. And so for us, it didn't really make sense to have to set up designated kind of star schema style pipeline to be able to leverage some of that dimensional data which really wasn't there to begin with.
And so we're passing through basically the account name with any other identifiers that are helpful to it through through the nodes that actually make sense to have that data in it. So, I'm going to zoom out real quickly, and I know you're not going to be able to really make out any of the detail here, but you're going to see it's a much bigger project where we're effectively building out this initial kind of staging layer, which is kind of the green coloration on the lefth hand side of the screen.
And then towards the right hand side of the screen is going to primarily be this kind of orange layer where we're primarily focused on building out these facts, these metrics that are going to be helpful for us to be able to pass through to our BI solution, to our reporting layer, which is ultimately where we wanted to be able to expose this data. And the reason I wanted to go through that demo a second ago is because we're going to see a lot of that functionality that allowed us to build this pipeline out really quickly available here in this pipeline.
So for example, if I double click into this very first stage node called stage orgs workspace steps, this is actually a node that is basically processing every single step, each of those execution steps that we saw a second ago where it was truncating a table and then inserting the data into it and then allowing the user to view that data. All that is basically recorded in our fire store database as an execution step or a workspace step as we call them.
And so you'll actually see in here a lot of transformations that are happening opposed to what we were looking at a second ago where a lot of them are these flow document parsing through of JSON data that comes through in this huge blob from our ingestion tool. And the primary ingestion tool that we're using for our pipeline is an ingestion tool called Estuary, which is really, really helpful in allowing us to move a ton of data really cheaply because it allows you to send data from one source to multiple destinations without having to set up the same connector for each destination that you want to send that data to.
So if I was connecting to my Salesforce account, I needed to send data to data bricks, snowflake, and fabric for example, using certain ETL tools, I might have to set up three different connectors. But in this case, we can just set up the connection one time to fire store and then send that data to each of the locations that is relevant to what we're trying to solve for. So that JSON parsing saved us a ton of time because as you can see here, we had dozens and in some tables hundreds of columns that were embedded in a huge JSON string that estuary was sending to us.
And again, really beneficial for us because we're able to use a tool in Estuary that sends us data really, really cheaply. It replicates data really cheaply because it is able to store everything in a really compressed JSON string. But for a lot of people having to work through that is really painful. And so for us being able to leverage coales and the JSON capabilities that it has allowed us to really immediately be able to take advantage of that data.
If we look downstream a little bit of that node that we were just looking at, we have this ddupe process which is taking place. And there's no real transformations happening here. But again, we looked at that join tab a little bit earlier. And this is where you can really start customizing and writing SQL for the specific business use case that you have within coales. And so in this case, we're actually joining data together.
We have a left join in here and we're joining on this org ID column that's associated between both tables. And then we have this really cool qualify row number window function that's running and again coales regardless of the complexity of the business logic if you can effectively do it in data bricks you're going to be able to do it within coales and so one of the big advantages for us is not having to worry about having that code embedded away in a notebook or in a a delta live table within data bricks but all that is available within a DAG available to us in this workspace in this pipeline that we're developing and so if we look a little further Further downstream we can go into some of these fact nodes that exist down here.
If we zoom into some of them maybe we'll click into this fact node growth rate node here. We'll see there's some more expressive and complicated SQL happening here. So in this case we're performing a case one that has a window function involved in it. Not only that but we have multiple window functions involved in it. We have a lag here. We have a lag here. We've got the partitions coming into play here. This isn't necessarily logic that just comes off the street and gets incorporated into your data transformation pipelines.
This is something that we were very intentional about and that coales was handling really easily for us. And that allows us to understand exactly how that occurred. And so in this case, if I rightclick on this specific node, you can navigate to what we call column level lineage. And what that's going to do is allow us to see exactly how this column flows throughout our entire pipeline, both up and downstream. And so this highlighted table here, fact, node growth rate is the table that we're looking at currently.
And then we can see exactly how that column is passing through the rest of our pipeline. So one table upstream of it, or I guess rather downstream of it, it is renamed to metric value and then it's actually parsed out. And if we hover over, I went in way too far. Let me zoom back out here and go back into this node. It'll be faster this way to just go back into our call mobile lineage. Let this regenerate here. If I zoom over any of these T's next to any of the columns that are flowing through my pipeline, we're actually able to see all the transformations that are occurring.
So even if I don't look into that node, that table specifically, I don't have to open a code file. I don't have to open another notebook. I don't have to scroll through and find a cell. I'm able to just look exactly at the transformations as they're occurring in each of these tables. So here's a table called fact monthly account ARR and we're able to see exactly how each of these transformations is actually being created allowing for me as a business user or someone who came in here and wasn't involved in building out this pipeline to understand exactly what's going on here to provide a lot more context into the logic of what is occurring here.
So after this pipeline was created we exposed all the results to our BI solution. In this case, we're using a combination of BI tools, but the original BI tool that we connected was Sigma. Built out a Sigma dashboard that we were able to present to our end users. And this pipeline when we created it is version controlled. And so this exists in our GitHub account. Anytime that we updated it, it's going to exist in our GitHub or version control system, allowing us to see exactly what's been updated over time, how logic has changed over time.
And once we've done that, we're able to deploy that into data bricks into a production environment, just a specific catalog and a couple different schemas in our data bicks account. And then we're able to schedule that on a refresh that basically refreshes for us on a weekly basis. The only reason we wait for that long of a cadence is because we don't really necessarily care over a daily or an hourly basis a cadence for our customer data about what's actually happening because most of the time we want to understand either week over week or month over month how our customers are actually utilizing our product to help indicate exactly where they're struggling or an area that might be helpful for them to continue to use certain functionality within coalles.
And if we jump into data bricks here real quickly, we're actually able going to be able to see each of the schemas that we're actually leveraging here. And I'm happy to jump into this just because there's nothing proprietary in here that's going to be shown. And so in this case, we have this coales app prod. And this is where estuary is dumping all of our production app data into. So you can actually see all those raw tables here.
But I think what's more important is our production target where we actually want to put all the fact tables that our BI tool can actually connect to do really exist here. And I think that's a misconception some people see when they're looking at coalles and a pipeline that gets built is okay this exists in coalles and I'm not actually able to get the value out of data bricks that I'm looking for. any of these tables that you see in here within this production target schema really were created within coales and we're pushing that code directly down into data bricks and those objects exist here.
So all that functionality that we saw and the ease of use to be able to build out that pipeline really does exist in data bricks and we're just using the data bricks SQL warehouse engine underneath the hood to take advantage of all that functionality. The last thing I want to show here is the second part of the coalles product, which is the coalles catalog. And this is where all of our sales team, our professional services team, and anyone else who's interested in our customer health metrics can come in here and basically self-s serve themselves to understand what it is that we built.
And so in this case, we basically have Google search-like functionality to be able to look through exactly what assets exist. And I'm not going to look for anything in health metrics right now. But at here at the top, if I type in sales, basically what's going to happen is the catalog is going to scan every single asset that I have documented within my catalog. And catalog automatically integrates into any of the solutions that you connect it to.
And it's going to try to find things that are relevant to the keyword that I typed in here. In this case, typed in sales. It looks like we have this tasty franchise sales dashboard here at the very top. And we're going to get some context immediately into what this asset is. So I don't have to guess about what this dashboard's doing, what the data in it contains, how users are using it. I know who actually owns this specific asset.
So if there's a specific person in my organization that I don't know to ask questions to. Now I know who actually owns this dashboard. So I can help they can help me with anything that I might have questions about from this uh dashboard perspective. And there's a lineage tab here which is going to provide a lot more context especially visually into what's happening with this specific dashboard. And in this case we're using some snowflake data here.
It looks like where I have my tasty franchise sales dashboard that exists in Sigma. And we're able to trace that all the way back through all the way to the source of wherever that came from. And if that table originated from say a fiverr, we'd have that integrated as well to allow me to see from fiverran all the way through to sigma in this case exactly how data is flowing through and transformed through my pipeline.
And it's possible that I can click this plus button next to v sales that there's additional dependencies here. So it looks like we have some powerbi dashboards that are also dependent upon that v sales table even though I was looking originally at that sigma dashboard here. So that allows me to have a lot more context into actually what's happening within my data pipeline. But maybe I don't know what I don't know. And I'll come over here to the AI assistant, which is basically going to allow me to take all the context of all the objects that exist in my entire organization and help me bring the most relevant results and surface those results to me by asking a plain text question.
And so in this case, I might be interested in what data we have about node growth rate. And I could quite literally ask that question. So I can say what data do we have about node growth rate and what's happening underneath the hood here is the coales catalog AI is going through like I said all those objects both the tables that exist and also any dashboards and any other assets that I've connected. This could even be like a notion page where we've documented something about our node growth rate that allows me to have a lot of context into exactly what it is that we've built to provide all that relevant information to me when I don't know exactly what I'm looking for.
And in this case, it looks like right at the top here, I have this data bricks table called fact node growth rate, which I can click into and explore a bit further to understand what's actually contained here. I can see the columns that are contained. I can see that there's some documentation here, but it's not fully complete. We don't have an owner yet, but it looks like we have a technical owner. If I go over to the lineage, we can explore that a little bit further just like we did before where we can see both up and downstream exactly what's happening in this pipeline.
And we saw this in coales, but if we expanded enough, we could see the ingestion from estuary and the dashboards that are downstream in sigma and omni, which is the other BI tool that we're using. But more importantly, I mentioned that there is a really cool component that allow us to understand a bit more about the logic of what we're capturing here. Right now, this readme, which is basically providing the context about this table, is empty.
And if I go ahead and click in here to start documenting this table, I have this AI capability here at the top of the screen that I can click on and I can say, hey, I want to select a template that's going to basically provide me with bullet points that are the same for all my tables. So maybe I want to know what context is, who the users are, how many queries typically hit it, that type of information. And for each table, it's going to give me the actual relevant information underneath each of those headings.
In this case, I'm going to select that template. I'm going to say, hey, let's go ahead and generate that description for this table. And what it's going to do underneath the hood is actually scan all the metadata that's associated with that table. It's going to include the transformations that we generated. It's basically going to build me a read me that provides a lot of relevant documentation and information about exactly what's happening in this specific object.
And so you'll see that one of the headings that I specifically asked it to give me in this template, this documentation template is this good to know heading. And this is really powerful and beneficial not only for myself who actually help build this pipeline, but for any of my users who come in here. And it's saying that, hey, the node growth rate, which is what I was looking for, what data do I have about node growth rate, is calculated using a formula that compares the current month's node count with the previous month's node count.
And then it gets into the actual exact logic here in plain language, plain English here without me having to read through a SQL query, read through the column of a lineage. I have all that information here available to me within my catalog. And not only that, but all the other context here that was created throughout this AI generated readme as well. So I have all that information to be able to pass along to my business users for them to be able to come in and understand what it is that we built.
And so with that we basically reached the end of how at a very high level we went from basically building really manual reporting processes to a real time pipeline that was built in coalles but built on data bricks that powers our entire customer health metrics reporting process. I'm happy to dig into some of the logic if people have questions about that. But with that, I think we're wrapped up here and I'm happy to looks like we have about five minutes left.
I'm happy to field any questions if anyone has anything. I've got an assistant in the back who's a microphone. Otherwise, happy to give people five minutes back to their afternoon. And I appreciate everyone coming out here today.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.