Measuring User Adoption and KPIs for Data Products Using Databricks

Summary

• System tables for adoption metrics: This video explains how to leverage Databricks system tables to measure user adoption and track KPIs for data products on the Databricks Data and AI platform.
• Product performance measurement: Learn how to analyze user behavior, assess engagement with data products and identify usage trends that inform product development and optimization on the Databricks Data and AI platform.
• ROI optimization: Discover how measuring KPIs like user retention, frequency of use and query performance helps optimize data products for better performance and return on investment.

Measuring User Adoption and KPIs for Data Products Using Databricks

Talk by Grant Stubblefield, Director of Product, Kythera Labs
In this video, attendees will learn how to leverage Databricks' system tables to measure user adoption and track key performance indicators (KPIs) for data products. The session will focus on how organizations can use system tables to analyze user behavior, assess engagement with data products and identify usage trends that can inform product development. By measuring KPIs such as user retention, frequency of use and data queries, organizations can optimize their data products for better performance and ROI.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How do I measure user adoption of data products?

This video demonstrates using Databricks system tables to analyze user behavior and track key adoption metrics like frequency of use and user retention. These insights help you understand how effectively your data products are being used on the Databricks Data and AI platform.

What KPIs should I track for data products?

Learn important KPIs including user retention, frequency of queries, average query complexity and data refresh times on the Databricks Data and AI platform. The video shows how to calculate and interpret these metrics for product optimization.

How do I identify usage trends and patterns?

This video shows how to use Databricks system tables to discover which data products are most valued, which features drive engagement and where users encounter bottlenecks on the Databricks Data and AI platform.

How can this data improve my data product strategy?

Learn how measuring adoption KPIs on the Databricks Data and AI platform informs product development priorities, helps optimize the user experience and demonstrates ROI to stakeholders.

Full transcript

All right, good morning everybody. All right, first session, first day. Uh, you all made it out. Um, so thanks for being here. Um, just real quick, uh, my name is Grant Stubblefield. A few like little uh things I'm going to go over today are definitely forward-looking. Um so these are probably going to change over the next bit. I've changed this presentation two to three times in the last three weeks because of new things that have come out in data bicks.
So uh just be aware it's current state of what's going on in data bicks. Um my name is Grant Stfield. I'm the director of product at Kyther Labs. Kyther Labs is uh a healthcare data technology company. We're based out of Nashville, Tennessee. Um, I have personally lived there since 2000. Um, so it's been home for a really long time. Um, my background is primarily in healthcare. Um, I have worked in it for about eight years now.
Um, for a while I was working with healthcare payers doing uh, claims automation, a bunch of stuff with member data. If you don't know much about healthcare, then I won't go too hard into that. You're you're you're healthcare. Cool. Um, so I got my start uh doing RPA, which is uh robotic process automation. And in that role, learned a lot about how data really drives what we do from a business process standpoint. Um, and then that got me into more of a data analytic side with healthcare data.
Um, and about three years ago, I moved over to, um, which is completely built on data bicks, and I got really into data bricks SQL, which is to me the greatest thing ever. It's, um, the like best syntax I've used for it. I can pretty much I haven't had to learn Python because of what I can do with SQL and data bricks. Um, personally, like I said, I live in Nashville. Um, music is a really big thing in my life. So, I still play out a bit.
Um, if you feel like talking to me about music afterwards, cool. If you want to talk to me about data, feel free to come up. Uh, so today we're talking about what data products are. Um, and do we all mean the same thing when we even say data products? I have read so many articles that don't really define it or maybe define it differently or it's different from every person you talk to. Um, so it's a really common term that we use, but it can be different for each one of our businesses.
Um, so some characteristics of data products. Um, these are not a comprehensive list. You may not apply all of them to your own uh data, but this is really the core at how I look at data um, from a product standpoint. So, usually it's very purposeful um, data. Uh, it solves a real problem for a real user. usually a business issue. Um, they're discoverable. So, it's data that you really are sharing with other teams. It's not something just for you or just for internal folks.
It's something that you'd either expose externally or to downstream teams. Um, it's addressable. So, by that I mean it addresses a business problem. It is uniquely uh there to solve um some kind of problem. Um, and it does so either through, you know, a schema, it can be an RS API, an endpoint. Um, really it's kind of up to you for how it solve that problem. Um, they are trustworthy. What do I mean by that? Um, it means when somebody goes and looks at that table, they believe it can actually solve their answer.
You don't get a bunch of questions about, hey, when was this uh table last created? What does it do? Is it the most upto-date? Is it the latest and greatest? The number of times I have been asked if a table is the latest and greatest um where it's not uh can sometimes blow my mind. That usually means it's not a data product though. Um and people kind of would want it to be. So um that actually drives a lot of what we develop as uh data products internally.
Um it's self-contained. Um, and by that I mean you're not having to do a ton of work to get the answers out of it. It could be a set of like four or five tables together that you have to join or it could be pre-joined. That's something that we do at is a lot of the denormalization in tables whether that be um from defined views in data bricks um or just by actually pre-joining and writing the tables. um that usually gets rid of a lot of the complexity uh that people need to do to understand how to work with data in healthcare specifically.
Um claims data is a very very star schema structure. Um and if you look at the actual backend systems at least from like uh insurance company side, it can be up to 500 tables that can get used for you to solve stuff. Um, in the strategy and business decision world though, um, from using like healthcare claims data, you can really narrow it down to about six tables and you can pre-join all of those and pretty much have minimal duplication just by doing some pivoting of columns and, uh, that's about it.
Um, so interoperable, um, it aderes usually to a common data model. Um we have an internal one at Ktherra that we use but it also is interoperable with other known um common data models in the healthcare space. Uh those being OMP fire uh HL7 fire is built on HL7. Um but uh OMP has been a really big thing that we've used recently. It's actually an open source framework um that get used primarily for electronic health record data.
Um and it because it's open source it gets heavily used by the entire market. Um so version and audible as well that usually means us maintaining pipelines over time. If the pipeline changes it indicates either a new version of the pipeline may not be a new version of the table because we do incremental appends to it. So there's a difference on that front of adding data versus are you really creating a new version of a table.
Um, and there's usually a team or owner responsible for it. Whether that's um, engineering or your analytics team is usually going to be more, at least for us, on the dashboard side, BI side of things. Um, where our engineers really manage more of the pipelines. So, data products are at its core package, reliable, and usable. Um, if it's not one of those three things, uh, if you don't want to give it to, uh, downstream teams or externally, it's probably not a data product.
So, um, some examples of what data products can be or data as a product kind of the terms can use interchangeably. Data as a product is more of a thought process. Um, but examples can be dashboards. Um, you can have chat bots that are not just, hey, interact with my data and answer anything. It's kind of purposely built to answer some questions. You see that with data bricks genie rooms. They're really kind of configured for a specific question that you're trying to answer.
Um, or they can be curated tables, which usually deals with some level of remastering, some level of cleansing, uh, kind of doing the medallion in framework. Um, just data is going to be like a raw data dump. I don't consider that a data product. It may inform building one. um an ad hoc analysis that you do just one time, send to somebody. It's not going to be something you manage as a product. It can be built off the products.
Um single-use reports um and just non-version controlled systems. So I mean I make dashboards for exploratory analysis all the time. It's not something that I maintain long term. I usually try and delete them after the fact because if somebody comes and looks at it, it's probably not up to date. So, one thing I do want to go over is the difference of measuring data products versus software products. Most people um or if you've been in product, you really think of things from a software as a service standpoint.
And the way that you measure KPIs um for SAS versus data is a little bit different. Um, a good way I like to think about it, I don't know if anybody's worked in the restaurant industry before, but I kind of consider uh software as the front of house staff. So, it's really your like servers that are interacting with tables and such, right? Like they are directly engaging with the users. they're getting feedback about how the food is and um you know is this some item that sells a lot versus where your back of house is really focused on do we need to order all of this get it in um how many do we need to create this evening or whatever you have your workflow of um you know the kitchen staff is very different from your weight staff so it's really kind of a very different uh model you can also think of it as kind of a grocery store even um analysis This is going to be like your restaurant even is like a software product.
The grocery store is your data, right? It's the things that make up uh the software. Um so you some of the same KPIs apply um for one versus another. Um but like with data, I don't really care about measuring session duration in data bricks for running a query for you doing your analysis. I care more about how many queries are you running about? How many tables, what columns are getting used the most, which ones aren't getting used.
Um, I've had actually brought this up with somebody uh as I was prepping for this um who has a disdain for product uh surveys and teams when some, you know, and a software or some uh survey goes out where it's like, how can we improve the product? And she's like, I hate these. Why can't they just leave me alone? they made features that I like. Like, please just stop asking to give me new stuff. Um, and I told her from my perspective as product, what I want to know is which features can I get rid of?
Which can I drop? Which ones are not getting used? What am I supporting that I shouldn't be? Um, and that's actually driven a lot of changes that I've done to um our product tables at. Noticed a ton of columns that were almost null that nobody was using that I also had no idea how to use. Um, so get rid of those or there were naming just even naming of columns that did not make sense. Um, for example, on a healthcare claim, you can have like I think it's seven different date columns that can all be used for different use cases.
Um, and we had a bunch of our users um, using and one of these is on a prescription claim. They were trying to find the date the prescription was written versus the day they were actually using the day that it was filled instead which can be a much longer time if you have like three prescriptions that you're going to the doctor for. That can be three months of time difference and that was purely from a naming issue that caused with that.
So kind of discovered that just from seeing how users were interacting with the data. So, a few examples that we have on the healthcare side that we have at Ctherra. Um, patient risk scoring. Um, a lot of that is about how sick somebody is, what do they do. You can kind of read through these. The one I'll highlight is going to be our facility taring work. Um, provider and provider directories. What that really is is measuring like where are uh providers, where are they working, how long they work somewhere, um what facilities do they operate at, etc.
What we realized with um a lot of the analysis our customers were doing um is just trying to get counts of how many surgeries does physician X do at this place. Um it was something that was really commonly getting used um and a few other volume just based queries. Um our main claims table is over 20 billion records um and you can do a bunch of joins with that to provider directories. So these could be some pretty costly queries or even they could be written in ways that could be very costly.
So something that we added to our directory base was just a set of five tables for these that did pre-calculations that runs um with every build that we do of the product. So that basically our users can now just go to an ancillary table and get the counts of numbers. So, just a uh way of doing denormalization that doesn't have to have our users come up with the best way to do counts of things themselves. Um they're kind of just relying on how we say the best way to do it is uh which my opinion is best because we're the ones most familiar with the products.
All right. So, KPI KPIs and user adoption. Um what do I mean by that? Um so I do consider user adoption to be a KPI that you can measure. Um but I think it is a very important one to kind of measure and look at things separately. Like I said active users top queries this can be things um active users in the state of a data product. I care more about that maybe from a dashboard that somebody's interacting with not if they're in an ID doing work.
Um so query patterns kind of talked about this with uh the facility directory work we had done. Um really measuring what are people commonly doing um and how can we provide a new feature or new data product itself that solves for that. I will say AI has uh data bricks specifically with AI and their systems tables have done a lot off this because you're able to actually see every query over history that users have run and I can just write a simple AI summarize is literally the name of the function and it will basically tell me what uh that query was about.
So it's a really quick way for me to analyze what people are doing. Um product reach. So, we get asked this a lot, especially on the healthcare front of how good is your coverage of data. Um, there's a lot of ways that you can define coverage. I don't know if you guys have ever thought about that, but when somebody says, "How much coverage do you have?" There's so many ways you can look to kind of look at your data coverage.
Um so uh we try and measure at least you know how many patients do we have out of the total known universe of patients in the United States for a given year. What coverage do we have for that? Um it kind of indicates how good and filled um our product is itself. So cost this is going to be something I kind of put more on our engineering team. I'll do a little bit of analysis on this. Um but that's it. the highest cost amounts really going to be in our pipelines that run.
Um but again, data bricks makes that relatively easy to see a trend of things off that front by using those systems tables and then data freshness. Um so really are things updating on the frequency you expect pipelines running the right way? Are they reliable? Etc. I'll go over some stuff data bicks has just come out with in beta um that kind of like I said tweaked my presentation because it's better than anything I had really written up before.
So the key areas that I really look at are going to be usage cost access for users query execution data lineage. Those are really kind of the core things. So for measuring uh usage and cost I'll go to the next table. uh usage and cost. It's really going to be the system billing usage is the name of the table. Um one thing I'll I'll mention off this that uh you definitely need to uh do before you can access this is turn on the systems tables.
I don't think they're on by default for everybody. So you may need whoever manages your instance to turn them on. Uh for users, they're also not on for everyone. So maybe doing a permission group level if that's how you manage access. Um so for usage and cost the system billing usage what that has in it um is going to have like the SKU costs and stuff based upon um your compute that you're using or the SQL warehouse.
Um and I think it stays up to date for uh what the current cost is. Um user access is going to be uh access audit that really is kind of more for measuring session. um query execution, query history, uh and then table lineage will show you which tables are used to build other downstream ones. It's the same thing that comes up in the UI of uh the catalog in data bricks. If you're just exploring through the UI, you can click a lineage button and it shows you uh the sequence kind of for tables.
Um you can use this a bit further to actually find column lineage. Um it's a little complicated to write it all the way out, but you can use it for that. Um so for the query history table, that's primarily what I use um right now. Uh so a few caveats with that and the systems tables in general. Uh you can't really see it with uh the text at the bottom, but the history of these systems tables only really goes back six months and then it gets deleted.
So if you want to be measuring this over time, you should be building some level of metric reporting that inserts into that either creates the table and then you insert it on a frequency or else you're going to lose greater than six months of history. Um so that's a really key thing. Uh the other thing that shows up in the query history table is going to be um a bunch of statements that I don't really care about. Show, describe, analyze, explain.
And when I mean show, I mean if a user just in the UI clicks on a catalog and tries to look at the sample data, like that registers and shows up in that query history table. I do not really care about that. It's more how somebody's interacting with data bricks. It's not how they're interacting with my product so much. Um, so I usually filter all those out and I or I just down select to uh filtering by just by select, create, alter, or update.
I usually don't even look at alter or update. I'm really only looking at create uh and select. Um another thing that shows up in the query history table is the application itself that executed um the query. So what I mean is it'll all the data bricks ones start with the word data bricks. Um so you see data bricks catalog explorer which like I said is the button clicking around. um notebooks, alerts, uh and jobs. So you can filter each one of those.
Um which is a really nice way to kind of aggregate what you're looking to measure. Um custom applications. So that is actually like written endpoints that people will write. Um so those could be any name that you define. Uh and then external connectors. So Sigma, PowerBI, um Google Appcripts, Tableau keeps going on. Um, it's the way that I don't know if anybody has looked at the execution engine of like PowerBI and how it writes queries versus the way a person writes it, but they're really really long.
Um, so especially if I'm running one of those AI summarize uh things, I throw out all of the dashboard ones because they don't really make sense for a way a person would write a query. What I mean by that is like when you select if you were selecting by like a free text field, it will say where name in and give you a list of like a thousand different things. Um, so it just doesn't mean much to me. It would if I'm like really trying to tell my analytics team to improve their dashboards.
Um, but that's really not the area I'm focused over. Uh, but it would be something I would analyze to then go to that team and be like, "Hey, you guys may want to improve the way that you configured the uh dashboard on the back end. It could use some improvements. run times are long, etc. All right, so a few examples of things that we've done on our end. Um, we get data uh every day. Um, we have an ingestion process and a pipeline that runs.
Um, so that ingests probably about 10 million records of claims per day. We get it from five different sources. So, five different pipelines that have to run. Um, and the volume can kind of be normal or it can jump around. Um, you see, I can't see the date specifically, but it's this guy right here. Gets a little bit low. Um, the table at the bottom is actually integration that I use with my Gmail to give me the information that matters because just knowing that volume went down doesn't give me any actionable insight.
I could just talk to our engineering team and be like, "Hey, volume looks down today." And they'll be like, "Okay, cool. What do you want me to do?" Um, which I'm like, "I don't know. Fix it." Uh but um some of our vendors give us data um you know through either their S3 bucket or there's an F SFTP. Sometimes there's alerting in a system that I have to log into to look at for them. But some of them actually just send me an email on an email thread.
So, I integrated our Gmail uh with data bricks to basically use the AI summary to just take their emails, summarize them, and then plot them by day on the same graph um that I'm measuring volume. So, I can see on this one, hey, there was a technical issue that caused a delay in pharmacy data. You can see at the top, this is our ARCs claims data. So, it kind of identifies why the spike went down. It tells me who to go talk to or at least it informs me that I should expect this and again inform downstream teams of that.
Um really interested to see where uh data bricks kind of goes on the uh the lickflow connect stuff. I know there's a focus on Google Drive integration right now. This had to use the Gmail server uh client which uh would not really work through um you know the Google Drive setup. So had to create a Google app script for That's where you saw me tracking that and one of the other runs below. So I actually track the way that this runs um separately.
It's funny to measure KPIs of data products. You almost have to build another data product to measure your data product if you're not picking up on that already. Um so anomaly monitoring is something new that I think is in beta on data bricks. This is where things that I had had to tell me what time like a table updated or such um I switched it uh to just start using this. Anomaly monitoring is done at the schema level.
Um so to enable this you basically in data bricks you go to uh the catalog click on the schema you want and it's on the details tab and it's called anomaly monitoring. You just turn it on. Um note with that I think it defaults to run every six hours. um which is a lot more frequent than I care about for uh monitoring some of this stuff. Um if you had, you know, live streaming jobs and things that were running, totally makes sense, but for me, everything's a daily update at most.
Um so it doesn't really matter. Um but what you can see on this at the bottom, it's a little hidden. Um is it shows the freshness of a table. Uh the red says unhealthy, the green says healthy. There's another one that's called unknown. Um, and it tells you when it evaluates it. So based off the frequency that it runs, the anomaly monitoring starts to dynamically figure out over time when should it expect the tables to be updated.
So this is a whole thing data bricks as you know using machine learning all the fancy things I don't truly understand um about. So it actually gives me a reason on the far side and it tells me when it expected the table to be updated and that it hasn't updated at that cadence. Um so really handy on that front. This is it. There's a dashboard that automatically uh gets built in AIBI the moment you turn on anomaly detection.
So it's immediately there for you to go look at. Um but also a table is created um in that schema that is your qua it's it's uh called quality monitoring summary um and it's called that for any schema you turn this on in uh but it has all these metrics written to that table. So I haven't got to fully look at it yet but that's a whole set of data that I could use to build more data products uh or uh reporting of the data products.
So, kind of same as like the systems tables, but a whole different look at this that's also applying some of um data brick's new features to it. So, I'm really kind of excited to look at that. Uh you also notice there's two tabs at the bottom. So, one of them is the quality overview. Other's table quality details. If we got enough time, I'll actually open up our instance and I can show you guys just what that looks like real quick.
Um table insights. This is not really a dynamic way of looking at the data, but it's directly in the UI. It's probably where I really started. Um, when I said that we renamed some of our tables and that I also renamed some of the columns and that I dropped certain tables, that really came about from just using this view when it first came out. Um, so this is for our medical submitted claims data product. Um, and I can basically see how often is something getting hit over time.
This is a subset of my own tablet off it that's built off the main table. So I don't see nearly as many queries as are actually running, but you can um see the frequent users of the table um show up here. So I can just see who's setting it the most. Um tells me any dashboards or notebooks that are connected to it. This one doesn't currently have any that are connected to that table. Um and then also what are the most common joins, transformations that happen.
And then further down it also has what like actual uh queries that are getting run. and it says like most frequent ones and you can just kind of scroll through and look at it, see who ran it. It's a little prettier to look at than just that system table. Um, you can also click, I think, one over on those tabs and it shows you the lineage uh, which like I said earlier maps to the lineage table uh, and systems, but it kind of gives you a dynamic.
I can keep keep clicking a button, seeing what it's all upstream, see what's all downstream. All right. So, this came about from um I'll let this run real quick and then I'll actually try and do it live. But we actually met with one of uh the data bricks folks. Her name is. She's right here. um uh a few weeks ago and she did a demo for us about how to use um the AI agent flow and stuff um in data bricks which to me was crazy intuitive.
I write a lot of UDFs in data bricks and a lot of UDTFS all in SQL because you register them to unity catalog but you can just add them to tools on these playground bots like really fast. Now granted, like I would need to give this to our machine learning team to go further with, but all I did for this um table, and I'll try and actually get a look here, um is real quick. I asked the question just who is the top user uh last month in our data.
Um and so it ends up telling me that Lance has run the most queries over the last month. And I'm like, cool, that was really handy. And granted, this is built with two functions that are basically just pulling the tables from systems tables. No prompting, nothing like that front. Um, and then I asked what he basically looked at. Uh, and it tells me he's doing pharmaceutical claims analysis, working with OMOP, and then doing some patient journey analysis, which is exactly what I actually know he's working on.
The guy's been doing a ton of work over the last month building out our actual OMOP uh common data model uh framework. So, it totally aligns with what I would expect him to actually be doing. Um, let me see if I can actually show how that is to build. Um, because I did this two days ago and it maybe took me five minutes. All right, cool. Everyone can see this, right? We'll zoom in a bit. All right. So, the first thing I do to get here, um, or to make this work, it's basically just creating a notebook.
Um, and I'm going to go ahead and set just my catalog and schema. And I'm creating two functions. So, one of them is just getting the daily query metrics. Um, so this is literally it's taking that system query history table. I'm getting rid of the show and describe. um and just getting the application that ran. So that's what tells me PowerBI or Sigma or notebooks, whatever. Um executed by I'm splitting the executed by just the at sign because it's people's email.
So it gives me the domain if I got people in multiple multiple domains. Easy way to filter out kind of the customer account. Um and then start time, it's in a time. I just care about things from a date level. Then I'm just doing counts of statements and sessions. um session count. Again, it gets a little weird with the PowerBI things and all. I've seen like massive counts off that. So, um I just kind of don't consider that at the moment for any of the dashboard things.
There's probably a way to get to that which I just haven't figured out yet. Um and then another one is getting the actual um table metrics themselves by the exeutor. So, this is how I ask the question um you know, what has Lance done is going to be running this one. Um, it's a lot longer. Uh, I do a few things to try and parse out the statement itself. Um, so if anybody wants to talk to me about this afterwards, feel free.
I'm happy to share any of this with anybody. It's not uh anything really proprietary. It's just messing with these tables. Um, so I'll just real quick run these two. And you know, because this is live, things will take a second. We've tried to use the biggest thing I can uh to run this. You can see it's it name is Big Beefy Boy. Um, another fun fact, you are able to put emojis in your cluster and warehouse names. So, and even tables.
So, have fun with that. Um, and then I'm just going to go over to this playground section. Um, and all I'm going to do is go to tools right here, and I'm just going to add the get table executed by metric. And I'll just add the other one I made, which is get table queries. Cool. And I went ahead and wrote up this little prompt. We'll start with this top one. suggest who was my top user last month. You know, it's a little different every time.
So, we'll see if it actually asks me some clarifying questions or not. Um, so, uh, immediate say to get this, I need to run this query. Um, cool. I'm good with that. Um, something definitely to refine off this would be actually adding a prompt um at the bottom. I just kind of wanted to see how this would perform out of the box. um as far as answers it could give me. So the first thing that comes back here is literally this massive table of you know uh basically the returned metrics from that function.
So when it ran, who ran it, um statement count per day, etc. I apparently ran a lot that day. Um and then we'll keep going down. Um, and then I've noticed that it doesn't do a great job actually doing like after the fact metrics and measuring. Um, so you can see here it's an empty output. So I I figured out I just needed to refine this further. Um, again prompting should pro would probably do a good bit here. Um, so I'll just ask the clarifying question of which user ran those queries in May of 2025.
Okay. And so it's saying it can analyze by user from the data. I can see entries in May of 2025. Tells me how it did it. Um so I started by doing just year and month. Let me sum the statements and it tells me the top users. Um, and so we see here as of May it's uh new. So, and then it also gives me um details on some system accounts. I mean, honestly, it's just giving me really good information about what's even going on in the system.
Um, and then I can just ask a clarifying question here for what kind of statements did run in May? And we'll see how this does from a analyzing session. All right. This is what you get for doing it live, right? It's not my friend. We'll see if I can get it to do the thing with Lance again. Uh but either way, the fact that I am as like just a product management person with a little bit of SQL can write something and immediately start asking questions like this um really kind of blew my mind that I was able to do that so fast.
Um it probably bothers a lot of the machine learning and data science people that want to measure this over time and how well is it reporting and like all those things, but that's on them to figure out. I figured this is a really good way for me to bridge the gap um of what do they need to do versus what I'm looking for. And that's where I've probably found like the best work with machine uh learning folks like data scientists like the best value they get out of me is just seeing how I interact with what they build and how I'm trying to sol like what kind of questions am I trying to solve.
It's the context that they miss when they're building things. Um and this to me helps bridge a lot of that gap. that lets me think a little bit more about really what do I care about before I either write up a ticket for them or like a feature request or whatever. Um, just very handy uh on that front. I will show you those two other tables real or the two dashboards real quick. um the one for the anomaly detection and data quality monitoring.
Um like I said, you have to so you do have to input on this logging table name. Um like I said, it's at the uh the schema level. So it has to run off um one of the schemas that you care about and you have to actually put in that monitoring table which is quality moni monitoring summary I think. Yeah, there we go. Um and so then it just immediately pulls back the data from there. Um I've talked to the product managers of this.
They're looking to kind of make this a little more dynamic or all of the data quality monitoring actually go into a system table. So, right now it's registered at the schema level, but they're looking to um kind of handle that in some way. I think they had said doing it in the systems uh catalog level or there may be another way they're looking at, but they're looking to make this easier is either a drop down or something like that you can just use.
Um but yeah, like I said, you could see healthy tables, unhealthy tables over time. Um, you're able to just filter. Like I said, this is out of the box. The moment you click monitoring on, it takes a little bit to run the job. And then you immediately have this dashboard to play with. Um, and so I can see here the evaluated time, the uh status, how many impacted queries are hit by this potential unhealthy um, status, which is really cool to me to actually see impact tables and impacted queries.
Um and then like I said the freshness itself. So if I like really zoom in table was predicted to be updated at 125 uh April 23rd but it was last updated uh in on the 5th. Uh that makes sense to me. This is not the schema and table I'm using um are not a data product. This is something I built ad hoc. So, uh, and probably did it a few times and that's where it came up, uh, with its measurement. It's also got the table quality details over here.
So, this is where you're actually able to, um, do further research at the table level. I can't remember off the top of my head which tables I have in here. Um, so yeah. All right. Um so future with data works AI the things I'm really looking into let's switch back um are going to be using the AI agents more looking at the anomaly detection um really digging more into what's happening with the l lake house federation connectors would really like to kind of sync that up to some of the Salesforce work I could really start actually measuring our accounts versus what gets used um so been trying to dig into that a bit uh they announced last year uh at the data works conference um and kind of came out a little bit after that.
So been playing with that. Uh and then there's also same as the anomaly detection there's um a new thing called custom data classifications uh or data classifications at the catalog level um which right now is really based around like PII and personal health information. Just auto classifies columns that way. But you're going to be able to set up custom classifiers that it can auto detect. Um so looking for that as well.
It's going to help me more upstream on what I used to build uh data products and yeah so thank you everybody.

Learn more about the Databricks Data and AI platform.