Monitor Quality and Compliance at Scale with Data Intelligence Powered by Unity Catalog

Summary

• Intelligent Monitoring: Lakehouse Monitoring brings automated quality checks and anomaly detection to thousands of tables on the Databricks Data and AI platform.
• Compliance at Scale: Data classification and automated monitoring enable enterprises to shift from reactive firefighting to proactive quality assurance across their entire data estate.
• Unified Governance: Unity Catalog integrates quality monitoring, data classification and observability, providing end-to-end visibility into data and AI pipeline health.

Monitor Quality and Compliance at Scale with Data Intelligence Powered by Unity Catalog

Talk by Danny Chiao, Staff Software Engineer, Databricks ; Jacqueline Li, Product Manager, Databricks
Delivering trusted, high-impact data and AI requires more than basic checks — it demands intelligent, automated quality monitoring. In this video, discover how quality monitoring scales with Unity Catalog and brings data intelligence to all your assets. Learn how Lakehouse Monitoring, anomaly detection, data classification, and Lakeflow come together to provide end-to-end visibility into the health of your data and AI pipelines. Discover how you can shift from reactive firefighting to proactive, scalable quality monitoring across your entire data estate.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and challenges
• 7:52 New product announcements
• 9:39 Product demonstration
• 27:55 Recap and product context
• 32:21 Future roadmap and conclusion

FAQs

How do you monitor quality across thousands of tables?

Lakehouse Monitoring with Unity Catalog provides automated quality checks and anomaly detection across large-scale data estates.

What is the average table count in enterprises?

According to research, enterprises manage approximately 1,500 tables in Unity Catalog, with numbers nearly quadrupling every year.

How does automation reduce risk?

Automated monitoring catches quality issues proactively, preventing problems from breaking downstream dashboards, reports and analytics.

How does Lakeflow integrate with quality monitoring?

Lakeflow components work together with Lakehouse Monitoring to provide end-to-end visibility and governance for data and AI pipelines.

Full transcript

0:00 Introduction and challenges

Hi everyone. I'm the product manager working on data quality monitoring joined by want to introduce I'm a tech lead on the team. Uh yeah and I work with Jacine all the time on this. Yeah. So this session will be all about ensuring quality and compliance at enterprise scale. Um, so what will happen is I'll walk through some slides and then hand it over to for the demo, what we're all here for. Um, and real quick, we'll first jump into some common challenges that we've heard enterprises experience uh with data quality and compliance, and then we'll go through some common situations or solutions that teams will often start with.
Then we'll announce some exciting products on how we plan on solving these challenges, go right into the demo, and then end it off with a road map of where we're heading. All right. So, with that said, quick question for the room. Um, has anyone here experienced a problem with data quality or maintaining compliance? By a show of hands. Okay, pretty much the entire room. Looks like we're all in the same boat. Um, and I'm guessing many of you actually tackled these problems table by table, maybe like writing custom validation rules or like manually running these checks.
Um, but that kind of leads us to this next question. How do you ensure quality and compliance across all your data at scale? So, we're talking beyond 10 or maybe like hundreds of tables, but thousands or 10 thousands of tables. According to our research, the average enterprise manages approximately 1,500 tables in Unity catalog. So, this is what we're talking about, enterprise scale. And if that number isn't crazy enough, the average number of tables almost quadruples every year.
So that's why what we're exploring today matters so much because regardless of where your table count is today, that trajectory is almost always upward. Okay, so this is probably like what you all have today, like a data platform. you're responsible for like tens of thousands of tables that feed important notebooks, dashboards, and reports. Um, but what ends up happening is if you have quality issues that you don't know about, this ends up breaking down trust.
And if you have restricted data, this actually slows down teams from getting the access they need. So what ends up happening is you're impacting all of these important stakeholders. For example, your CEO could be reading this revenue report or your whole team of business analysts are relying on this dashboard, but little do you know, you have these quality issues or even data restrictions that prevent them from getting the data they need.
Okay, so how do we turn from this world to something like this where your stakeholders have access to all the data they need and everyone in the organization trusts the data and is able to use it. So, as you see here, instead of getting stuck at maybe only like 50% of your data platform being usable, you could be leveraging closer to 100% of your data, having it open and reliable. Okay, so let's pretend we're in a situation where this is like probably a familiar scenario to many of you.
You work at an online retailer and the most important part of your job is ensuring that all data sets are high quality and compliant. You're managing thousands of tables. We're talking product data, customer information, transaction records. All of these things are critical to keep the business running. But here's where it gets challenging. You're facing a looming compliance audit. For example, you have someone from the governance team saying, "Hey, we actually need this audit by like the end of the month."
And your data consumers are bombarding you with messages about broken queries and dashboards. And somehow through all of this, you're expected to ensure quality and compliance across all those tables we just talked about. That's a pretty tough job that you all have. Okay. So, in our work with organizations facing this exact situation, we've seen consistent solutions emerge. Firstly, we often see that teams will often lock down an entire data source as something like PII containing overrestricting the data and preventing your users from accessing maybe like the columns that they truly need.
Secondly, we'll also see that quality is only understood by a central team. So, this creates bottlenecks when anyone else in the organization is trying to use the data. They simply just don't know if it's safe to use. So this usually happens because a platform team can't completely trust individual teams to own their pipeline and then what ends up happening is you have all of these different mechanisms of ensuring quality across se separate teams and it becomes very hard to scale up.
So for example what will end up happening is you'll have individual teams manage their data quality writing these bespoke quality rules per table. So for example, you might write specific rules for one table that has one threshold. Another table you might have to configure another threshold. This becomes really hard to maintain and scale as your enterprise grows. Okay. So today we're going to show you how we can shift from this world of what solutions might have worked better in smaller environments to when you're managing larger data environments.
So let's walk to what like a scalable solution could look like. So first instead of a segregated catalog as we saw in the previous slide we want to get you into a world where these schemas are open and users can access the data that they need. Secondly quality can be understood by one central team instead of instead of quality being understood by one central team it could be understood by everyone. And then thirdly instead of writing bespoke quality rules per table we can shift to intelligent monitoring at scale.
So think about like setting it up in one place and then the system being able to like infer those thresholds for you instead of you having to manually configure those table by table. So today we have some exciting product announcements um as how we plan to help you to get to this world. Okay, so let's see what that looks like. Okay, so this is like a picture of Unity catalog today, a table um and we want to show you how you can get to this state where you have like beautiful metadata um data intelligence right there uh embedded alongside your columns of data and also like certified assets.
You have like AI generated comments but what we want to highlight here is for example sensitive data is protected. So there's this SSN column and we put a tag called like class us SSN and you can see that with an Aback policy you automatically mass that sensitive data. But while that's happening there's still open access to what's needed. So I have this user ID column. I can still perform important joins using that column.
It's not blocked off from me from using even though SSN is protected as needed. And then if you look on this right right hand side, yeah that's your right hand side. Um you see this bright green indicator. So this is our health indicator. So anyone coming to this table can understand that the data is safe and healthy to use. Cool. So wouldn't it be incredible to get here? Everyone in your organization could be autonomous in building their own data products, reports, and analytics with this view.
And today we'll show you how to get there. All right, so we're excited to announce two new products, data classification and data quality monitoring. These products are specifically designed to solve the challenges that we just talked about and to further enrich your Unity catalog with data intelligence. So both products are in beta today. Okay, so quick overview of how that actually works. Um what we've basically built is a scalable approach to data quality and compliance that democratizes your data even as your environment grows.

7:52 New product announcements

So what we have on the left hand side here with data quality monitoring is essentially you can intelligently detect and prioritize quality incidents across your data state. So across all your tables. Um, and how this works is we basically build like per table models uh that use AI to essentially detect if things look stale or incomplete and then that list will grow on with more quality metrics. And on the right hand side we have data classification.
So essentially we'll also intelligently detect and protect what's sensitive in your catalog across all your tables. So both of these products are built on top of Unity catalog. Since we're natively integrated into Unity Catalog, what advantage this gives us is we have access to historical data, your metadata, as well as your lineage. So what this means is we're able to intelligently prioritize based on your downstream impact.
For example, if you have important dashboards or reports that are using these tables um or we can identify root cause because we have access to look upstream. So these are kind of the things that we want to leverage in order to build a scalable and smart solution that helps you automate monitoring um across all your tables. All right, so I'm just going to stop telling you what it does and let's just see it in action.
Um what I'm going to do is I'm going to head over to data bicks as a business analyst and basically I have this report that I'm building for leadership. I need to like use my retail transactions data set. Um, so I have like all this data I need to use like transaction ID, customer ID, customer name, and emails. Okay, so that's kind of weird because I don't think I should have access to emails. Um, that looks like PII definitely shouldn't be using and almost looks like it's a potential leakage problem.

9:39 Product demonstration

Let me see who's responsible for this data set because that looks like an issue. I don't want to get in trouble for using PII I'm not supposed to use. Um, so let's go to that table. Retail transactions. Oh, wait. It also looks like the table's unhealthy. Okay, so it looks like this table has like some out of compliance issues and potentially there's a problem with health. It looks like our data platform team, might have a few issues to fix today.
Hope you don't have a long day. Um, but I'm going to pass it over to him now. Yep. Uh, so hello. I'm again. uh but now I'm the data platform team, data governance team and uh my goal is to ensure that all my critical data assets are uh high quality and also well well secured so that any API uh you know only is accessible by people who should have access to that. And so clearly we have a couple of problems here. Uh what we kind of saw here is that Jacine was able to access emails and she shouldn't have.
Uh I'll show you how this feature data classification is going to help us lock that down. And also there was that quality issue that we saw and uh we'll see how using the data quality monitoring I can detect and figure out what's going on uh with that situation. So with that let me first go into how data classification will work. So uh the way that you enable it is we go into the catalog explorer. Uh you'll see that there's a details tab on the catalog.
So we click into that and now you'll see that there's this new section called data classification and when you enable this what's it going to do it's going to continuously intelligently monitor all the tables in this catalog for PII and under the hood we're basically doing this smart scan so that all the tables that exist right now we're going to scan those one time but then any new data that comes in so say there's new columns that get added or new tables that come in or even if a column is changing schemas we're going detect that and the system's going to figure out we need to incrementally scan that as well.
And all this means that you don't really need to think about like manually detecting PI. You don't need to think about scheduling or doing anything. We're going to figure out the most costefficient way to detect all this PI and lock it down as as data is coming in. Cool. So, let's look at what it's I've already turned it on because it takes a little bit for it to scan through all the data, but let's look at what the results look like.
So, right off the bat, you'll see that it looks like it's already looked through all everything. It's found a good amount of PII. Um, this is kind of what I was talking about where uh it's monitoring tables over time. So, as new tables come in or as it's detecting new things, it's going to continuously protect the system without any manual intervention. And it looks like there's a good amount of PII. Maybe like the most common ones I'm seeing are like IP address or maybe some locations here.
Uh so then my question is how do I lock that down? And the answer is uh attribute based access control Aback and a quick and which we just announced is in beta uh as of today and you guys can start trying to use it I think you know later you know soon basically uh and a quick TDR of what that is is you can now trigger logic that happens if a table is uh tagged or if a column is tagged. So for example, maybe I find that there's a specific column that or set of columns I think are important.
I want to ensure that only maybe I have access to that. I can create an Aback policy. I can tag those columns as important and say within the Aback policy, no ensure that nobody can see it but me. So that's that's what aback lets us do. And that's actually a perfect setup for locking out PII. uh if only I had a way to detect all the PI in my system and also create tags for those columns. Uh and conveniently that is exactly what data classification is going to do for you.
So let's see kind of what that looks like in practice. Uh so down here I can kind of uh see another section that shows us here's all the different kind of classification types that we support today and whether or not they're currently protected by some sort of aback policy. And you can see that we've also integrated with lineage to figure out how many tables have this kind of PII in them. How many users are quering these kinds of tables and also who has access to these per the Aback policies.
But let's look let's think back to kind of what J we just saw in case. She was able to see email address and I can see actually yes there this is not protected right now by an Aback policy. Uh before I kind of look into the policy, let's just take a quick look at uh some of these examples here to see are these actually real API. And you can see that we now, you know, have this filter view and I can see some of the sample values I've detected across all the different tables.
So yeah, this these look like reasonable uh email addresses to me. I think I also maybe saw some locations in there. So let's take a quick look at that. Oh, I typoed. And yeah, these also look like maybe some city names. So, uh, so yeah, definitely it seems like this is this is worth uh looking into. These are real issues. Let's look at the actual table itself because what I just said was the system is going to produce all these tags for you, right?
So, if I click into this table now, now I'm going to see that the system's already automatically detected everything. It's figured out that this customer name column is a name. this email is an email address. All these tags are automatically applied by the system. So you don't have to think about that and it's done at this at scale across all the tables in this catalog. So that's that's pretty amazing. No manual uh effort needed automatically protected over time.
Great. So let's go back. Okay. So now back to the problem at hand. I want to ensure that I can lock this down. So I can go into this button create aback policy. I'm just going to very very quickly walk you through how that would work. So maybe I'll lock down emails is my policy name. I'll apply this to everyone. And then I'll create a column mask on and then I have to specify a function. So uh I've I've already built a function that's going to redact any sort of PI columns.
So I can see that uh this is my mask email function that I already created. So let me just select that and then I can now take advantage of all these tags that I've detected where now if it has a tag you can see now all these might have to zoom out a little bit. Yeah, I need to zoom out. Can I kind of see this? Okay, well you can kind of see it. But now I can see this class email address. So I've now what I've done just now is I'm going to redact any columns in this catalog as long as the systems detected them as an email address.
That's kind of what's happening right here. So now I create this policy. Not a valid name of course. Emails. Okay. So now I've successfully locked down all email addresses in the entire catalog. So I think what I've done just now is with a couple of clicks I have this will eventually update but with a couple of clicks I've now locked on more PI and I can continue to do that for all these other PI uh classifications that the system has detected but with just a couple of clicks I've locked on PI I've continuous I now have continuous protection of my entire catalog uh for any PI leakages and uh yeah and I can rest assured that when I go to my next compliance or audit meeting that I can prove that I'm I'm protecting all my PII.
Uh, taking another step back. So, how does this work under the hood? We've actually noticed that a lot of the solutions out there maybe don't have the highest quality. They have a lot of false positives. And so, what we've done is we've built a compound AI system that leverages a combination of huristics, NLP as well as LLMs to ensure that we're producing really high quality u detections here. And uh yeah, the list of PI is only going to expand in the future that we support.
But it just means that with this system, with this intelligence system, I'm going to have a costefficient solution that constantly monitors my entire catalog. It's going to be high quality because of this compound AI system and I don't need to tune any sort of uh do any sort of manual configuration in the future to continue to continue to ensure this production. Great. So I think I've now solved my uh PII problem. Let's solve that quality problem that we also had to solve.
So, um there was a table that had improper that had unhealthy data. So, let me go into show you how data quality monitoring is set up. So, it's very very similar. I go into the catalog explorer. There's a uh I go into the a schema this time and I go into the details tab. And now, similarly, there's now this new section called data quality monitoring. And when I turn this on, what it's going to do, as mentioned, is it's going to scan through all the tables in this entire schema.
Uh figure out create a definition for what is normal in that. So whether it's like freshness or completeness, figure out what is normal by building individual forecasting models, figuring out the seasonality of each table and automatically inferring thresholds and schedules for us to scan these tables out on. So yeah, so all that smarts means that I don't really need to think about per table thresholds. I don't need to think about per table schedules.
The system is just going to figure that out for me and do it in a costefficient manner. So let's see what that gives us. Actually, before I do that, um remember how Jacine saw that unhealthy table? It' actually be even better if I prevented that from hitting her to begin with. And I actually did get an alert from the system a couple hours ago. So, let me show you. So, this system has actually alerted me through an email that there are some potential issues that happened a couple hours ago.
So, I can take a look directly in this email at what's going on. Of course, I don't have access. Uh, let me go back and figure out what's going wrong. Okay, live demo issues, but okay. So I can see that there this is this is I've already run data quality monitoring so it's already detected everything. It looks like generally uh most of my tables are healthy but there are a couple of unhealthy tables and you can see that over time it is typically pretty healthy and it's just very recently that actually a lot of something started happening that caused tables to be unhealthy.
And if I go to this so generally what am I doing when I'm looking at quality I'm actually looking for to do a couple things. First I want to figure out what's the highest impact that you know confirm is this current issue high impact or not and then check is this a real issue is this like not a false positive and if it is a real issue I got to tell everyone that something's going wrong and that they need to not trust their dashboards and then finally I got to fix the issue.
So let me walk you through how that works in this product first. I mean immediately you get to see that here's a list of incidents sorted by impact and we're computing this by relying on integrations through uh un catalog and lineage and what not to figure out there this particular table that was crying is actually a very high impact table. It's impacting thousands of queries uh and a couple downstream tables as well and it looks like there's even like a potential identified root cause.
So seems like it's probably a real issue. Uh, but let me kind of look into this a little bit more to see what's exactly going on. And again, we're now back in Unity catalog in the UI and in this quality tab that has always been there. Now, we have all this in all these rich insights from data quality monitoring. So I can see that this is unhealthy as of a couple hours ago and that this table it looks like it typically updates every hour but in the last couple of hours it really start started just failing and not not updating.
So something's wrong there. And then you can see here with completeness that there is the systems also figured out what is a typical number of rows that are going to be inserted into this table on a daily basis. And you can actually see here some of the intelligence coming in where it's figured out there's some sort of seasonality pattern in this table, right? Maybe on the weekends it's a little bit lower data volume than on the on the weekdays.
So that's you know there was no user configuration needed. The system figured that all out on its own. Uh and so yeah, it does look like there's a real issue here. Okay. Now that I've confirmed that there's a real issue, uh as I said, I want to communicate this to everybody, all the people who might care about this table being unhealthy. Uh fortunately I actually don't need to do that anymore because this feature has computed all these health indicators that are propagated throughout the entire UI.
So as as showed we already have this uh you know health health information in UC here but also if I go into even in the rest of the UI I can see that on the schema list for example there's also going to show these health indicators here as well. you know, I can see that this is healthy, this is not healthy. Uh, so as basically as all these data consumers are quering throughout data bricks, they're going to know that their tables are unhealthy.
So I actually don't even need to communicate with them. I can just focus my attention on mitigating the issue. Let me go back to that. I guess I just go back to the retail transactions table. Okay, so I got to fix this. Now, it looks like the system's actually also figured out a potential root cause. So, let me look at this. It's the upstream job. Looks like it started failing. And I can click into that to see what's going on.
And yeah, it does look like kind of relatively recently it started failing. So, let's look at what's going on. Uh, looks like this append table task is failing I guess there must have been just some some data engineer that decided to break my pipeline. But I can see that there is an exception being raised. So I can just go ahead and uh fix that up right now and just remove that. And uh yeah, I think now going forward this job should continue succeeding.
I can probably just run now to ensure that now this is going to continue being up to date. And with that, I think I have now just go back to the dashboard. Okay. Yeah. So with that, uh, what I've just done is I've already set up data quality monitoring. I've detected these issues. I've confirmed it's a real issue, right? It's a high impact issue. I've communicated all the issues with these health indicators and I found the root cause and I was able to fix it.
And uh I guess that's kind of it. You know, it's it's a it's an end to-end solution that I set up with just one click and I set up an alert and now I have continuous data quality monitoring on my entire system. And just as a quick recap, right, like this is happening. We're building models per per table. We're doing uh intelligent scanning so that we're not going to scan not all tables are the same. So for example, if you have one table that's updating every day, we're not going to scan that more than once a day because we we can figure that out.
Whereas if you have other tables that are updating more frequently, we're going to scan that more frequently. So that's part of this, you know, overall very intelligent system that we've built. So you don't have to tune any knobs. And this is also powered uh integrating with uh Lineage and the rest of UC to figure out what is the actual impact here. And uh overall, yeah, we've built a system that hopefully is going to reduce your time to actually mitigate these issues and triage through all these high importance quality issues.
So, with that, I'm going to hand it back to and let her carry on with some roadmap discussions. Yeah. So, actually, I'm going to come in here now and I'm still that business analyst. I'm not Jacquine yet. Um, gonna go into my report and then once I rerun this, so click the rerun button. You missed it. Voila. Okay. So, what I got here because just told me the data is fixed. It's reliable. It's complete. It's fresh. So, I can go ahead and feel safe to use this data.
But alongside that, we see that sneaky little email column is actually redacted as well. So what just happened here is all the sensitive columns are redacted as needed while I can still access the columns that I need. So if I still want to perform those joins on transaction ID and customer ID, I can totally do that without any friction from a formal access request process. So thank you so much for saving me time and um providing me reliable results that I can use.
Okay, let's switch back to the slides. Awesome. Okay. So, we showed you a lot there in our demo. Um, I'm just going to go through a quick recap of exactly what we just showed you. So, beforehand, we were in a world where there was actually risk of PII leakage. So, I was a data consumer and I potentially could have just published further downstream. And if you wanted to lock down the entire table as a way to solve this, you then put your business analysts like me in a place where data is way overrestricted and your users face friction to get access to the data that they truly need.

27:55 Recap and product context

After with data classification and Aback policies that just showed us, instead PII is automatically detected for so for example that email column. Um, and we saved you the hassle of finding out where PII lives and what you need to lock down. So now you can ensure that your consumers still have access to what they need while protecting what they shouldn't. Okay. So beforehand you also had business users encountering quality issues first.
Um, data producers like would find out too late and have to reactively look into issues. But after you can set up alerts so you're the first one getting notified. And data consumers will also see health indicators directly in their workflows so they can tell if data is safe to use ahead of time and see if it's a known issue. Before it was also hard to tell how big of a problem a quality issue is. You're getting all of these pings from multiple data consumers.
It's unclear which one you want to tackle first. Um, and in terms of root cause, you're trying to dig through multiple job logs and table history, scrambling to figure out where an issue came from. After, we actually prioritize those incidents for you based on downstream impact, ensuring that critical issues don't slip your radar. And we'll link you directly to the root cause, saving you hours of time and allowing you to deliver high quality data sooner to the business.
Okay? And guess what? All of that is available in just one click. So, as a quick recap, also need to switch. Um, you head over to any catalog. So, for example, that was like the retail catalog that we saw. Go to the details tab and then you see data classification. You can enable that in just one click with the toggle. And then for the schema level, you get data quality monitoring in just one click as well. Just enable that in one toggle.
So we wanted to make it super simple for all of you to get started with this. Really easy to scale across your schemas, your cataloges, um, without you having to manually configure per table checks. So yeah, we're really excited about all of that. Um, and wanted to kind of show how that ties in with the rest of our products. So we just covered data quality monitoring, data classification, all built on top of Unity catalog.
Um but last year, if any of you were here, um we had lakehouse monitoring. So that was our data profiling product that we announced in GA. So it was previously named lakehouse monitoring. We're now naming it data profiling. Um and then we had attribute-based access control. So Aback that was announced on the main stage today. So those are all two those are the two products that we see these working together with. So let's look at quality first.
So data quality monitoring where this differentiates is this is our scalable solution for your entire platform. You can ensure quality across all your tables in one click. Data profiling you can essentially go dive deep into the metrics of tables that you really care about such as your important tables for things like exploratory data analysis. Um so that's kind of the difference between the two products today. And then for governance, if we look on the right side for data classification, that's our product that will automate compliance at scale when you use it with Aback.
So we're essentially automatically detecting tags and then Aback will automatically protect what's sensitive. So when used in combination, that's how you can scale your compliance across your entire catalog um instead of manually tagging and then manually applying those policies on top. And all this is built on top of Unity catalog. Cool. So, what's next with all of this? So, we just showed you a lot of products that we're really excited about.

32:21 Future roadmap and conclusion

Um, and we wanted to share also where we're going with them. So, this is the road map. So, for data classification today, um, it's in beta, but we're targeting our public preview, end of summer 2025. Um, but in beta today, we have 12 US-based classifiers. And, um, we also have a dashboard to help you audit the results. So we'll show you exactly what we found across your catalog and then you can understand like what the access control rules are so you can stay stay on top of access control coming soon.
So this is what's on our on our road map custom rules. We definitely want to be able to support this because you all have business unique rules that you may consider sensitive. So we'll start off with reject space to allow you to like define those. But long term, we believe that we can power this with AI um and the power of us knowing your metadata in Unity catalog to help you derive classifiers that are traditionally hard with reax.
Um we'll also be supporting more out of the box additional classifiers. So for example like PHI, PCI, um very common types of PI want to extend our list out of the box. And then lastly, we also want to be able to help you with rowle detection. So right now we'll classify like the column but we understand if you're like there's leakage in each row we want to integrate with lakeflow longterm to help you make that process seamless.
Cool. And then for data quality monitoring um we're also in beta but we're targeting similar timeline public preview end of summer 2025 and it'll support UI and API experiences. So today in beta as we saw we have freshness and completeness monitoring out. Uh we also have segmentation. This is kind of like in a private preview. Um where you can review the quality results we had by slice. For example, if you had like a store ID and you wanted to see like the freshness of those stores, um you can kind of like partition it like that.
Uh and then you can also manually set up alerts today using the logging table. So that's what showed you. If you want to stay on top of the quality issues beforehand, you can set up alerts with our logging table. What's coming soon is we'll have additional quality checks come in. So we understand freshness and completeness is kind of like the foundation, but we want to build on top of that with things like schema changes, uniqueness, very common um quality problems that we also want to help uh you all detect and fix.
Um also we're shifting to more of like automated alerting. So instead of you having to like manually configure those alerts, uh we want to be able to help you automate them by understanding how important the asset is by looking into the downstream impact to kind of help you prioritize that alert. Um as well as to see if there's any like certified assets that are impacted or if the table itself is certified. All of those pieces we can pull in to kind of help you reduce noise in alerting and also help the system um intelligently alert you instead of you having to manually set that up.
And lastly there we want to help you also monitor custom quality rules. So instead of just like the out of the box set if there are specific business rules that you all have uh we want to develop this unified syntax uh for quality. So many of you probably seen we have like multiple quality solutions like DQX DT expectations um metrics SQL alerts. So we want kind of want to bring in like one unified way for you to define these alerts for your custom rules.
And then we want to bring you observability in the same way that we showed you today in the demo across those metrics. So we'll be integrating with LakeFlow for that. Um so we can make sure that you have like one common way to do it. And that's also something that we'll be working on near the end of year. Cool. So showed you a lot of exciting stuff today. We're really um happy that you all came to our session to kind of learn about all the cool stuff that we're working on.
We can't wait for you all to try this out. Um, as a reminder, this is like how you can get started today directly in the UI. You could go head over to the detail tab in the catalog and the schema. Um, you can turn on these products in just one click. You can even do this on your phone as you head out the door. Um, so yeah, super easy to get started and we're really excited about everything that's to come. Thank you so much for coming to our session today.
Um, and we'll be sticking around for questions. So, yeah.

Learn more about the Databricks Data and AI platform.