Skip to main content

Genie ZeroOps for Data Quality and Compliance at Scale

Summary

  • Genie ZeroOps is a background agent that monitors the entire Unity Catalog data estate for quality anomalies and PII exposures, traces root cause through lineage, and presents a proposed fix with a validation loop before any human escalation.
  • Data Quality Monitoring provides schema-level anomaly detection without manual rule-writing, while Data Classification enables smart PII scanning with built-in and custom classifiers across Unity Catalog assets.
  • A live demo shows a data engineer resolving a broken revenue dashboard in two minutes and a governance lead closing a PII exposure affecting 1,000 users with a single confirmation, both through the Governance Hub.

Genie ZeroOps for Data Quality and Compliance at Scale

Watch: Genie ZeroOps for Data Quality and Compliance at Scale
Organizations have more data than ever — but less confidence to use it. As data estates scale, manual quality checks and compliance rules can't keep up. A stale table feeding a revenue forecast isn't just a data problem, it's a problem feeding critical business decisions. PII in an unmonitored catalog isn't a governance gap, it's compounding compliance risk. In this video, we'll introduce Genie ZeroOps: a background agent that watches your entire data estate so that when issues arise, a proposed fix is already waiting for you. Built on insights from Data Quality Monitoring and Data Classification, ZeroOps detects anomalies and PII across your assets, traces root cause through lineage, and drafts the fix with a validation loop. In a live demo, you'll see a data engineer resolve a broken revenue dashboard in two minutes, and a governance lead close a PII exposure risk with a single confirmed policy. Both get centralized visibility across quality and compliance in the Governance Hub. The agent detects, investigates, and fixes. All you need to do is review and accept. That's ZeroOps.
Talk By: Jacqueline Li, Product Manager, Databricks ; Viswesh Periyasamy, Staff Software Engineer, Databricks ;

Chapters

FAQs

What is Genie ZeroOps?

Genie ZeroOps is a Databricks background agent that continuously monitors your Unity Catalog data estate for quality anomalies and PII compliance risks. When an issue is detected, ZeroOps traces root cause through data lineage and drafts a proposed fix with a validation loop, so the data engineer or governance lead only needs to review and accept.

How does Data Quality Monitoring work on Databricks?

Data Quality Monitoring provides schema-level anomaly detection across Unity Catalog tables without requiring teams to manually define rules for every column. It generates impact scores and root cause indicators that help prioritize which issues to address first based on how critical the affected assets are to downstream consumers.

How does Databricks detect and manage PII in the catalog?

Data Classification scans Unity Catalog assets using smart PII detection with built-in and custom classifiers to identify personally identifiable information across tables and columns. Genie ZeroOps extends this by automatically proposing policy updates — such as column masks or access restrictions — when PII is found in monitored assets.

When will Genie ZeroOps be available?

Based on this video, Genie ZeroOps is planned for private preview in July, with data quality checks in beta and expanded HIPAA classification support also on the near-term roadmap. For the latest availability information, refer to the Databricks release notes and product announcements.

Full transcript

[00:08] We're gonna get started, but folks can keep on trickling in. Um, we're just going to go through some quick intros. I'm Jacqueline. I'm a product manager here at Datab Bricks. Um, working on some of the features that we're about to demo today. Joined by
[00:21] >> My name is Vish. I'm an engineer here at Data Bricks. >> Yeah. So, we're super excited for all of you to be joining us. Looks like we have a full house, which is awesome. Um, but yeah, if you're here uh to learn about
[00:34] data quality and compliance as well as how zero ops is going to help with that, you're in the right place. Um, so we're going to talk about how you can proactively ensure data quality and compliance at scale with some of the
[00:48] existing capabilities we have here at Data Bricks, as well as some new product announcements. Um, hint hint with zero ops on the screen there. Um, okay. So,
[01:00] kind of a quick agenda of like what we'll be going through. We'll be talking a little bit about like the common challenges that we hear customers face in this space. Um, as well as some of current capabilities. We'll demo that as
[01:12] like a first demo and then we'll walk into zero ops as like the new product announcement and then show what that looks like. uh do a quick recap at the end and then finally road mapap on like when all of these things are landing.
[01:26] So before we really get into it, I want to get a feel of the room. Can we get a show of hands? How many of you all get pinged when a downstream dashboard or report looks broken today?
[01:40] Okay.
[01:53] Okay. Not not the whole room, but we have like a good half. Um, so it looks like we're all relatively in the same boat. The business finds out about something and then we're the ones to get pinged about it. Um, and here's the
[02:07] thing. As the scale of your data estate keeps growing, so do the amount of data issues. What we've seen here at data bicks is large enterprises on average have over 10,000 tables in Unity catalog and that
[02:22] number is growing year-over-year. So the volume of data is growing up and up to the right and so is data access. As you can imagine now with more workloads, apps and now agents there are more
[02:36] surfaces than ever that are accessing this data. So the stakes have never been higher. What this means is that a stale table isn't just an outdated table today. If
[02:48] that table is being used in a critical downstream report that maybe your CEO is looking at like your revenue forecast, now critical business decisions are being made on stale data. The downstream
[03:00] cost isn't just the data issue, but it's actually a revenue issue. Meanwhile, if you have PII in some obscure table today, it doesn't just stay there. So you might think it's just
[03:12] in that one table up there, but little do you know, it's being unknowingly propagated downstream and accessed by thousands of users. So this now becomes
[03:24] an endless chase to find all the possibilities of where this PII is and where the exposure lies. This kind of leads us to ask, how did we get here in the first place? Seems harder to kind of
[03:36] like find these data issues and find them at scale. So at one point if you needed to catch PII or if you had um wanted to know the quality of your tables, you might have wrote some rules against the tables that mattered. So
[03:49] maybe these were just like a handful of tables that you knew your CEO or like your downstream business reports pull data from. Um but as your data estate grows,
[04:02] there's more use cases downstream and demand from the business for more data. So you're bringing in more jobs, creating more jobs, more tables, and it continues and now you have this big
[04:15] world, your big data state um and only a few rules and a few tables that are really monitored. So maintaining custom checks for every data set becomes unsustainable. You're
[04:27] in one world where you're making maybe writing some rules for some the most important tables, but now what happens if that kind of like quadruples in size? This means that a limited number of
[04:39] moniitored tables means that your data most of your data state is unchecked today. This creates blind spots, costly audits, and it limits the overall trust in your data estate. So the result is
[04:52] we're now living in a world where organizations have more data than ever but less confidence to use it. So how we see it is this is a two-part problem. First, we want to be able to
[05:06] like find all these issues and where they're lurking. So, where is PII? What are my most unhealthy or problematic tables or schemas? Um, and the second part is actually how do we resolve it and resolve it at scale. So, today we're
[05:19] here to show you how data bricks solves both. Okay, so today current capabilities there's two products that launched in the last year designed to solve this very problem at scale. First, we have
[05:32] data quality monitoring. This is this one's in public preview. Um and essentially data quality monitoring is enabled across all your tables. You enable this from a schema level or across schemas. And then we'll actually
[05:45] learn from the historical data patterns of those tables to forecast if there are anomalies. So if your table isn't getting updated on time or if it's actually missing the expected amount of data, we'll actually flag that for you
[05:58] across all data sets. No manual rule writing required for data classification. This is a GA product. Um following a similar model, you can enable this across all your tables from
[06:11] your catalog. So you can enable this cross catalog and then we'll essentially look across all of your new data or change data to figure out if there's PII lurking in there. So you kind of get a sense of like where that um where PII
[06:24] and where sensitive data lives across your estate. Both of these products are built on Unity catalog which means that we leverage the native context um such as the metadata, the lineage um to help us
[06:38] adapt to your data patterns over time. So these aren't just static rules. They'll learn based on how your d your data and your business evolves. So enough of me talking about it. Let's just see how this looks in action.
[06:52] Demo time. Okay. So, here we are in datab bricks. Um, and I'm essentially in the catalog explorer page. So, we have a new drop
[07:07] down up here that's like govern as you can see from there. Um, and then what lies under this dropdown is a bunch of capabilities that you can use to govern all of your cataloges and your data. So
[07:19] alongside govern tags, we have data quality monitoring as well as data classification to the two features that we just called out. So I'm going to walk through data quality monitoring and what that looks like here. If you click into
[07:31] this uh dashboard, you get a holistic view of the quality across all of your cataloges. So right away you can see how many healthy tables versus how many unhealthy tables you have across all of
[07:43] your data sets. And then across all my cataloges, it looks like I have 88% healthy tables. So, relatively okay. But what's the problem here? So, I can see
[07:55] bubbled up at the top that the customer catalog actually looks quite problematic with almost a 100,000 tables monitored. Um, and there's almost 2,000 tables that are critical that are unhealthy.
[08:10] You can see exactly who owns the catalog. Um, so that you can ask exactly like what's going on. And it actually looks like I own this catalog. So I'm responsible for figuring out um why all
[08:23] those tables are unhealthy. Okay. So if you click in now, you get a drilled down view, a filter down view to the specific catalog customer. And then right away you can see that we reveal all of the 11,000 tables that are
[08:36] unhealthy. Um, and if you scroll here, you can see exactly why they're unhealthy. For example, if there are high nulls or if they're stale. You can see what the root causes, such as if the upstream job
[08:50] failed. And you can see who's assigned for actually fixing those quality issues. On the right here, you see there's an impact score. So, this is kind of like an aggregate that will tell you what the
[09:02] downstream impact is. So we'll look at how many users are um querying this table or if it's actually feeding critical certified assets.
[09:14] Alongside that you can see the scan frequency. So this is how often the table is actually usually updated and also how often data quality monitoring goes in to scan those tables. So we sync our scan with how often your tables
[09:27] actually update and then you don't need to maintain like a per table schedule. All right. So right here, if I see there's actually a gold schema, which I know that the business uses all the time. Um, and it looks like it's stale.
[09:41] So it's not even getting data at all. You can actually go in and review exactly what's going on. So, looks like this customer order summary has stale tape stale data because it was expected to receive data by June 15th around like
[09:56] noon, but hasn't been updated since 11:00 a.m. So, right away, I know that okay, in the last hour, I expected data, but I didn't get anything. Um, and there's actually a root cause bubbled up
[10:08] at the top that tells you it looks like the upstream job actually failed. So one zero like the direct job upstream zero levels upstream um is actually failed meaning that this table isn't getting
[10:21] data at all. So you can see you're usually getting commits around like every hour and then right around that time frame the stale window you didn't get any updates. Um
[10:34] similarly with completeness you can see what the row volume looks like. So usually this table gets around like a 100,000 rows and then it dropped right around when the table stopped getting data and
[10:48] then this pattern is revealed for all data sets that you have for this catalog. So right then and there you kind of get this holistic view for all of your quality issues. Um, and then you can
[11:00] also go filter down to a deeper level from like catalog to schema all the way down to the table to figure out what's problematic. Now I'm going to pass it over to Vish who's going to quickly demo what data
[11:12] classification looks like. >> All right. So from the same page that Jacqueline just showed you, I'm going to show you the classification page. So this is kind of like a metas storewide view of the results. So you can also
[11:24] drill down by different cataloges, but you can have like a a single view over everything that's going on. Uh the first thing I want to highlight is this chart which shows you which tables were scanned in the last 30 days. This kind
[11:36] of highlights our uh smart scanning feature. So basically it's intelligent and it's kind of like optimized to scale with your data state. Uh which means we're only focused on incremental data um and new and changed data. And this
[11:49] means that we can deliver high performance results um at scale. Um down here you can see the different PII that we've actually detected. So for example, you can see name, email
[12:01] address, IP address. Um we actually cover lots of different geos and regions as well as compliance standards such as GDPR, HIPPA, CCPA and others. Um you can also see these uh blue tags. So these
[12:14] are custom tags that our organization set up. So for example, internal account number and session ID. So let me show you how we set those up. So you can come to this custom classifier section and if you click add classifier, um you can
[12:28] pick a tag and then you can just provide some sample data. So uh let me go back and show you what this looks like for one that we've already set up. So this is device fingerprint and all I had to
[12:40] do was pick a few columns that I know already contain what I want to tag. So in this case, device fingerprint from this device registry table as well as from this login anomalies. Um, so you can see that this interface is pretty
[12:53] declarative, right? You just point us to the data and we're going to look at the values. We're going to look at the context. We're going to look at all the rich metadata that's inside Unity Catalog and we're going to learn the pattern for you, whether that's based on the metadata or the data or both. Um, so
[13:07] that we can detect anything that you throw at us. So let me close that. Um, and then you can see all the detections here in one place. So once we've actually gone and scanned your tables, you can click here to see actually which
[13:19] detections came up. So here's a full list of all the columns that were detected. So this is for name, right? And we can see for this table, for this column, we can also see the sample value. So this is exactly what our
[13:31] system detected to decide that this was a name. Um, and you can see this for all the different columns. Um, so one thing I want to highlight here is this exclude column. So if I go here and I click exclude, what this is going to do is
[13:43] actually untag this column and it's also going to inform our system that I have a preference not to tag things like this. So this is super powerful if you want to make some contextual changes. So for
[13:55] example, if a location is only sensitive to you if it comes from a user or an IP address if it's non- internal, um you can come here and kind of show us um your preferences by excluding certain detections and we'll extrapolate that to
[14:07] all of the future detections. Cool. Um, the last thing I want to highlight is that all of this is backed by a REST API. So, you can use this with Terraform or whatever way you prefer to
[14:19] productionize this in your system. All right, I'm going to hand it over back to Jacqueline. >> Yeah. Okay. So, Oh, clicker.
[14:32] All right. So, what we just showed you is how you can kind of go from this world um where you have blind spots. you don't know where PII is. You don't know what the health of these tables are to full visibility over your estate. So,
[14:46] it's like we're living in a world where like these blind spots now become monitored. You're monitoring everything. You know what you get a pulse of all of these tables, what the health is, where PII is, and then we actually flag for you if there are issues. Um, so what we
[15:00] showed you is you get full coverage over all of the the tables in Unity Catalog. And then you have both of these holistic views for both data quality and classification in one place.
[15:13] So this allows governance to scale as your data does. All right. So we talked a little bit about the first problem there like detection and how we find all of those issues across all of your data sets. Now
[15:27] let's start to explore the second part of this problem. um how do teams actually resolve all of these issues that are found today? So we don't want just like a bunch of problems. We want a
[15:39] a lot of solutions. All right. So let's look at like what the process currently looks like. As we saw earlier in the room today, all of us kind of raised our hands like we get pinged when a dashboard looks broken
[15:51] downstream. So the business is usually the first one that actually finds out about these issues and then we're the ones who actually have to go investigate. So what that means is we'll actually check if it's a real issue and
[16:03] then maybe that means we start looking at the logs or the metrics upstream. Um look trace through lineage manually to see where issues are coming from. And then finally we take a proposed fix that
[16:16] we have to like a dev environment where we try to run some tests and then finally push that through our CI/CD process to actually ship it. So that's kind of like what the loop looks like today. Um, and what we've been hearing
[16:29] from our customers is that the main pain is almost like in step number three here, like root causing and finding what where an issue is coming from when there's a bunch of upstream dependencies. And maybe we started using coding agents
[16:43] to help us with that. So maybe we actually spin up a coding agent and ask, hey, can you like traverse the upstream to figure out what's going wrong? But oftentimes what that means is you're then adding more context here and there.
[16:55] Sometimes you need to feed in, hey, like look at this workspace, ignore this, include this set of um context to look at, but don't look at the other irrelevant ones. And then what ends up
[17:07] happening is it's a very inefficient process to actually get to the right proposed fix or answer. But what we can do here at data bicks is we actually have all of this context um directly in
[17:20] the platform. So, we want to make it the most accurate, efficient, and best experience to debug and maintain your assets um on data bricks with the the agents that we're building.
[17:34] Um so, what we're seeing here is every time there's a new issue, if you go through the same loop over and over again, we see a lot of customers end up thinking, do I need to build a whole ops infra as a result of this? How do I
[17:47] actually handle these issues at scale? And that's exactly what we're here to solve today. So announced on the main stage today is the next generation of how you monitor
[17:59] your entire data estate. So we're announcing Genie Zero Ops. Woo. And this is a background agent that watches the entire platform for you so
[18:11] that when these issues arise, it already comes with a proposed fix. How this will actually work is first we'll actually pull insights from these scalable detection services that we just
[18:24] showed you like data quality monitoring and data classification. So we'll actually look at what the problems are across your entire estate. And because we have signal on all of the tables, this makes things like the second step
[18:38] um more efficient. So we can actually do root cause with a complete picture of what issues there are and how we can correlate them together. Um so you're not stuck trying to debug symptoms, but
[18:51] rather you get a clear signal on the actual root cause. Because it's platform native, we'll also use lineage. So we'll look all the way upstream um to figure out where the problem is coming from as well as look
[19:05] all the way downstream to see what the business impact and exposure is. All of this is done um so that a fix is ready for you before you even log on. Um and because this is also integrated in
[19:18] Unity catalog, this means that we also leverage uh UC governance and the insights that we pull are tailored to you to the assets that you care about as well as for the assets that you have permissions on. Um so Genie Zero Ops is
[19:33] kind of our way to actually help from help you from finding an issue to resolving it um overnight so that when you log on the fix is already there for
[19:45] you and you can actually just go ahead and merge the PR. So that's kind of like the dream. Um, one last thing I want to mention here is we often hear customers ask like if I do this with my own coding
[19:59] agents, um, can I get to the same result? So with that, what we've seen is we actually have so much context from the platform on previous issues, previous common job failures, the most
[20:13] there's so many breakages out there and we're feeding all of this context directly into Genie Zero Ops. um and basically taking that off your shoulders. Um so what we would recommend is give zero ops a try here. Uh this is
[20:28] the best way to actually manage, maintain and debug your assets on data bricks. So I'm going to enough of me talking. I'm going to demo exactly what this looks like.
[20:40] So we go. Oh, actually this is a different type of demo where we're going to be working at a retail company this time and I'm going to be a fellow data engineer and then Vish is going to be from the governance team.
[20:53] Um, and then this is a typical Monday morning for us. So typical Monday morning is I usually get pinged from maybe someone downstream that a dashboard looks broken. But what
[21:05] I see here instead is a message from Genie Zero Ops that tells you exactly uh what the issue is. So it looks like there is a problem with my customer profiles table. Um and then there was
[21:18] actually an issue with customer ids. It pulls out the root cause directly in the alert and then tells you what the proposed fix is. So, if I go click into that, it'll take me to data bricks
[21:31] with a thread already pulled up. Um, and what you can see is it'll give you a summary of the problem. So, say customer ID in customer profiles is now
[21:43] 98% null, which looks like a huge issue. Um, it should never be empty. So, the row volume actually looks normal and the table is updating on time, but nothing failed. So it looks like this column is
[21:58] blank. It then pulls up that there was this ETL notebook that reads customer ID from an upstream table called raw Salesforce contracts and that column was deprecated. So it was tagged with a
[22:11] certification status of deprecated and it actually pulled metadata from the column comment saying that we should be using customer UU ID instead. So, it's leveraging all of this native metadata
[22:24] that we have in Unity catalog. Um, and then giving us a summary of exactly what the issue is at hand. If you look here, uh, there's actually a graph that shows you exactly when nulls
[22:37] spiked in the customer ID column to 98%. And if I go down here, I can actually see what the impact is. So, it looks like 11,000 tables are actually impacted. There's
[22:49] six certified dashboards and there's actually two Genie spaces being renamed to Genie agents um that lose the customer key. So as a result you can actually see like
[23:02] the whole blast radius of this issue downstream. It's not just in customer profiles but impacting all of these downstream data sets uh and these downstream dashboards.
[23:14] All right. So if I go down here at this point, this is when I would be like looking through a bunch of logs, pinging a bunch of different channels trying to figure out where this issue is coming from. But the root cause is actually right here where the customer ID is read
[23:27] straight from this upstream table where it was deprecated. Um, and I should be using this new column called customer UYU ID instead. So if you look in the lineage, I can actually verify that yes
[23:39] indeed upstream of customer profiles is that raw Salesforce contacts table. Um, and there's a proposed fix right below. So instead of using customer ID, it actually just makes the change for me.
[23:51] Um, and the coolest thing here is it's validated in the sandbox. So customer ID is sourced from customer UID. With this change, the null rate will go back to 0.4% and then all the rows get
[24:03] repopulated. So that's awesome. Um, it didn't actually make the change for me. I get to actually make the review here. If it looks good, I can then create the PR. Um, but it did test it for me. So, I do
[24:16] have more validation um than just seeing the proposed fix alone. What happens here is now a PR is created and then I can follow through with the rest of my
[24:28] development life cycle and then push this through to CI/CD. So, that's awesome. Um, what I just showed you here
[24:42] See? Yeah. What I just showed you here is how you can go from these many operations to zero manual operations. So before you have to go through all of these tasks yourself, you have to go through lineage, try to investigate the alert, ping new downstream teams, um try
[24:57] to ping upstream teams to see where the problem is coming from. Instead, all of that is pulled into like one single thread for you. All you have to do is review and accept. So as we saw this is like the thread
[25:09] where we have the root cause directly in line. Um and then you also see the impact in the downstream lineage as well as the proposed fix all in one singular thread. So this allows maintenance to scale as your estate does and you don't
[25:22] have to spend more time manually debugging. So that's why it was useful for me as a data engineer. I'm going to pass this off to Vish who's going to show us why it's also helpful for the governance team.
[25:34] >> All right. I know there's probably a bunch of you out there who also care about the the governance side of things. So this is kind of where I go every day. This is the governance hub. So governance hub is this account level single pane of glass which gives me
[25:48] visibility across my entire data state. So that includes things like uh usage compliance security health and more. For example, I can see you know how many queries are being issued against all my
[26:01] data each day. I can see what percentage of tables have detections or might be out of compliance. Um I can even see like what percentage of tables are unhealthy. So this doesn't quite look 100%. We can
[26:14] click review data quality here. Um and this is going to automatically give me this like metas store level breakdown across my entire data state. But I can also see bubbled up insights from Genie Zero Ops. Um so look at this. We have
[26:27] customer ID is 98% null from a deprecated upstream column. It's breaking 12,000 columns, but I can see my top data engineer, Jacqueline's already on the case, which is great. Um, as a governance lead, I still want to
[26:39] learn more though. So, I'm going to click into this. This is going to take me to that workspace level dashboard that Jacqueline was showing before, but now I can kind of see the Zerops insights
[26:51] specific for this customer catalog. Um, if I go ahead and expand this, we get visibility into the same ZerOps insight. But notice now that I can see that Jacqueline is the owner here. And I can
[27:03] see that she's already got a PR that she's reviewed and the remediation is in progress. I can even click into the PR if I want to. So this is super important because even if my data engineers are going and fixing things, I need
[27:16] visibility into these kinds of issues. So I can communicate that with my team and my different stakeholders. Um and if I'm curious, I can come and read the same sort of report um everything that Jacqueline was able to see. but um just
[27:29] better communication on who's working on this. So that's great. Looks like Jacqueline's on this one. Um let's go back to governance hub and see how everything else looks. Um so over here
[27:41] on the right, looks like I missed this before, but there's a data recommendation that says there's unmasked PII and five cataloges. That doesn't sound good. Uh names, email, bank, and tax IDs. Um, it looks like
[27:54] this customer catalog is missing an Aback policy. And because of that, a thousand users have unmasked access. That sounds terrible. So, let's click into that. Um, it looks like there's a a few other zero ops insights here, but
[28:06] the first one is the the most critical with a thousand users. So, let's take a look at what's going on.
[28:23] All right, live demo. That's how you know it's live. >> Yeah. Okay. So, here we are in the data classification results page. So, this is going to pre-open the zero ops insight. So, let's take a look. Um,
[28:36] right here we can see unprotected customer PII exposed to thousand users. I can see all the top level metrics that I care about. What's the impact? How many users? How many columns were detected? How many PII classes? Um, it
[28:50] even lets me run this query to validate what's going on. So, let's run this query. Um, while we're doing that, um, we can take a look at the other information that Zerops provided. So, if
[29:02] you take a look at the thoughts here, you can see exactly how it came to this conclusion, right? So, it's quering the system table that contains all the PII results and finding which tables were flagged. Um, it's checking the access exposure by seeing um, who's querying
[29:16] through the lineage these columns. Um, and it's even auditing the existing policies that we have in other cataloges. We can also take a look at exactly which PII was found. So, for example, looks like there's a lot of name and email
[29:28] address, um, which is what I would expect because this is the customer's catalog. And we can see the per table breakdown, which shows me exactly which tables were detected, what PII, and what tags. Um, okay. So, here, look at this query.
[29:42] We can see that all the values are unmasked, right? I should not be able to see Sarah Clark or her SSN. That looks pretty bad. Um, and we can also see exactly which tables this issue is spreading to. So, it looks like because
[29:54] of these four tables feeding into these jobs, these notebooks, this dashboard, even this Genie agent, it's propagating to downstream tables as well as all the users. Um, okay. So, problem is confirmed.
[30:07] Let's take a look at the proposed fix. So, ZerOps has two proposals here. It says we should tag the tables by sensitivity tier. Um, and that we should also create an Aback policy. Um, and we
[30:20] should create this on the customer catalog and it should be for these two policies. Um, so this overall looks good, but I know that I need to exclude the security team um because they need
[30:32] to retain access to the sensitive data for their operations. So let's say it looks good, but can you exclude the security team?
[30:45] Okay, so that's going to go ahead and update for me. So this is kind of demonstrating that zero ops can kind of iterate with you live if you have adjustments. This is why it's super important to have a human in the loop. And it's going to go ahead and update this policy and exclude the security
[30:58] team. That's great. And then you can see now the 12 members of the security team group which I have in this account are going to be excluded. Um so let's go ahead and click apply.
[31:10] So while this is applying, let's just take a quick look at what was going on here. So we had this first step, right, which was tag tables by sensitivity tier. So what zero ops did here is it looked at the other cataloges that I have and it kind
[31:23] of identified the tagging structure like the best practices that my organization has to decide which table should be classified as restricted and confidential because my organization doesn't think in terms of PII tags. It
[31:35] thinks in terms of sensitivity tiers. So it kind of looked at the other cataloges and saw that we should organize them this way. Um and then it looked at those cataloges to figure out um what are the other policies we have set up and what
[31:48] is missing from this one. And so it's using the same masking function um to mirror what's going on in sales and finance. Um and this is probably the most important part. It's telling me the projected impact of the policy, right?
[32:00] Like how many users are about to get masked access, how many columns and tables. um why this is super important because um as a governance team even though these issues are urgent I have to
[32:12] make sure that I'm not breaking all of my downstream users by taking these actions. So Genie Zero Ops is kind of giving me the evidence that I need to feel confident in this kind of decision. And when I went ahead and iterated on
[32:24] the policy it actually updated that impact. So it's showing me live what's going to happen. Um this is going to kind of help me build confidence in what I'm about to do. Um, and then we can see that Zurops went ahead and did it. It tagged all these tables by sensitivity
[32:37] tier. Um, it created these two Aback policies and excluded the security team. So, that's great. Um, it's even giving me a link to verify this. So, let's open that Genie agent which it told me was
[32:51] impacted by this issue. And let's run a quick sample query um, just to confirm that we're actually getting the masked data. And we are. That's great. So you can see we're
[33:04] applying like a Shaw 2 function to all of these values. Um this is great because it lets me preserve the um the identity of these rows while still making sure that nobody can see the actual values. So I can still do
[33:16] aggregations and grouping and u any other analytics I want to do. Um but that's great. We're we're able to see that the data is now masked. Um as the uh governance lead, I also care about
[33:29] the quality, right? And if I look at the data tab, like what data is available to this Genie agent, I can see as expected that these tables are unhealthy, but I can see that Jacine's assigned to the issue and there's a fix in progress. So,
[33:42] this is super important for me and for the rest of my users to kind of understand what's going on with respect to the quality of the data that I'm querying. Um, and it helps users build confidence in in what they're the
[33:54] answers that they're getting from my data. All right. Um, so that looks great. Looks like we solved all the issues. And now let's go back to these slides.
[34:12] Okay, so what you just saw is that ZerOps made it super easy to handle all of these operations in line. That means my team doesn't have to do any digging through all the noisy failures. Um, we don't have to chase down stakeholders. Um, and we don't have to wait for issues
[34:25] to be fixed. Zero Ops did everything overnight for me and all I had to do was review and approve just like Jacqueline. Okay, so now you're all probably wondering when can I have this? So if
[34:37] so, this slide is for you. Um, so for data quality monitoring, we're moving to beta this fall to add quality checks by slice. So this means that you can segment your data and apply quality
[34:50] checks for different partitions. Um, and then by the end of this year in beta, we'll have more quality checks out of the box. So Jacqueline showed you freshness and completeness. We'll also add nulls, uniqueness, schema changes, and your other favorite data quality
[35:03] checks, as well as a tight integration with Genie Zero Ops for root cause and impact analysis. Then for data classification, um, by the end of this summer, we're going to take
[35:15] custom classifiers to public preview. So that was the feature that enabled you to have any kind of custom tag classified by data classification. Um we're also going to expand on our existing compliance standards including GDPR and
[35:27] CCPA to cover the full breath of HIPPA as well as the other important regions and geos. Um and then on an ongoing basis we're going to continue expanding all the out out of the box classifiers so that you
[35:40] don't need to build any of those yourself. Um, and then by the end of this year, we're also going to integrate that with Genie Zero Ops, so you can get the same kind of compliance, breach, investigations, and remediations that we just showed you. Um, lastly, Geniezer
[35:53] Ops is going to private preview in July. Um, so that's the feature that's going to help you diagnose and fix all your failures automatically.
[36:08] >> All right, I think that is about it. >> Last slide. All right. So, if you're interested in those detection capabilities that um we demoed as part of the first half of this for data quality monitoring and classification, um you can get started
[36:22] through your govern dropown in catalog explorer. And then if you're interested in Genie zero ops and resolving all issues at scale, we actually have a QR code. So, it's like super easy um to join the private preview list and then
[36:35] we'll be in contact there. Awesome. All right, thanks for coming. >> Thanks everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.