Skip to main content

Databricks Apps for Data Quality: Adoption and Observability at Scribd

Summary

  • Scribd built a data quality platform using Databricks Apps that replaced a config-driven YAML approach with a self-serve UI, enabling data analysts, scientists, and engineers to define, test, and monitor quality rules without SQL expertise.
  • The architecture combines Databricks Apps as the interface layer, Databricks Workflows with table triggers for automatic execution, Lakebase as the rule store, and Unity Catalog for governance across Scribd's four product lines.
  • The new platform increased adoption by 445% in its first month by prioritizing user experience — including one-click rule testing, historical trend views, and actionable Slack alerts with direct links to debugging notebooks.

Databricks Apps for Data Quality: Adoption and Observability at Scribd

Watch: Databricks Apps for Data Quality: Adoption and Observability at Scribd
Most data quality tools fail not because checks are wrong, but because users do not understand how to use them. Scribd built a data quality platform using Databricks Apps to shift from a config-driven, low-adoption YAML approach to a self-serve, user-centric interface where data analysts, scientists, and engineers can define, test, and monitor quality rules without SQL expertise.
Learn how to design for adoption: build rules through an intuitive UI, test with one click, track historical trends, and send actionable Slack alerts linking to debugging notebooks. Scribd's architecture combines Databricks Apps as the interface layer, a portable quality framework, Databricks Workflows with table triggers for automatic execution, Lakebase as the rule store, and Unity Catalog for governance. See how this approach increased adoption 445% in the first month across new teams.
🤝

Chapters

FAQs

Why did Scribd's original YAML-based data quality approach fail to drive adoption?

The YAML-based approach required technical expertise to configure and generated alert fatigue because alerts lacked context and actionable debugging information. Non-technical users could not create or manage quality rules independently, which kept adoption limited to engineers.

What Databricks components make up Scribd's data quality platform?

The platform uses five Databricks building blocks: Databricks Apps as the interface layer, a portable quality framework for rule execution, Databricks Workflows with table triggers for automatic pipeline execution, Lakebase as the rule store, and Unity Catalog for governance and lineage.

How does Scribd's data quality app alert users to data problems?

The platform sends actionable Slack alerts that include direct links to debugging notebooks, giving recipients immediate context and a clear path to investigate the issue. This contrasts with the previous system that generated alerts without sufficient context for users to take action.

What results did Scribd see after launching the new data quality platform?

Adoption increased 445% in the first month as new teams onboarded to the platform. The self-serve interface enabled non-technical users — not just engineers — to create and manage data quality rules, expanding the reach of data quality practices across Scribd's organization.

Full transcript

[00:08] All right, how's everyone doing today? Ready to wrap up? All right. Who here has ever created a data quality check before? Raise your hands. Who here is a non-technical user or non-engineer that's created a data quality check before? Raise your hands. Okay, so a little bit less. Maybe we can
[00:24] speak to you a little bit more. Yeah. All right, so let's go. Um okay, so for today Ryan and I will be talking about data quality app and talk about how we use the Databricks app to drive data quality adoption.
[00:39] Okay, so next slide, please. Yeah. So, hi everyone. I'm Trini I'm a software engineer of data platform team and I have various responsibility on the team and I complete the task assigned to me. Um and also I work on data infrastructure sometimes.
[00:56] And my name is Ryan Friedus. I'm the principal data engineer on Scribd's data platform team and apparently I do questionable things. Um some of those probably ended up in this presentation. Uh let's get right into it. So, Scribd at a glance, Scribd is a knowledge company and our mission is to advance
[01:12] human understanding. And that translates into 400 million pieces of content scattered throughout S3, 250 million monthly users, 50 plus languages that we support, 30 million hours read through our platforms a month, and then 2 million plus uh paying subscribers. So,
[01:30] we got a lot of users, we got a lot of content, and we got a lot of data to power all that. So, we're actually a family of products. The first, the OG, being scribd.com. And this is the world's biggest library of research papers, legal filings, academic
[01:45] material, and manuals. And scribd.com alone contributes to 300 million plus of that content. And then slideshare.com, where the world's expertise gets 25 million presentation presentations contributed by subject matter experts
[02:01] all across the globe. And then we have a digital subscription service of audiobooks, ebooks, and podcasts with 1.5 million titles including best sellers through Everand. And then Fable, which is our social platform for people who love
[02:17] books and reading and contribute and to interact with different book clubs. 5 million readers, 200,000 book clubs. So, this is where we come in and we run Scribd's data platform. Um and and we run Scribd's data platform and
[02:34] you can imagine four different products, four different apps, four different user bases, four different event streams. And the whole value proposition of data platform at Scribd is to bring that all together and do cross-product analytics, build shared machine learning features, and build one view of the subscriber.
[02:50] But that's also where the pain lives. You have four different schemas, four different user bases, and four different upstream teams all shipping product features on their own timelines and trying to bring that all together is also creates a big wide surface of all
[03:06] sorts of data quality issues. Now, over the years we've made strategic investments into investing in a Databricks to for our platform team to build platform on top of the Databricks platform. And what that translates into is thousands of Delta tables within our data lake and thousands of orchestrated
[03:22] DAGs and tasks running every single day. Yeah. So, like Ryan mentioned, like we definitely have lots of data and we have like lots of new data coming to the system every day. Next slide, please. So, that means we
[03:37] also have like high data quality issues. So, I'm pretty sure you all know who that is on the screen. So, Yeah, so I would say like for data quality issues it's not like if they exist, it's about like when they will surface at some point. Uh next slide,
[03:53] please. So, we are looking to some like common example of their quality issue. I'm pretty sure everyone here have encountered either one or all of them. Yeah. So, one of the common of them is like let's say we have a data pipeline that's been running for a long time, but suddenly start dropping data silently.
[04:09] So, usually the issue won't get caught until some point like um maybe maybe like a month later until the data made into the finance final report. In addition to that, sometimes we have developers they try to make a data pipeline and they will try to use a table that looks promising and they will
[04:25] try to validate probably like only a portion of the data, which they think, "Okay, looks good. I'm going to use this table to build analysis pipeline." And then the pipeline and now then the table end up being having some of data corruption issue, but they just end up using the table anyway.
[04:41] And in addition to that, sometimes the source data can change. So, sometimes there may be schema change happen at the client. So, the client just send out those bad events to downstream and downstream is not aware of it at all. Okay, so next slide, please. So, what is
[04:56] oh, sorry. It's okay. So, what does data quality mean at Scribd? So, data quality means data that the people and the AI models relying on it can trust. So, we can measure those across all the usual dimensions: accuracy, completeness, freshness,
[05:13] consistency, validity, and uniqueness. And as the saying goes, if you put an error into a distributed system, you get distributed errors out. And we continue that let's not over let's not continue to overlook data quality, rather let's emphasize the importance of data quality, especially in the time of AI.
[05:28] So, we just sat through a whole week of AI agents, Omni, or um yeah, Genie, and all the AI enablement that Databricks is doing. As we continue to build our medallion models or our models in our data lake, we cannot continue to overlook data quality because those errors and those issues will continue to
[05:43] propagate into our AI activities for not only us, but for our customers. So, trust in the data is what makes every single AI bet pay off not only for Scribd, but also for you as well. So, this is a quick glimpse into what we have built. Um so, five Databricks
[06:00] building blocks, one data quality app that we ultimately shipped. So, the data quality the Databricks app is the UI serving layer where our users can now interact with data quality, manage their checks, and monitor their those checks. And then we have our data quality
[06:15] framework. We built a framework that is portable, that can be consumed by that app, and we use this framework to not only you know, write new data quality checks and rules, but also that framework executes it is responsible for executing those checks, and also storing results
[06:31] into uh the back end, which will be lakehouse. I'll get to that in a second. And then Databricks workflows. So, as people are creating data quality checks in our app, what we're actually doing is we're creating a Databricks workflow for that user, um and that will utilize
[06:46] things like table update triggers on the Delta tables that we're running those checks on. So, as data lands into our data lake, those workflows will fire, run those checks, and then lakehouse. So, lakehouse is our transactional back end. That's going to store all the data quality check config, all the rules, and
[07:03] all the execution results. The results in the checks that were executed from those Databricks workflows. And then tying everything together, Unity Catalog. So, governance over everything. We're investing in Unity Catalog. Um it gives us the lineage, the
[07:18] permissions, and also gives us the visibility on what tables have data quality checks and what do not, so that we can start to build trust across our data lake. Yeah. So, now I'll show you how we get to the point where we have the data quality app. Um so, initially we don't
[07:35] have the app. We actually have like a very simple common solution. So, there are many people that I talked to from different companies, regardless of the industry, they all have something a little bit similar. So, what do we have here is basically the YAML based approach. It's that we will put data
[07:50] config rules into a YAML file, and then we will have nightly Spark jobs. So, then the nightly Spark job will pick up the YAML rules and will try to run some data quality checks or some usually secret query based on the configs. And after this runs, we'll save the results
[08:06] into a data table, and also we'll send um alerts to a Slack channel if any of the like checks fails. Okay, so next slides, please. We like YAML, right? Um do you? Yes. Okay. So, this definitely work. So, this
[08:22] this is simple and this is work, but what's the issue of it? So, first of all, we have very low adoption. We actually have other teams who actually have a need for data quality, but they are not enjoying, you know, using YAML based approach at the field check data quality checks themselves. So, actually
[08:37] we have been adding data quality checks on behalf of them. In addition to that, we also like sometimes the alerts gets really noisy and it's hard for people to take action. And sometimes also the big picture is missing. So, usually after months of running data quality checks every day,
[08:54] people will get just a month of Slack message. Nah, next slides, please. I like YAML. Yeah. Okay, so we're looking into each issue like in more details. So, low adoption, as mentioned, not everyone is into YAML engineering. So, usually like
[09:11] so for the YAML based approach, people have to put into the YAML file in a very specific format. And also it's not easy for them to test the data quality checks in that. They will have to kick off a Spark job manually in that just to run one single
[09:27] check. And in addition to that, if they will happen to make like a typo, they type the wrong table name, they will have to go back to open a PR just to change that. So, as you can see, this whole process is not very easy. Next slide, please.
[09:42] Yeah. And also, as we mentioned, like all day with something failed, all the guys just Slack message you saying, "Hey, you this is the data quality check that failed. Can you please take a look?" And this just like not easy for people to act on. And sometimes some people
[09:57] they might feel passionate about troubleshooting their data their data quality checks. They might try to click the link, but when they got this like repetitive Slack message every once in a while, they get tired of it. So, we want to That's why like sometimes it's so hard for people to act on it. Next
[10:13] slide, please. Yeah. And in addition to all of that, uh like we mentioned, it's just so hard for anyone to know how data quality has been going on for the tables owned by their teams. They just like no one's going to analyze all the Slack messages they got
[10:28] like in a few months. Okay, next slide, please. Yeah. So, Yeah. So, like kind of like we mentioned that most of the quality tools, they don't fail because just the checks are wrong or maybe the framework is just like uh was not working correctly. They
[10:45] fail because the users don't understand how to use them. This is a This is like user adoption issue. Uh next slide, please. Okay. So, that's why we started thinking about building the new solution for our data quality. So, there's several things there's
[11:01] several requirements we want to meet. So, we want to make it self-serve. It should be really simple for people to use. So, everyone in a company, regardless of their background, even if So, someone if you someone's platform engineers or someone's software engineer or someone's data analyst, data scientist, it should all be very easy
[11:17] for them to use it. And also, we want to in have a enrich alerts. So, the alerts should help really help people to troubleshoot their data quality check easily whenever something went wrong. And in addition to that, we also want to help people to understand the overall
[11:33] trend of data their data quality, not just a single check, but like what has been going on during the time of the history. So, next slide, please. Yeah, okay. So, now we're going to finally we come to the point that we are going to show you some demo. So, every
[11:49] live demo I have been to myself um had something go wrong. So, that's why we're going to do a live-ish demo here. So, basically we have some pre-recorded GIF um that we're going to play and I'm going to speak behind it. Um so, I've been playing too much video games back
[12:05] in the younger age, so my sense of speed is a little bit different. You know, like 5 seconds in video games like forever, but like in a demo it's like a probably nothing. So, if you feel, "Wow, this is going way too fast." then that's probably true. So, if you feel like anything's like a not, you know, like not clear to you, please feel free to
[12:22] talk to us after the you know, after talk. Um we can definitely look into very specific functionalities together. Okay, so next slide, please. Okay. So, for the first thing on a demo, uh we're going to see some rule operation. We're going to create a rule
[12:39] and we're going to test the rules. And after that, we're going to update the rule and then we're going to retest it. Okay, next slide, please. Yeah. So, there are many different type of like rule we can create, so we are just going to use one that's a little bit
[12:55] simple, which is the allowed range of allowed values for the rules. So, for this kind of rules, basically we'll pick a column of the table and the column of this table can only have certain value. So, let's say that if you only have two client, which is web and mobile, this is
[13:10] something already decided by the team, and which means if you have like um if you have a So, if you have like a client column, you should only be web or mobile. So, if someday is something like iPad showing up, then that means something's wrong. Okay, so next slide, please. Okay. So, let's go to the UI and try to
[13:28] create a new rule here and let's select the rule type and try to get the table name. So, by the way, the table name here, they're all pulled directly from the Unity Catalog. So, there's no way for people to type it wrong.
[13:43] Okay, so select the table name. Also, the column. So, we're getting the cost type column here. So, the Yeah, so we should only have the value we're going to select DS partition which means only going to validate one partition at a time. And also, let's put in a lot of values
[14:00] and let's try to save it. Okay. So, if we type the try to search for that rules, that rule should exist. Okay, it's there. Okay. So, next slides, please. Okay. So, let's try to test it.
[14:17] Okay, we could just click one button and that should run. Okay. So, that failed as expected. As we can see here, the column actually has two different values. So, which means we're missing one. So, in this case, R&D is actually a valid value. We want to add it back. Okay, next slides, please.
[14:36] Okay. So, let's try to add the loud value. Um So, I should add a R&D here. Okay, let's type it. Imagine typing it. Okay, let's try to save it. Okay.
[14:52] So, if we go back, then we should be able to see the updated rules and the wrong task and now it should pass. Okay, yeah. So, that's work as expected. Okay. So, next slides, please. Okay. So, as we just showed you like how
[15:09] you can do all rule operation, like how you can create the rules on the UI and how you can update the rules and how can you test the rule within one click. And now we're going to show you a different feature, which which the historical trend of data quality. Um
[15:24] next slide, please. Okay. So, if we go to the historical history tab, it's going to show you the past rate of all the tables owned by individual team. We can also filter by individual table
[15:39] as well. Okay. So, in addition to that, if we go a little bit a little bit slower uh sorry, a little bit lower on the page, you will actually also have individual data quality run can take you to the Spark
[15:54] UI. So, if you want to see logs or more detailed message. So, next slide, please. Okay. And also on each individual data quality rules, we have all the historical run attached to that rule as well.
[16:09] Next slide, please. Yeah. And in addition to that, we can also examine rules by team. So, if we use the top-level filter to filter by team, we can see all the rules owned by individual teams. And in addition to that, we can also filter down further by
[16:26] looking to different type of rules owned by that individual team. Okay. And also the history tab will change as well. So, now we only have rules, tables
[16:42] that's like owned by that one team. Okay, next slide, please. Yeah. Okay, just to recap very quick what we just showed you. So, we show we have in this app, we have very easy rule management. Like I
[16:58] mentioned that you can try to create a new rule, update a rule, and then like very easily just on the UI. And in addition to that, we also have a streamlined testing feature. So, which means everyone they can come into the UI and then click one button to test the
[17:14] rule. And they will get an instant feedback from the UI as well. So, if they happen to like miss anything in the rule configuration, they can update update rules afterward. And in addition to that, we also have um we also have historical trends on the
[17:30] historical tab like like you see it's actually show the aggregated pass rate of rules over the time of the history. So, next slide, please. Okay. So, one thing that we didn't actually really show you on the app, but this is the improvement that we made is the actionable alert. Like I mentioned,
[17:46] usually the rule failed, it will send a Slack message, and people will just see a message saying, "Hey, this rule has failed." But what we are what we try to do here is now the rules the the Slack message will actually include a link of Databricks notebook. So, if someone they
[18:01] can click into notebook, the notebook will have the exact failed rules with pre-filled parameter. So, people can just click the run button on the note on the in a Databricks notebook, and they can see why their rules failed exactly.
[18:16] So, next slide, please. Okay. So, you you might feel like, "Wow, this is a lot of efforts to actually make a UI and then try to make it all of this happen. Is it actually worth the time and the efforts?" Yes. That's the questions we also ask and ourselves
[18:31] every day. Like if you ask anyone about data quality, and they will say, "Yes, this is important." But if you ask, "Would you like to spend 20 minutes every day just to on data quality to validate your data?" They will likely to say, "No." So, that's why we're trying
[18:46] to make data quality like a second nature of all our developer using our platform. So, we want to build our platform like a high like a freeway. Um probably like a not like 405, but like some you know good freeway. Um so, yeah, so we want to make it like a
[19:01] good freeway, so people will actually get on it and use it. And when we make our platform like a freeway and people want to use it, the adoption will just increase automatically. So, next slide, please. Okay. So, what happened since we
[19:17] deployed this app? So, we had we onboarded three new teams to the quality app. In addition to that, we have 400 more new data quality checks getting created.
[19:35] So, 445% in the first month. Is that a big number? Um depends on who you're comparing to. Okay. So, let's look take a look at who showed up at the door when we actually launched this. Um the first being data analyst. So, data analyst going to the app, adding checks on the semantic metrics that they're
[19:50] creating in Unity Catalog. But not even just that, they're actually adding data quality checks at the boundaries of Medallion as well from between silver and gold and bronze and silver as well. So, now as we've always talked about with these types of users, they're shipping data products to their to their customers,
[20:07] right? And we want to certify those data products and ensure for their customers that they can trust those data products. Well, now we can actually do that. We can give them a certification process as we vet the upstream with the qualified data to quality checks that they have added. And now we can actually apply
[20:23] that certification. And then data scientists, right? So, they can now start creating checks on the features they're developing in the training tele- tables that they're creating. And they're doing this right in the lakehouse. It's a Databricks app. They don't have to switch to other apps and they don't have to switch that they don't have to do that context switching
[20:39] to to add their checks. And engineers, not just talk about platform engineers, but also data engineers, software engineers, we are opening the door for any anybody in the organization to add checks for whatever they're doing in the data lake to again build trust in the
[20:54] features, the experiments that they're doing within the data platform. And the one thing that is missing from this slide is the level of data stewardship that we are raising by offering these tools to these users. So, data stewardship shouldn't be a mandate or policy document provided by the data
[21:11] platform team, but rather it's arming the people who are actually creating these data assets with the tools to protect it. And that's what the app does, right? So, we're giving them Databricks features without having to leave the lakehouse and actually create and manage their checks and giving them the visibility
[21:27] needed to require to add additional checks where they are missing. And this is my favorite because the YAML actually never went away. We actually took the previous framework, we took what worked, the check engine, the the YAML entry point, and we
[21:43] actually kept that and we kept it portable as part of the application so that we could still add repo base U rules. So, the app is intended to and serve the 80%, but now we still have that YAML path to serve the 20% and those power users because we still have things like cloud code, we have cursor,
[21:59] we have codex. So, people can still interact with data quality inside of their repos using AI tools, bulk edit data quality rules, and still review them via a pull request in GitHub. Um so, it's still using the same rule registry, it's still using the same data
[22:16] quality engine, but now for these users, they're actually incentivized to still use the the the new framework because now they can also go to the application after they added through their repo and look at their data quality trends over time and also from a UI perspective visually see where they are missing
[22:32] checks. So, why not an off-the-shelf tool? Um it's a crowded space. There's observability and quality platforms like Monte Carlo, there are open source tools like DQ or Great Expectations. And we
[22:47] have continued to evaluate both sides of the spectrum, but I'll go three different points of why we decided to build within the Databricks Databricks platform instead of investing in those other tools. First being flexibility. When you purchase a tool like Monte Carlo, um you are conforming to their check engine or their alerting opinions.
[23:04] If you need to customize the the tool, it's typically a feature request rather than a pull request. And then tools like DQ, an open-source tool, it's a fantastic Spark native check engine, but there's no UI. So, we're still back at the same um, issue where
[23:21] we had low adoption with our previous tool. And then adoption. So, our users live in the Databricks ecosystem. Databricks is continuing to offer new features so that we can customize our platform to our user needs. When we go and purchase off-the-shelf tools or use
[23:37] open-source tools, those already have, you know, predetermined ways on how we can do checks. But now, since we own it, we can customize everything as regards to the checks, how they interact with the checks, and how they consume those to our user needs. Um, and also no new
[23:54] vendor UI, right? No new mental model for our users to learn, no context switching between interacting with Databricks inside of the lakehouse, and then going to different platforms to run checks. And then data governance. So, we're continuing to invest in Unity Catalog, and we want to and we don't want to lose
[24:10] that. Um, so moving data, like data quality across different tools, is not uh, something that we are willing to invest in. Continuing to invest in Unity Catalog, build that observability, um, and that will continue to pay off.
[24:26] So, again, drilling into how it works. Um, I think in the beginning, we had a brief slide on the different components, um, but five different points just to drill in a little bit deeper. So, the first, with Unity Catalog, unit users act on themselves. Interacting with the
[24:41] Databricks app, we can take advantage of on behalf of authentication, we could every single user logged in, we can retain their Unity Catalog permissions. So, that they see the tables that they have access to. They can add the checks to tables that they have access to. And we don't want to lose that, but it's a
[24:58] benefit to us because we don't have to build our own permission model or our own federation engine. We just inherit it. And then Unity Catalog governs both sides. So, we'll get into Lake Base, but when we stand up the Databricks app, we also get a service principal out of the box. And then all we have to do is just
[25:13] grant a role to the Lake Base backend. And then as the app interacts with Lake Base, it mints time short time to live OAuth tokens to the app. And this is all kind of wired behind the scenes almost automatically for us. We didn't have to do a whole lot to set this up. And then
[25:29] Lake Base as this as this shared rule store. So, the app writes the rules into Lake Base, the workflows read them out, run the checks, and then write the results back in. And then we continue to write the same existing primitives that we're all used
[25:44] to within Databricks platforms, Databricks workflows. So, again, as we create rules inside the apps, we deploy a Databricks workflow with table update trigger. So, as data lands in our Delta Lake and updates a specific table, it then runs those checks.
[25:59] And then the loop closing itself. Again, with Lake Base, we have other features like Delta mirroring. So, as data lands into the Lake Base project, we can mirror that into a Delta tables and then expose that to other analytics that we want to take advantage of, even other features like Genie or Zero app um and
[26:17] other features that have been announced throughout the course of this week. And then again, continuing to customize our alerting patterns to our estate holder needs via Slack and Data Dog and the types of alerts that they really need to act on the data quality checks. So,
[26:33] again, our investment in Databricks, Databricks has been the strategic bet. Two big things to call out that help make this happen. The first is declarative automation bundles. Um huge unlock. It's becoming a first-class citizen in the Databricks ecosystem for CI/CD. So, everything that
[26:49] we we seen today, the app, the jobs, lake base projects, those are all defined as dabs. And we now we can give these tools to resources like Trinity to help and go build these data quality apps or other platform tools for us via
[27:05] declarative automation bundles and it simplifies a lot of the things with its YAML nature. I still like YAML. Um and then Unity Catalog. So, our investment in Unity Catalog, the one governance, lineage, and permissions layer, we inherit all the the
[27:20] OAUTH and all those things on on behalf of authentication from Unity Catalog and that um really uh sticks everything together. So, I will again mention the developer experience. You know, obviously all the features in the Databricks ecosystem with apps and lake
[27:36] base help us to stitch these things together, but really the glue is dabs. Um I don't I will try to oversell it a little bit because there's so much unlock here for platform teams like us, right? Since the declarative nature of dabs, we can
[27:51] continue to use existing things like the Databricks AI dev kit or we can build our own skills on top of that to complement it that encapsulates our governance patterns in the way that we work and how we want to build platform for our customers. And so, unlocking with dabs is a huge
[28:08] win for our platform team to realize these things where now building a tool rather than going to open source or purchasing an off-the-shelf tool is no longer such a monumental task for us. And the side effect is that our conventions are now data. So, all the
[28:25] rules, the runs, and the results are all captured in lake base. It's no longer tribal knowledge buried in YAML configs. And we're continuing to build our platform now with the added data quality data, we're building the AI substrate to expand additional features for our
[28:42] users. And that's going into where we're going to take it next. So, now we can start to envision other features now that we have this foundation. We can start to integrate agent bricks. We can create an agent that when a data quality check fails, it will then pick up that check and then
[28:58] it will try to triage itself and still conform to our alerting patterns. We can continue to develop AI functions so that we can scan Unity Catalog tables that are being hit the most, which tables do not have checks or missing certain checks and we can start to suggest those
[29:13] checks inside of the app to our users and they can either approve or deny. And then continuing to, you know, take advantage of other features like Genie, right? So, again, when a check fails, people can prompt against it inside of the Databricks ecosystem and ask what's wrong and continue their investigation
[29:29] there. So, two takeaways from this talk, if you're going to leave with anything, the first being as a small platform team, Tao can own the whole thing, right? We didn't scale our team vertically. We didn't have to coordinate
[29:45] with other engineers in the organization with front end and back end and infrastructure teams. We were able to do this all ourselves using dabs and just the inspiration to improve the platform and serve our stakeholders, right? The apps, the workflows, lake base and Unity Catalog. No cross-functional
[30:01] coordination to ship what we shipped. And with AI tooling, cloud code, cursor and codex, building our own skill plugins to not only complement existing AI skills out there that Databricks offers, but build our own skills and
[30:17] plugins and agents to conform to how we develop, how we build platform and our conventions, it only accelerates our velocity and allows us to take on these types of projects with a lot of confidence in what we're delivering and that we're building it right. And lastly, prioritize the experience
[30:34] and adoption follows. So, again, we're building this ourselves and it allows and with the AI velocity unlocked with all this tooling, um we get to continue to reach out to our customers, find out what they really need, how they want to interact with data quality, how they want to interact with an app.
[30:50] And with that, we get to spend more time with them, gather those requirements, and really prioritize the experience. So, don't skip that.
[31:05] Okay, all right. So, hope So, I know maybe some of you signed up this talk not because you are super into data quality, maybe just because this talk happened to fit in your schedule, or this talk happened to be the same floor as your previous talk, which I did that sometimes. But anyways, I hope you still find some of the content of this talk helpful.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.