Skip to main content

Elevating Data Quality Standards With Databricks DQX

Summary

  • Proactive data quality: DQX is a Python framework on the Databricks Data and AI platform that validates PySpark DataFrames with real-time quality checks at the point of data entry.
  • Batch and streaming validation: Support both batch and streaming data quality—with granular row and column-level insights—to catch quality issues before they impact downstream analytics.
  • Rule inference and governance: DQX infers quality rules automatically and enables teams to define custom validation logic, making data quality a shared responsibility across your organization.

Elevating Data Quality Standards With Databricks DQX

Watch: Elevating Data Quality Standards With Databricks DQX
Talk by Marcin Wojtyczka, Sr. Resident Solutions Architect, Databricks ; Neha Milak, RSA, Databricks
Join us for an introductory session on Databricks DQX, a Python-based framework designed to validate the quality of PySpark DataFrames. Discover how DQX can empower you to proactively tackle data quality challenges, enhance pipeline reliability and make more informed business decisions with confidence. Traditional data quality tools often fall short by providing limited, actionable insights, relying heavily on post-factum monitoring, and being restricted to batch processing. DQX overcomes these limitations by enabling real-time quality checks at the point of data entry, supporting both batch and streaming data validation and delivering granular insights at the row and column level. If you’re seeking a simple yet powerful data quality framework that integrates seamlessly with Databricks, this video is for you.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is Databricks DQX?

DQX is a Python-based data quality framework on the Databricks Data and AI platform that validates PySpark DataFrames with real-time quality checks and granular insights.

How does DQX differ from traditional tools?

DQX enables real-time quality checks at data entry, supports both batch and streaming, provides row/column-level insights, and integrates seamlessly with PySpark workflows.

Can DQX prevent bad data downstream?

Yes, DQX supports quarantining, stopping, or dropping bad data automatically—you can define actions to take when quality rules are violated.

How do we define quality rules?

DQX can infer rules automatically from your data patterns, or you can define custom rules using Python to match your business requirements and data governance policies.

Full transcript

0:00 Intro

Not too long ago, I was working with a manufacturing company. They wanted to do a predictive maintenance. We were collecting tons of data, you know, machine data, sensor data, and at the beginning everything looked okay. But then we run some quality checks. What we found over 30% of that data had serious quality issues. Even worse, they had no clue. But how do you know if your data quality is is good? How do you check for data quality or do you check it at all?
If your answer to any of this question is I'm not sure, you're not alone. That's where the QX comes in. In this session, I will cover why we built the QX, what the QX is, and how you can use it. We'll also have a live demo coming up. So, just forward-looking statement. If you are into sailing, you can probably judge from this picture that I like sailing. I'm a Deuix co-author and contributor. I love software engineering and data engineering. and I'm senior resident solutions architect at datab bricks.
So with that let me hand over to my colleague. Hi everyone. I'm a resident solution architect with data bricks based out of London. I'm a DQX contributor. For work I love architecting, implementing and fine-tuning data platforms with focus on data quality. Uh outside of work I love to travel. That's me and I'll hand you. Hello everyone. I'm a lead specialist solution architect based in Germany. I'm co-author of GQX and I'm working a lot on different topics data engineering, data quality.
Let's first talk why we built the DQX. Really the story of DQX has started four years ago when many customers started to reach us to get the clean data in their projects like reporting or machine learning especially like predictive maintenance. And uh common uh requirement for many of them was that it should support not only batch but also should support the streaming. It should have a different reaction. So like for example we can stop the processing of the data.

2:00 Why we built DQX

We can quarantine the data drop the bad data. We need to apply the checks on the row level or data set level. Also we need to uh provide very detailed explanation why something happened because if you have a table with a thousand of columns and billions of rows you could not figure out why the things uh failed. One common uh requirement was also like to split the responsibilities between the uh data engineering who know how to program the things and data owners who know the uh business logic.
One requirements was also like how we can speed up the development of the data quality checks especially like for example if you have very wide uh tables but four years ago the situation was following we didn't have anything built in on data bricks now of course we have lakehouse monitoring we have dot expectations but that time we didn't have anything also we still have some functionality in GPX that uh doesn't exist in built-in products also we are improving uh that situation in data bricks labs we also have a pro project for data quality but it was in scala and it was not uh really used by uh customers especially for the people who are not so familiar with programming we tried to use open source frameworks like DQ and great expectations but there were some uh big issues like uh relatively a steep uh learning curve, not so good integration with the streaming and also uh it was problems with integration and it didn't give you the detailed explanation why the specific uh check failed and where and this why we built the DPX and we will pass the microphone to the to talk about DPX capabilities.
Thank you. Uh so now that's the why is clear. It's my pleasure to introduce you to the qx. So the qx is a datab bricks uh quality framework. With the qx you can validate data quality of pispark data frames and it's developed under datab bricks labs. So if you don't know datab bricks labs are open source projects that we developed by datab bricks uh in datab bricks by datab bricks engineers and uh we also have wider community contributing in and these projects are to innovate and experiment and most of datab bricks labs projects eventually mature into core datab bricks products.

4:36 What is DQX

So we only released it in January this year and we've already got fantastic feedback from customers and community. We have very good adoption over 300 customers using the QX now close to 1 million downloads per month and customers are using the QX for pabytes of data and we've also been amazed by the number of you know videos and blog posts people are sharing about the QX. So we are very grateful for that and there are many reason to love the QX.
Uh you've got detailed information uh to reduce your troubleshooting times you know from days to minutes and we cover all spark workloads so that you don't have to think which framework to use forward for which uh spark engine. We define quality rules at row level and data set level. So you have a choice. We provide profiling capabilities and ability to generate quality rules and you can define the rules in code or YAML file.
In my opinion, there are two like critical areas in the lakehouse architecture where it's very important to have data quality in place. The first one is processing data from the external systems. Yeah. because this is the data we actually don't know so we cannot trust it. So it's very important that we do quality checks there. So this data typically sits in the bronze layer. So before we process further we want to make sure that the data is sane.
Second use case is making the data available for consumption. So if you have if you make any decision making you want to make sure that the quality is good. So this data usually sits in the goal layer. uh and in the example here we start with profiling. So we have the bronze layer, we profile the data and we generate quality rule candidates. These candidates can be then refined uh by uh by human that have domain knowledge.
Once these checks are ready, you can do quality checking. So the QX will do quality checking and it it will quarantine all the data that hasn't passed the quality checks. If the data can be curated, we can ringest it back and we can monitor the trends using databicks uh dashboard. Now you may ask okay but which framework should I use right because we have now lakehouse monitoring we have DT we have DQX what should I use I would say if you need reactive quality checking that means quality checking of data that has been already processed and written to a table lakehouse monitoring today is a very good choice if you use DT so you have DT pipelines your f first choice probably should be DT expectations but where the QX comes in.
So three three use cases proactive quality monitoring before you write the data out. So as you process the data you want to make sure that quality is good because often it's too late once you write the data out it might be too late so you want to do it proactively. Second use case is to quarantine the bite data so that it doesn't end up in the final system. And third is custom quality checks. The QX is really really good with custom quality checking.
Yeah. So if you have any custom uh requirements for your quality checking, the QX is very good. Now this is a comprehensive summary of most popular frameworks for data quality. Not all. Uh I know it's it's quite horrible to read and uh I know it's hard to comprehend but probably it's a good time to take a picture. Now I won't go into details here but just three things I want to mention. So DeQuix stands out with role level checking uh ability to quarantine data and very good integration with data bricks uh which is a big actually issue with other open source frameworks.

9:40 How DQX works

Now let's see how the qx works. So the qx can perform quality checking on pispark data frames. So we get a data frame as input and it produces two data frames out. So good data frame contains no quality issues and bad data frame of course contain all the issues. So that bad data frame we can then quarantine and issues are reported as additional columns. So you get the original data set with additional columns that will tell you exactly what has happened for every single row you'll get detail log of what has happened and it's very easy to query this as well.
So if an issue occur in that case in column city so the column city is null we run the decuix uh to check the quality and it's going to be reported inside this additional columns. So you get the detail doc there. Now there is a second option. You can produce just one data frame out. Yeah. Without quarantining. So if you just want to do a pass over your data to check quality, you can just return one data frame out. Uh and it's a good option if you don't need quarantining.
Now the QX offers a number of standards built in libraries. Of course, you can very easily build your own library or your own checks, but there are there is a very good set of standard built-in checks. So, we have row level set and what we call data set level checks. So, let me explain what I mean by this. So, row level checks are all kind of checks or rules that operate on individual rows. So for example, if I want to check whether a particular value in a column uh has issues, I would use row level checks.
Data set level checks on the other hand operate on a set of rows. So these are usually aggregate type of checks. Yeah, if you want to check the count uh statistical information, average outlier detection all these kind of uh things will be uh classified as data set level checks. And we are currently just adding this to the QX. So let me show you now how to use the QX. So you can use the QX as a library or tool. The library provides the quality checking ability and you can just install it on a cluster using pip and you are good to go.

11:55 How to use DQX

So very simple using the QX as a tool means installing additional things in the workspace. you do it with datab brick CLI and you will get additional cap capabilities. So things like profiling job, quality checking job uh will be there. Now there are multiple ways to define checks. In this example here we define the checks in the code using the qx classes. Uh so we have two checks the row rule level checks. So uh the first check we we check whether the column two is not not null and we give it criticality error.
We can also give it criticality warning. That means we'll just flag this data but it won't go into uh error uh uh column uh inside the reporting uh columns. Now how you can apply it? So okay we have first date touch first check and then we have have the second check as well. In the second check we check whether the value in column one or values in column one are within the allowed list. Very simple. Now how we apply it?
So we create this DQ engine uh we instantiate the Q engine. We give it a workspace client and then we just apply the check. Very simple. So if you just apply it you get all data out with the reporting columns. So original data set with the reporting column and if you do the split you get the good and the bad data frame. Yeah. So you can quarantine that data. On the right hand side you can see the result uh it will be uh for every row if there are multiple issues they will be reported there.
So checks can also be defined using yum is exactly the same. Uh these are exactly the same checks and the syntax is very very similar. Uh so in this case we check if column one is null and in the second check we verify if column one is in the allowed list again and we can apply the checks. So almost no difference there. Now the checks in YAML can also be stored in a file or delta table. So we provide functionality to load it from a file or data delta table and write it to a file or delta table.
This has just been released last uh last week. So very useful if you want to scale your uh quality checking. With that let me hand over to NA to do the live demo. All right. So as promised that we will do a live demo. I will just do a run all here. Uh as you can see the screen this is a datab bricks workspace and a notebook. Uh here I do have a cluster that got started and I'll do a run all uh saving some time. So yeah as explained that we did saw a lot of issues uh coming from manufacturing industry but that's just an example uh that we we will expand on for this demo purposes.

15:07 Workspace

Uh we have taken an example of a fictional manufacturing company called machinometrics. Uh machinometrics do have machines. Machines that have sensors, sensors that sense the environment like it looks at temperature, it looks at vibration, it looks at compression and take those readings. Uh machines also require maintenance. So the other data set that we are using for this daybook is uh for machine maintenance. The historical maintenance information for machines.
Yeah. Uh so showing this information here. These are the two data sets that I just spoke about. Machine sensor data set. So as you can see there is this sensor ID column, machine ID column, uh reading type, reading value, uh reading time stamp. So machine ID indicates what is the machine ID sensors within that machine. What are the different sensors maintenance uh sorry the sensor type here it is? It could be temperature, it could be vibration, it could be compression.

15:55 Machine Maintenance

There are three different categories that we have taken here. uh at a particular point like on a particular time stamp the reading time stamp what is the value that has been observed by machine is captured under the reading value uh one important column here is the calibration date because I'll show that to you in the data quality rules it's the time when the sensor was calibrated on that machine so it's it's a start point u and then u other columns are also there uh one more important column here for this example is the firmware version it indicates that what is the middle um firmware version number is.

16:45 Firmware Version

So this is a quick example of how my sensor data looks like. As you can see there's a specific format for machine ID. There are three alphabets hyphen then three numbers. Yeah. And sensor type is like temperature, pressure, vibration and more things around it. This is the maintenance records data set. Uh two key fields here are machine ID and maintenance ID. The first two columns uh these are kind of a primary key columns which indicate how the machine maintenance look like.

17:12 Key Fields

Uh machine ID here is another important column because we will showcase that how things when we join the sensor data with maintenance data machine ID becomes the join column. Why is important that uh we are focusing here on machine ID is that again it needs to be in this format MH so two uh three alphabets hyphen three numbers. Uh so quick example of maintenance table data set as well over here uh you would see that the maintenance type is null for the third row.
So uh with these two data sets what machine is trying to do is to do proactive machine maintenance uh which will help them to reduce the downtime of machines and increase the lifespan on machines as well thereby saving a lot of cost. Uh they were able to do all of this their data engineering team was ingesting these two data sets through medallion architecture. uh there is a bronze, silver and gold and then their data scientist team was having a machine learning model which was combining these two data sets on machine ID column and then predicting that are there any outliers are there any situations where temperature is too high that machine will explode and then they will send an indicator or an alert to the maintenance team that please go check there is a problem with the machine.

17:45 Maintenance Table

So they go and repair that before the problem happens. Uh it was all going good uh perfectly working fine until a fine day uh when the machine maintenance team received large number of alerts indicating that a good chunk of machines are going to explode. Uh the temperature has shooten up a lot. So they ran to the uh to the site uh only to see that none of the machine was having any problem. They were cool, calm, doing their job as usual.
No temperature spike was reported. Uh they managed that situation but then they reported it back to IT team. IT team took nearly two weeks of time to find out what the issue was. Uh the DS team looked at the data. Uh there was indeed a uh temperature hike like 60,000 was a temperature reporting. Data engineing team looked at the data. They also saw a lot a lot of spike on the data. Uh no one knew that there was a firmware version upgrade happened that night.
Uh what happened with that is it added additional three zeros in the temperature column uh without a decimal. So that upgrade uh was given by the upgrade team which was outside of the data engineering team. This is a very simple example where there are different teams playing with the data and there is no um way for the teams to know what is the problem. Right? And that's where you need a tool or a framework uh which can ensure that the data entering into your system is actually usable.
Right? 60 became 60,000 40 became 40,000 machines will explode. Right? Uh so uh on this point machine knew that they have an urgent need to find out a solution which can solve their three core problems. One which can proactively identify these kind of quality issues so that they don't enter inside the system. Uh so that they have good data and they can make good decisions. Uh two is the time. Uh remember I said that it team took two weeks of time to find out what the issue was. uh they needed something that can help them reduce this time so that they can figure out these issues sooner.
I'll show how DQX help you over there. And the third and most important one is that it should be extensible enough so that no matter which industry you are into uh think from not just from the manufacturing industry from any industry because this is a DQX framework we are talking about uh it should be extensible enough to any quality rule that you are bringing in right should not be sticking to what we have developed for you what we have concluded as a field engineering experience it should be beyond that so should be extensible enough uh to adapt to any quality role that you want to bring in.
So these are the three primary thing they were looking for. Uh let me show you how DQX helped them. U so yeah as showed to us earlier that there are different ways of installing using and applying the quality rules. I'll show you the first part of it. I'll deploy it as a library. I'll apply it on Spark data frame and I'll define the quality rules as a YAML file. You can do all of that in the second column um as well.

21:08 How DQX Helps

So there were two key points where DQX can definitely help you. the profiler and the quality checking. Uh before I show you further, uh one more situation happened with machinometrics. Uh they thought okay we have the DQX framework all good to go but we don't know the quality rules for our maintenance data. What do we do? Uh should we wait for two more weeks to find out what quality rules to apply? No, you don't have to do that.

21:26 Rule Inference

Uh DQX comes with rule inference capability where it will look at your data and it will generate the quality rules for you. So you do have a start point. you don't have to wait for a long period to find out what is uh what is a good quality rule for you. So let's have a look here. I have installed DQX as a library. Uh this is one of the way that Mar showed to you. You can deploy it as a workspace tool as well. But this is the easiest to get started.

22:05 Installing DQX

Yeah. So library has been installed. I have restarted the environment. These are some of the parameters to indicate uh where are we? So as I said team doesn't know what quality rules they want to apply. What do they what do they do? So here you see um we have DQ profiler and we have DQ generator. Both of these are coming from the DQX platform. It will profile your data and it will generate the quality rules for you. Uh so here we have instantiated DQ engine calling the workspace client.
This is the maintenance data set that I showed you in the start of the session. And after I have instantiated these engine and workspace client, we have applied the profiling. So profile your data to know how does it look like right? Are those integer columns? Are there upper limits, lower limits and more things like that. Then it generate the rules for you. So generator generate those rules based on the profiles. Both of these are done.
Uh let's see are the rules already generated or not? Is it just me saying that yeah everything is looking good. So yeah, these are the checks. I just have printed these for our review. So as you can see, this is the check zero, check one, check two and more checks like that. Uh know that I did not define these rules. These are the inferred rules, right? Uh this is a common structure. This is what uh DQX expect as a YAML file.
So you do see there is a column called maintenance ID. Remember this was a key column uh which should not be null. So you define what column you want to apply check on what function you want to apply what quality rule you want to apply is it a error like is it very uh is it a kind of a rule without which your data doesn't make sense you define it as error if it makes sense but you still want to drag it warn is the other uh way of and then this is an output I'll show you where it is being used but with this we conclude that DQX can profile your data and can generate quality rules for you to get started Now you do have three destinations to save these rules because uh it has done the inference but you do want to do a review right so you can save it as a delta table as m said that we just released this it makes things much easier because most of the people are familiar with tablebased structure to store the information you can store it as a workspace file as well as a local file so there are these three options that we just saw here as you can see I have saved the checks into a workspace file And here I have saved it into a delta table.
Yeah. So both of these are executed. Now it's time to apply these checks. So remember we saved it as a workspace file. Yeah. And then we are loading those checks from the workspace file. You can do the same for the delta table as well. You can read it from the delta table as well. U okay. So this is how the checks were applied. you have applied on the input data the quality checks that got generated and this is the bad data that got generated.
So as you can see I showed you that maintenance type was null and that's why row has landed into a bad data zone. So moving to the next section uh sensor data we just saw that rule inference is possible with DQX they don't have to have your rules already. Now this is another part the sensor data set there they knew that what kind of rules do they want to apply they they know that there is a complete uh one important thing here on the left hand side have given some business terminology that when business comes to you what kind of rules do they ask you right I want my data to be complete before consumer use it I need this to be following the timeliness check I need it to be in in a particular domain uh I want to make sure that the format is standardized we often hear data harmonization problem right so it should be harmonized between the two ecosystem.

25:28 Sensor Data

Uh in this example, I said that they're joining sensor data with maintenance data on the machine ID. So machine ID should be similar between the two ecosystems. Uh that's why a format standardization is required. Uh timeliness here the sensor reading time stamp the time when it is reading the value it should not be in future. It should be either current or uh the past date. So we will validate whether timeliness is matching or not.
And then the correctness check that when the sensor was installed the reading time stamp should be after that it cannot be before that. Uh have a look to the YAML file. How do we define that in a YAML? So this is the YAML. How do you define this extending to what Men showed to us? Uh this is the block that you will see for all the checks I want to apply. Completeness check. As I said machine ID and sensor ID are two important columns.

26:40 YAML

Those cannot be null. So that sits at the top. I say if those are null, my data is not usable. So I say it is a error criticality. Uh what rule I want to apply? Uh there are good number of rules that we have already built for you. Uh which you can just use out of the box. We can extend that further. I'll show that in a minute. Uh this is another beautiful thing that we have added recently is that you don't have to repeat uh the same quality rule for every column you want to apply that rule.
If you have more than you know one column, you just mention all of these here. So it's like a clean YAML file that you are having right you are not repeating in this case I want to apply the same rule to both sensor ID and machine ID that's why I've specified them here. Uh the next check is around filter and rain check. Uh remember I said there are there is a column called reading type it captures temperature vibration and compression.
I want to do a check if temperature is the reading value then it should not be beyond 100. So it should fall between zero and 100. How do we do that? we just use a this additional parameter here called filter. So sensor type is temperature then it should fall into a range. Yeah, this is a good example. You don't have to do explicit filtering and then applying the check. You just specify it here. Uh then I said machine ID should be harmonized between the two data sets.
It should follow a particular standard. So this reax check allows you to validate whether there are three alphabets and then followed by a hyphen by the three digits. You specify it this way. Timeliness check is not in future. My reading time stamp should not be in future. So this function is already available uh into the framework. You can simply plug and play. The last one which is my favorite is a SQL expression. Why it is my favorite is because reax and SQL expression would probably cover 95% of the quality checks you do want to apply.
So you don't have to rely on the checks that we have built for you. You can simply do whatever you want to as a SQL expression. So in this example uh the sensor calibration date uh should not be uh beyond the reading like it should be calibrated or installed first before it starts taking reading right so that is how you specify what is the rule you want to apply and then what is the error level check what is the expression type so with SQL expression you can define any kind of SQL expression that you want to apply as a quality check so with all these rules that we have looked at let's go and read this file uh this is the table that holds the delta table data for sensor.
I have loaded this file uh from workspace file using DQ engine. Uh you could see this is sensor DQ rules here and then we have applied the checks. showed you that you can you do have two possibilities. You can segregate the good data with bad data or you can keep them together. For this example, I have segregated good data with bad data. So I've only used a simple rule a simple method here. apply the checks by metadata and split it split my data frame into good and bad.

29:21 Quarantine DF

Uh this is your input data frame and these are the checks that we just saw. Uh and it is generating these two data frames. Uh let me show you the quarantine DF. So you do see there are 13 rows which are into bad zone. Uh now interesting thing uh it has arrived into bad zone but what is the problem with these two? Uh we promised that we reduce that time that two weeks time to a very small time to find out what the problem is.
Right? So let's scroll to the right. Uh that was your original data frame with these two reporting columns at the end. These gets added to your data set which helps you to query what is the problem. So if I expand on this, it is structured this way. Uh so it says that machine ID is null or empty for this row and this is the first check that got failed. Right? This helps you to know what is the problem just by looking at these reporting columns. uh if I go a bit down for an example this or maybe I want to show you where two checks has been failed.
Let me see. Sorry, it looks like to the end. Okay, I think maybe I just show this nump here. Yeah, there we go. All right. Uh so if you look at row number six, there are two elements that are showing up. Machine ID is null or empty and second is reading time stamp is in future. Yeah, it means that you are able to find out an issue with the row at the column level. So it is not the case that it will only capture your first issue.
It will capture every single issue that you have in the row which is not matching to the quality rule you have defined. Yeah. So you don't have to worry about or writing those complex where conditions to find out where the problem is. If I scroll to the left uh here we see that the machine ID is missing and then the reading time stamp is in way too ahead in the future. It is not today 2026 for the next. Yeah. So what we just saw is that we were able to uh segregate good data with bad data and we were able to see the reporting columns um in terms of error columns and warning columns.
Uh this is the there are two reporting columns out here error and warning. I said two criticality levels. So you can decide whether uh if your data set have lot of errors is it really useful for the consumer team to really use it. If it is warning you can identify the issue and refix it and reingest it. So that is the uh thought that we had uh to have these two reporting columns in here. Okay. Uh so so far loving DQX uh with me uh start using it uh right away on your mobile.
Okay. Not on mobile maybe on the workspace. Okay. So we are not just limiting it here. Uh with SQL expression and re reax capability and all the uh data quality rules that we have already built. If you're still left with a question from a business user, I have a very complex check for you. Do it with DQX. You can still do that. Uh you can bring your own rule or you can build your own rule. We were just arguing whether we say B or we say build your own rule.
Okay. So for this example, I have said that uh from my story the firmware version upgrade that did that blunder of 60 to 60,000 missing the decimal. I want to ensure that uh when the data is coming my firmware version is starting with V. It should not be anything else. Uh so here we are looking at custom rule development. Pretty simple. Uh you just define what rule you want to apply. It's a very simple Python function.

32:48 Custom Rule Development

Uh in that function you define what is the rule looks like. Right here we are saying it should start with V. It should not be anything else. So you define what rule you want to apply and then you import this dqx make condition method which will allow you to pass this expression to dqx and dqx would adopt to it. So this is number one you have defined what check you want to apply then you add it to the yaml file. So this is my all right this one.
Yeah perfect. So everything else is exactly same as I showed you. Um, not really. Let me Okay, one second. That's okay. Okay, I let me try to show that you afterwards. But what will happen with that is you will add an extra block that will tell that what column you want to apply data quality check on. Exactly same structure and you define what naming you want to apply. That's the only change we will do to the YAML. Uh so let's go back.
We have defined the rule. We have defined or configured it into YAML file. Register it and then execute it. Yeah. So we are loading this from the custom YAML file and then executing it. Uh this is the registration part. So let me summarize it this way. Four steps. You define what rule you want to apply. Add it to the YAML file. Register it and use it. That's it. So we added it to YAML. This is how you register the check you want to apply and then you simply apply that into the same function.

34:27 Summary

The how do you apply? You specify what check you want to apply apart from the check that you already mentioned. Uh so we do have a quarantine data frame. I do love segregating my good data with bad data all the time. Uh so if you look at this the data that got generated as a quarantine there is this row which has uh yeah there we go. This is the firmware version which does not start with V. So if I scroll to the right as you can see here there are two issues that it has captured the calibration date is before the reading time stamp and then the second part is the one that we just built for firmware version is not starting with V.
So we do see that we are able to add new checks as well. You are not limited with what we have built. Last but not least uh this is all good all good for developers. what about my business team or what about my uh execs who want to see is the data quality good or not in a visual way. I don't like Python to you know run my code and see what the problem is. So we have built this dashboard for you uh as an output. Uh it's in an early state of maturity but yeah it do capture how does your data set look like and then you can decide whether uh it is good enough for the consumers to consume.
Uh in my opinion if it is showing me 75% of errors maybe it is not usable. I want to stop my pipeline right over there. And then the right hand side it also go to further grain to tell you what is the error out of that 75% which particular field is contributing to that error. You can decide if machine ID is null then it's an orphan record. Lot of orphan records doesn't make a good data. So you want to stop over there.
Uh so with this I conclude the demo. I summarize it this way. We just looked at DQX. We installed it as a library. We were able to do inference. We segregated good data with bad data. Uh we were able to extend it. We were able to bring your own rule. Uh and then we looked at the dashboard that helps you to see whether it's in a good state to go to consumer or not. Uh with that I conclude it and pass it. Okay. Thank you.
So there will be even more reasons to love the QX in the future. Okay. So we've got a long list of features that we want to add such as data set level rules, quality summary, uh outlier detection, automatic curation of curated data and data quality rule configurators so that even business can configure these rules. And if you have any new ideas or you want to discuss any theix topics, feel free to engage with us you know on GitHub or datab bricks community site.

36:58 Future features

We'll be more than happy to hear your feedback and you know thoughts and we are also looking for contributors. So if you want if you are interested in contributing to this project please feel free to visit the site and uh familiarize how to do it. Now some links for reference uh we've got github repository pi package documentation and some YouTube videos. And with that now I just want to take a moment to thank everyone on the team for making this project a reality.
So none of this none of this would be possible uh without you. So thank you. Uh thank you Pratik Ivan Thomas Ggo Jan Dhundar Kartik and Jacine Gagan Kristoff and. With that we are now ready to take questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.