Sponsored by: Qualytics | Your Agents Are Flying Blind: How to Close the Data Control Gap

Summary

Qualytics CEO Gorkem Sevinc introduces "validate-at-use," a principle where data quality is enforced at the moment an AI agent reads or receives data rather than only after batch ingestion or transformation, closing the control gap between traditional quality gates and where AI actually consumes data.
This video demonstrates a live Databricks lakehouse pipeline where an agent encounters stale-validated Unity Catalog tables and net-new inputs from external APIs, then closes the gap using Qualytics' MCP integration that lets LLMs pull live anomaly context in real time.
The Qualytics platform combines rules-based and ML-based anomaly detection with explainability and auditability, enabling both copilots and autonomous agentic workflows to operate on data that has been validated as fit for purpose at the moment of consumption.

Chapters

FAQs

What is the data control gap and why does it matter for AI agents?

The data control gap is the space between where traditional data quality is enforced—at ingestion, after transformation, at pipeline end—and where AI agents actually consume data, which is often on demand from Unity Catalog tables or raw external API sources that have never passed through a quality gate. This video argues the gap is especially dangerous for autonomous agentic workflows because decisions made on unvalidated data are acted upon before anyone can catch the problem.

What is validate-at-use and how does it differ from traditional data quality approaches?

Validate-at-use means enforcing data quality checks at the moment an agent reads or ingests data rather than at predefined pipeline stages. Traditional data quality runs at fixed checkpoints—after ingestion or before a report—but AI agents query data dynamically and ingest from sources that may bypass those checkpoints entirely. Qualytics embeds quality enforcement into the agent consumption loop so every data access is validated in context.

How does Qualytics integrate with Databricks and AI agents via MCP?

Qualytics exposes its data quality scores, anomaly detections, and rule violations through an MCP interface that LLMs and AI agents can query directly. This video demos an agent calling Qualytics via MCP to retrieve live quality context for a Databricks dataset before using it in reasoning, so the agent knows whether the data it is about to consume has passed quality checks or carries known anomalies.

What types of anomaly detection does Qualytics use in its data quality platform?

Qualytics combines two approaches: explicit rules-based detection, where defined business logic and thresholds trigger alerts when data violates known constraints, and ML-based anomaly detection, where the system learns expected patterns and flags statistically unusual values without requiring pre-defined rules. This video also highlights that Qualytics supports entity resolution and cross-system data quality rules for scenarios where quality must be assessed across multiple source systems.

Full transcript

[00:07] Thank you for coming. We're going to make this fun. I'm I'm not going to go through too many slides. There are just really two slides that I'm going to put on the screen, and then I'm going to show you a product demo because I I I like things that are real. And what I'm
[00:20] going to talk about is obviously data quality, obviously data controls, and how in the AI world, the importance of data quality has become significantly more
[00:33] than what it was in the past. So, let's talk about that. If we were to talk about context without control, to me, what that looks like is I have a traditional DQ product sitting in the
[00:47] middle. I have my sources on the left side. I have my traditional consumption, my my my dashboards, my analytics sitting on the top right there, and I may see a spike in my sales dashboard. I may actually interpret that as
[01:00] that's a good thing, right? Like, oh, they this lead type is performing well, so I'm actually going to spend more money on it. But that relies on a person maybe spotting anomalies, maybe not.
[01:12] Well, in the AI consumption world, it's a little different, right? So, we have copilots. We have AI agents, and a core Let's Let's Let's look at this example. We have a CFO who's looking at CAC to
[01:25] LTV ratio. And traditionally, he would go to a dashboard. He would ask uh his analyst some questions and then add some more analysis. That person may be going to a copilot
[01:38] now and coming up with previously unknown insights. Any of you run into this yet? I have. And do I trust the data that the CFO is looking at and actually
[01:51] making decisions off of? Not really, right? There are that those copilots may be going through your traditional DQ, or more likely, they're going to go connect to the sources. That's scary. That data has never been looked at. It is not
[02:03] checked. We don't have rules running on it. We don't We're not ensuring that it's fit for purpose. And now we have the next level of of autonomy in production as well. Some Some
[02:15] organizations have this in production, some don't, which is agentic workflows, autonomous workflows. And those same signals are being consumed by autonomous workflows. That's extra scary.
[02:27] So, how do we close that gap? I like to call this the control gap. Because we have to control the data that we're basing our decisions off of. And it's too late to check that after the fact, because decisions have
[02:39] already been made. So, I like to think of this model of validated use. It's not that I'm only checking my data quality at predetermined steps,
[02:52] but I'm actually checking it as it's being used. So, I'm I may have a dashboard that is highlighting a quality score. I may have a co-pilot that is pulling context in real time from both a data catalog, for
[03:07] example, and also what are the most recent anomalies? How were they resolved? What was that resolution context? That's really important context. And actually highlighting that to agents so that agents can actually act
[03:20] in a validated fashion. That's the concept that we're going to go after today. That's the demo that we're going to look at. The way we do that at Qualytics, by the way, hi. My name is Gorcom. I'm the founder CEO here at Qualytics.
[03:32] I'm coming from a CDO and CTO background. I've lived and breathed this problem of data quality, and I've been yelled at way too many times in the past, our data is go fix it. And what do I do? I go write a rule to catch
[03:45] that specific anomaly, wait for the next thing to break. Does that sound familiar? To everybody, right? Well, I cannot just replace all of that work with complete autonomy. That does not
[03:58] work. Why? We need human governance. We need to get what's in people's heads for what is this capital LTV ratio was in the past and how do I actually how do I
[04:10] what was the resolution context in the past that lives in people's heads? We need to get that into a system of record. So, the best practice that I have seen is AI augment the data quality where
[04:22] we are still doing rules. We have explainability and auditability of rules. We are generating those rules by looking at your historic data. We're generating
[04:34] metadata over your historic data, understanding shapes and patterns. And we're saying we're inferring using machine learning methods to say these are the
[04:46] technical and some basic basic business data quality rules that hold to your data. Our average customer has 20,000 rules in production with about 95% of those rules
[04:58] being automated. That's the augmented piece, right? I'm giving your your human some superpowers. So, let's talk about those superpowers actually. The 5% that remains, I would argue that's more important. Those are
[05:11] your business rules that your people know the best. The data quality is a business problem. It's not a technical problem, but it's trapped in technical tools.
[05:23] We are built for the business user where business users can come in and define their own rules. Can they can let's say that they have an entity resolution problem. That should be three clicks. It's not a bunch of SQL queries
[05:36] or a Jacquard algorithm that you're going to have to write. Or that person may may need to be able to converse with a system where they can
[05:49] and then we can we take care of putting that into production and maintaining that. Now, that's great. We take all of that but we have to expose those signals to everybody that is consuming it. It's not only humans anymore. It's humans behind
[06:01] co-pilots, humans behind dashboards, and autonomous agents. I have to give the signals and the rules as controls at the same time to all of these consumers treating
[06:14] everyone equally. So, let's see what that looks like in reality. I'm going to I can speak for an hour about our demo, of course, so I'm going to try to keep this short, but I'm going to start with
[06:28] the Databricks TPCH demo. This is a um public data set that we took and this is just my high-level overview. I'm looking at its quality score, what are the quality score details, where
[06:40] actually you can bring your own LLM and integrate it so that we can generate richer context around what are your eight dimensions of data quality what are some of the recommendations,
[06:52] you want to review those anomalies, etc. Let's not pay too much attention to that. Let's go look at how was the sausage made? So, we're going to tell I'm going to tell you a story. Let's go look at my tables here. I we start with profiling your data.
[07:05] When I'm generating your metadata, you may target all of your tables or specific tables or even tag data assets and you may say, you know what, let's go train on the last
[07:17] two years worth of data. I have 20 years worth of data, it's too much. And I want to do statistical sampling because I don't need all billion records to be trained on. I need maybe 10 million or 1 million.
[07:29] And you're going to tell the system how deep to go, how much effort to put into the DQ checks that we're going to do generate. These DQ checks are deterministic. I'm going to underline that. They're deter they're not non-deterministic.
[07:42] So, this is machine learning inference that is happening ensuring that we have confidence in the rules that we're generating for you. The output here let's go look at the this customer
[07:54] table. Everybody has a customer table, right? Let's
[08:06] distribution of the observed values. Here's my entropy and my this unique distinct ratio and min-max length. I didn't have to do anything other than two clicks. I can see how my entropy changed over time. Right? And why did it change? I can
[08:19] start asking those questions. I can see how my profile compared to 5 days ago. These are all meaningful things that we have to keep track of. But then what we did is we actually
[08:31] suggested these DQ rules. Let's go look at I'm not going to look at the not null check. That's as basic as it gets, but you would you would you would crack up that when you have
[08:43] 100,000 fields in your ecosystem and you got to make sure not nothing is ever null, that's a lot of effort that you would have to do manually if you were to do it from scratch. But here's an AI managed check that we generated where market
[08:55] segment's acceptable values, not only did we look at the acceptable values, a subset of those holds true 79% of the time. So, we looked at the normal distribution and looked at
[09:07] your historic data's training set with some recency bias to suggest this specific DQ rule. Now, things are going to get interesting. Let's go But before before we do that, let's go look at some
[09:20] another example. Numeric values. I'm looking at median, kurtosis, skewness, quartiles. All this is auto-generated. You don't have to write any queries. We take this and generate other DQ checks. Uniqueness
[09:32] check, negative check, between check. Here is the acceptable range that we found because we're looking at your min max looking at your couple standard deviations around it. Very highly mathematical approach that we take to
[09:45] this deterministic rule generation. And I can talk your ear off about what kind of DQ checks we generate. Anywhere from here's a referential integrity check where we saw all these customer
[09:58] keys be present in a lookup table that is in a different database. We have seen this relationship because I put the AI effort into Max. Now I'm looking at those cross data store relationships
[10:10] because guess what? Every enterprise has every alphabet soup of all the technologies that are around us here, right? And we have to be able to go across those technologies. Looking at another example, I may be
[10:24] um I may be looking at a referential integrity check. Uh sorry, a um a matches pattern check where I'm looking at here is the regular expression for conformity check that was
[10:38] again auto generated by the system. I don't have to write that regular expression. We're only getting started, guys. I have 9 minutes, so I'm going to try to rush through here, but here are these are some of the basic
[10:50] rules that we generate. A lot of these are this is the 95%. The 5% is complex business logic. I may come in here and say, "You know what? I want to write an entity resolution rule
[11:02] and that entity resolution rule I want it to be on the customer table and I want that distinction field to be customer keys. I want to do composite match thresholds." Everybody has entity resolution
[11:14] problems, right? You I'm guessing you all know what that means, but JP Morgan Chase Bank versus JP Morgan versus Chase Bank all being in the same field, they resolve to the same real world entity. That's a massive problem, right? I have
[11:27] to run seven algorithms to resolve that or I can use a product like this where it's three clicks for a business user to come and define that very easily. There are many examples of that anywhere
[11:40] from your time series anomalies, time series detection to really complex reconciliation checks. Now, that's great. What do we do with that?
[11:53] If I only detect and give you a little alert, I've already failed. That's not enough. So, let's look at a simple anomaly. Again, I can talk your ear off about all the different kinds of anomalies we
[12:06] find, etc., but in this limited time, let's go look at this. This rule has failed. These are the records that have failed. 70 records have failed this, but we are highlighting eight of them in the in the UI because you don't need to look
[12:19] at all 70. Sometimes this is a million anomalies, million records failed that assertion, but you get one anomaly because that's a more important anomaly than others.
[12:31] Now, here's what I'm going to do. I'm actually going to bring an LLM in context. This is bring your own key. You decide which LLM we're using, and I'm going to say, "Tell me more about this anomaly.
[12:44] Is it happening often?" Now, we are using through MCP, we're using your LLM of choice behind the scenes with DQ context, with all the metadata that we have, with all the
[12:56] anomalies that we have, everything that we know in a deterministic fashion, everything the LLM can reason with, and we're telling you, "What is the frequency? What is the trending? Is this happening often? What are your suggested
[13:09] actions that you should do?" This is meaningful signal when you are a DQ analyst, when you're a business analyst who's looking at the business impact of this problem.
[13:22] When I'm resolving anomalies, this is a very very very important context. There are of course other that I can do here. I can tie this anomaly to a workflow that I'm going to run. So, every time that I
[13:37] detect an anomaly that comes from the CDE tag that is above a certain threshold, let's go take all these actions. Let's do a Slack message and we're going to do send this to a specific channel.
[13:50] We're going to create a Jira ticket. After 2 minutes, we're going to or 2 hours, we're going to escalate it. All these remediation workflows that you got to run. Now, we have so far talked about the traditional DQ
[14:02] sense. Now, I'm going to show you something a little bit more interesting. Let's close this up. I'm
[14:14] with my Databricks. Uh this is integrated with my Databricks and it's integrated with Qualitics, all through MCP. Just so that you don't watch me type, I'm going to copy paste here.
[14:26] So, I'm going to query our TPCH Databricks using Databricks MCP. Let's look at the market segment field in the customer database where the market segment is machinery and the account balance is
[14:38] between 2,000 and 5,000. Pretty typical Databricks query that I could run. I'm enabling my people to come and interact with uh with the data, right? This is we're all seeing this in
[14:51] our organizations now and we're we're actually seeing democratizing access to data without needing to have BI in the in the mix, right? Okay, so
[15:04] it's giving me a summary of what it has found. It has found uh found things. It has given me results. Now, what I really want to do is I want to ask Cloud
[15:16] if this data is ready for AI consumption. So, let's review what Qualitics has to say about this data set. Does it have active anomalies? Are those active anomalies important? Is this ready for AI consumption?
[15:31] I am sorry, I need to scroll. And I need to allow. So,
[15:43] quality context. And those quality those quality scores, the integration. It's good data. Let me render this into a clear quality dashboard. It's finding it. It's telling me
[15:56] it's going to give me a report that actually shows that quality score and highlights. Let's see what it decides to do this time. Every time it does something different, non-deterministic, right? But, it's actually pulling context out
[16:10] of my data quality tool in real time with the most recent anomalies and telling me what to do. It's rating is good. It's completeness is 100. Its coverage is 100. It's It's giving me all
[16:23] that context that I need as a user to decide if this is good or good versus not. It's going to give me its analysis if this is ready for AI consumption. Oh, it's telling me that volumetric is
[16:36] crappy. That we should do something about that. And it's going to give me that analysis in a second. Now, here is where things are going to get Oh, again, it's showing me active
[16:49] anomalies. Okay, Collibra, speed it up a little bit. Sorry. This takes a little time, but we're all used to this. But, isn't this incredible that we're putting all of this meaningful context
[17:03] at the hands of the users? So, it gives me a full picture. It gives me its analysis. Is it ready for It's largely trustworthy for structured queries, but I would not call it fully AI-ready.
[17:15] That is an important signal. Now, I'm going to do one more thing. So, I'm going to give it new data. I'm going to go into
[17:28] bringing a CSV file. This could be another MCP. I could be pulling data from somewhere else. And I'm going to say I have new data for this.
[17:41] This is the same as an agent going and reaching in for data and using controls in real time. So, this is what it's Here's what it's going to do. It's going to do a real-time DQ check. It's not waiting for a pipeline to run. It's actually pulling that data into memory.
[17:54] It is evaluating that against all the data quality rules that we have. It's going to try to do this all in memory and it's going to tell me what the result what what what the
[18:07] output is. Are these generating anomalies? Are these good to use versus not? Come on, Claude. Speed up a little bit. >>
[18:19] the the real results. And if it doesn't, I'm going to give it 20 more seconds. Oh, there it goes. It's going to complete it. It's pulling the anomalies.
[18:32] Both records triggered violations. Look at that. Because I wrote anomalies in them. Shocker. But, I put is address format check. We did an address validation check. Apparently, there is one in this customer table. And
[18:45] the crappy data that I put failed that. I mistyped household. That also didn't pass the checks. So, it's actually short worded. Both records should be blocked from loading.
[18:58] That is control. That is not data quality. I'm controlling the data that is flowing into my ecosystem. I'm pulling it in memory. This doesn't mean that I'm not doing the validation that
[19:10] is in predefined steps. It's an end. It's not an or. And that's how you manage data quality at scale with the AI workflows, with what's coming next, with all the way AI work
[19:22] that we are doing in our ecosystems. And that's it, guys. That 20 minutes flew. Thank you so much for listening.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.