Sponsored By: Quest | From Model to Marketplace: A Better Way to Manage Data End-To-End

Summary

This video presents Quest's end-to-end data management approach that automates the journey from data modeling and semantic enrichment through data quality and cataloging to publishing trusted data products into an internal marketplace within Databricks.
Quest's AI-powered semantic enrichment layer — which generates entity definitions, attributes, and ontologies — is described as reducing LLM token consumption by 3 to 5 times by providing richer context, while making data products more reusable and valuable with every use.
The solution integrates directly with Unity Catalog, writing enriched DDL and metadata back to Databricks so that Genie can immediately leverage the semantically enriched data products for natural language queries and AI workflows.

Chapters

FAQs

What data management gap does Quest aim to close?

Organizations often skip foundational steps — logical data modeling, metadata enrichment, and governance — when building AI applications, leading to poor AI performance in production. Quest brings these steps together in a platform that automates the journey from a well-defined data model to a trusted, self-service data product that business teams can use inside Databricks.

How does Quest's semantic enrichment benefit AI applications on Databricks?

Quest's AI-powered semantic layer generates entity definitions, attribute descriptions, and ontological relationships that provide richer context for LLMs. The video states that this semantic enrichment can reduce LLM token consumption by 3 to 5 times while improving the quality and relevance of AI-generated responses by giving the model better-structured input.

How does Quest integrate with Databricks and Unity Catalog?

Quest writes enriched DDL and metadata directly back to Databricks, making semantically enriched data products immediately available in Unity Catalog. Genie Spaces on the Databricks Data and AI platform can then instantly leverage the enriched metadata for natural language querying without requiring additional configuration steps.

What does the automated data product factory workflow look like?

Quest's platform uses AI to create and enrich data models, then applies data quality rules, sensitivity classifications, and domain routing before publishing the final product to an internal data marketplace. The demo shows a natural-language prompt triggering this automated pipeline, resulting in a governed data product discoverable in the Databricks gold layer with enriched metadata ready for Genie.

Full transcript

[00:08] All right. Hello. This is like our favorite session because it's action packed. >> Can you guys hear us? >> Can you guys hear Okay. All right. All right. Just a couple slides to introduce it, but hello. Um but we want to just
[00:22] set the table basically for the demo. So, uh what we've been doing at Quest is uh bringing all of our point products together because basically customers
[00:35] like JP Morgan Chase, which is our largest modeling company uh customer came to us and said, "Hey, you got to help us out. People are skipping the metadata. They're skipping the logical modeling. They're skipping the governance, and they're using GenAI,
[00:49] and we know when you skip that stuff, you're going to fail. So, you need good rich context. You need good logical information, and there's a space for this. There's a big state space for this right now to really help you prosper
[01:03] with AI. And this is not moving forward. That moved forward. Um I'm not going to go into all these slides. Everybody knows that definitely
[01:16] um there's a there there's a lot of money being spent on GenAI right now. >> Good point to click it towards here. >> Aha. You're so smart.
[01:28] All right. >> But I can hit the button for you. >> Please. All right. So, three things. There's a trust trust gap, so we're not seeing the data that's feeding AI. We're also not
[01:41] having the context in there to really get the right responses from the right prompts, and the models are having a problem when it comes to the execution inside of production. Um they might be working great on that
[01:54] testing area, but the minute you put it into production, you're using production data, and it's not always the same as what you tested. Speed and scale, that's another problem that we're solving for, and then lack of interoperability. So,
[02:07] that's lack of interoperability between these data management tools and these data technologies. You know what? I'm just going to skip by a lot of a bunch of this stuff cuz I
[02:20] just presented it, and I see a lot of people that were in my last um last section. So, we would need both speed and scale. If you don't If you have speed without
[02:32] trust, it's a liability, and trust with speed without speed is a bottleneck, and you need them both. So, what did we do? We brought everything to a platform where we're using AI now to create data
[02:44] models, to extend data models, to create data products, to classify your system, understand the patterns of that information from a data management perspective.
[02:56] Uh we're observing the data on a regular basis. So, when data starts to drift, we'll alert you that, "Hey, it went above the max value. It went below the min values. PI information crept into that data, or just bad data is uh now in
[03:10] your AI model." Trust by design, we've embedded um every time you create a new data product, every time you create a new data set, it's going to get a trust score, and that's based off of these different parameters. How good is the
[03:23] quality? Um the end user ratings, how complete is the governance, the life cycle timing, um popularity, etc. And you guys decide which facets you want to
[03:35] use, and you put what weight you want on it, and then we do the rest and classify it into gold, silver, and bronze. So, you're always getting a trust score. So, when we look at modeling today and
[03:48] cataloging today and data quality today, those are the table space stakes. Those are the things that you should have already going on and on top of that is
[04:00] where we built that that AI. And the first thing that we created was the automated data product factory, which is what you're going to see here in a moment, and then a policy execution engine. And we also want to show you our
[04:12] new context engine that we designed just for Databricks. Um so at the main level the contextual layer,
[04:25] we're doing a really great job of understanding the business and the physical information and enriching that data with all that good business flavor. The ontology layer, we're helping you
[04:38] understand what domains that data needs to be in, providing you with knowledge graphs, and just really understanding it more from a domain perspective. And then that semantic layer is giving you all that
[04:51] great 360° view of your new data product and your data sets. So I'm not going to go too much into that. Where is this helpful? This is helpful
[05:04] when you start to try and understand that data. When you're using your LLM, if you take the example of what is our most profitable customer segment? Well, without a semantic layer,
[05:16] how is it going to understand what do you mean by customer? Is it an internal customer? Is it an external customer? How do you define profit? What domain of data is this in? And what is the actual
[05:29] metrics involved? And without that semantic layer, it's going to take you three to four attempts and probably 5,000 to 15,000 tokens. With a semantic layer, you're going to right away
[05:41] understand what is the metric involved with calculating this, what domain do I need to go after? Can I even trust this data? And it's going after the best data that it can,
[05:54] maybe an average attempt of once. And that's going to decrease your token size down to 500 to 2,000. And all of that adds up. So, that's one of the basis that we
[06:06] created the context engine, which is coming from your Unity catalog and our logical modeling, so that you always get that good rich context. And we're creating and we're manipulating that
[06:19] logical data model right inside of Databricks. And so, it's capturing that context as you're modeling. We're Sherpa you will Sherpa you through where are the design flaws, where do you have some
[06:32] gaps from a logical perspective, where do you need some enrichment around the data, and you're fixing it on the fly. Hi, Savani. Good to see you. And then putting it right into the Databricks
[06:44] environment. Okay, so what is a data product? From our perspective, a data product is a packaged container of internal data, external data. It could have unstructured data associated with
[06:57] it. It could have some metadata reports or some BI reports with it. It's everything you need to solve that end user's challenge. And instead of just giving them a a grouping or an extract of raw data
[07:10] products, we're packaging it up to solve their problem. So, these are four different types of data products that we see from the most basic to the most refined. Uh we see master data products
[07:23] now. We see insight-based data products like churn prediction model, quarterly sales, that's the output is aligned with that particular question. And then the composite data um products are products
[07:37] that are created off of other products and used for AI. So, the key is in the reusability. So, every time you go in to create a new new data product, it's going to say, "Hey, I think we already have one. Why don't you
[07:50] reuse this one?" And you make them better and better. So, before we do the actual demo, this is what you're going to see. So, we're going to do a human prompt around a data product. It's going to go out to your
[08:03] logical model first. So, it's going to go out to your enterprise logical data model and give you a data product back. So, it's just a small entity and attribute diagram of that new data
[08:15] product. Next thing it's going to do is go out and find that data. Where do you have this data? So, it's going to go out to all those physical environments and say, "This is what data you should use for this back." And then it's going to tie the guardrails around that. So, what
[08:28] policies, what sensitivity layers do I need to do to follow every time I use this data product? And just keep making it better and better from there. So, with that, Yacine, let's show
[08:40] >> Uh you guys hear me, right? So, I'll I'll show you guys a live demo. Fingers crossed that my internet still works, right?
[08:57] So, today, let me give you a hint about the story. So, the story is between two personas. So, on one side, you have a business person saying, "I need data. I need data for my use case. Where can I get the data from?" And on the other side of the table, we have our data professional. You can call
[09:10] it a data steward, data analyst, data different roles. And the this data professional saying is like, "I got you. I'll get you your data." So, the story starts with
[09:23] with a prompt. The business user describes what it needs. And it I kept it very simple for the sake of this demo. Hey, I I need some customer 360 profitability,
[09:35] blah blah blah. Like it's a very typical scenario. So, what you want to do is if you are a data professional, how would you react? So, basically when the business user speaks, they don't speak in terms of tables and columns. They don't speak
[09:48] your logical language entities and attributes. They just speak in business terms. So, the first thing what we are doing is we are looking at what data we have to fulfill this request. So, we LLM does a
[10:02] great job of breaking this request into a understandable structure and we translate it to a logical model. In the first step, what you will see here, if it works, uh it's a little bit
[10:15] slow. Guys, sorry. What you will see in the first step is a logical representation of this request. If it's not coming fast enough.
[10:27] >> So, >> Okay. >> We typically hear that it takes a good three uh three to six months to create a solid structured data product.
[10:39] >>
[10:58] I shouldn't rely on this internet, but that's fine. We're getting there. >> There it is.
[11:10] >> So, what it does is it translates this input, the use case, into a data structure. Now, it is not any data structure, it is your data structure. Like based on your models, if you have any logical physical models, or if it's not
[11:24] it's based on your data. And it says, "I can offer you these data sets and these attributes. Will that fulfill your requirement? In this step, the conversation might go
[11:36] back and forth. The business user might say, "Hey, I like it, but can you add the customer region information? And by the way, I need this other information. And by the way, I don't need that information." So, that conversation just
[11:48] by prompting back and forth uh can go. The other thing what we really um pay attention is we call it source attribution. So, whenever we recommend something I
[12:00] mean, imagine you ask ChatGPT a question, ChatGPT answers to your question. And then sometimes you say, "How did you come up with this answer? Like, where did you find this answer, right?" So, this color codes what you see on the screen is indicating that. If it is
[12:13] green, like you are seeing here, that means you already have some logical data models exist for those entities. So, it's just reusing them however your logic um data modelers modeled it.
[12:27] Blue means you don't have a logical model representation. You didn't create a model. Maybe you guys don't use data models at all, but you have some database structures, you have some tables and columns and uh things in your
[12:41] database. So, those are the ones blue. So, based on this recommendation, once the business user says, "All right, this looks like what I need. Can you please now recommend me where
[12:54] can I find this data?" Now we are going into the physical layer. Now we are starting to look for the existing data sets which will fit the bill. And remember, sometimes the data set might one-on-one
[13:08] match the request. Bingo, great. Sometimes you need to aggregate the data sets and do different things. And as you can see, for every entity, it comes back with matching scores. So,
[13:20] what does it mean? Like, how does it know, like, based on what we think this is the best data set for you? We look at six, seven different parameters. We look at the domain match. We say
[13:32] Let's say you are looking for customers employee information, salary information. We say, "Okay, now you are talking about HR domain. Now, I will look at the data sources which are flagged as HR domain. So, it
[13:44] will match your domain." Like domain match is key. The other thing is trust score. Yeah, at a given time you have 10, 15, 20 instances of customer table.
[13:56] Which one is the one you should be using? So, we look at the trust scores which Sue explained. And then, we look at other options, too. So, you might say, "I have a preferred data source. What's my preferred data
[14:08] source?" Say, "I have data bricks at gold layer." If you find the data set there, please use it. Don't use the ones over here. Fine. Or the user might have some regulations around the like guardrails around him. So, saying,
[14:21] "All right, so you are part of this domain. You are not supposed to touch this data over here. So, I limit you to the" So, anyway, it's a long story short. Based on the structure which we
[14:33] described in the previous step, now in this step, we are able to find the matching data sets. And from here on there are couple of caveats. I'm not going to go too much in detail, but as you are going through this
[14:46] uh step, you can switch to synthetic data. Now, synthetic data is the another uh key thing people start to to recognize and started loving it because a lot of the times you are looking at a
[14:59] data set. You are like, "Can you show me some sample data?" Well, I cannot really show you sample data because there is some PII in it. Sorry. But I can generate you if you expand one
[15:11] of them, the top one. I can generate some the top one. Go up. Like here. >> Uh sorry. >> I can generate the synthetic data for
[15:23] you based on all the metadata we know and based on all the profiling results we got. And it is as close to as real data it can be. And most of the time the data scientist or the dashboards BI developers, they can just
[15:36] grab this synthetic data and start building their own you know, systems as they are still requesting and getting access to the data itself, right? So, this is in a nutshell
[15:48] um From here on, once you pick all your entities and data sets and when you are good, what you do is you you hit the publish button. So, the what what this publish button does is
[16:00] it pulls the data model, it pulls the data sets together, it pulls all the metadata, like whatever the ontology semantics information you have available,
[16:12] and then puts together the data set, puts a data product, puts the SLA and the contract in place, and publishes it a marketplace. So, going forward, what happens is whoever comes in and says,
[16:25] "Hey, I need customer 360 data." They can find this data product in the marketplace. It's well curated, it's being monitored for drift analysis, yada yada yada. They can trust the data, they
[16:37] will see a trust score on top. And that's how they will be able to reuse and access the clean governed data. >> 3 minutes. >> 3 minutes. Actually, um >> A little of the other one or no?
[16:49] >> Yeah, um Now that I have 3 minutes. By the way, I mean, these these demos I'm I'm really keeping it very light, but if you guys want to come to us at Quest over there, right over there. The other thing is I want to highlight.
[17:03] Whenever we do this demo, people come back and say, "Yeah, I know what you're saying it, you know, I mean, it is all fun and games if you have nice semantics, right? If if
[17:16] you have enough semantics, I can put a prompt and you can understand what I'm saying and give me the right data. Like, but I don't have anything. I have just raw data. Like, where do I start? Where do I begin, right? So, this other product we released,
[17:31] what it does is two-fold. So, number one, you can point it to your Databricks database schema. You say, "I have this schema here. It's just raw data. I don't know much about it. Tell me what you know."
[17:44] It will take this in. It will infer the relationships. It will break it down to domains. It will recommend you descriptions, definitions, logical names, synonyms.
[17:57] It will create a logical model out of it and a conceptual model along with it, obviously. And it will highlight the most frequently used data sets. So, basically, at the end of this exercise,
[18:09] when you say, "Okay, I have enriched enough." By the way, I'm not going to go in enrichment, but if you guys want to see it, come over. Once you are done with it, um
[18:25] you can just publish it to back to Unity Catalog, to Databricks. And as you can see in this DDL, uh it's a little bit technical notation, but you guys can see everything what we enriched. Synonyms, tags, categories, domains, models are embedded
[18:39] in this DDL. It's writing back to Databricks. So, next time you run a query by Genie, it will be already using this enriched information. So, that's all I wanted to show you guys
[18:52] today. Um thank you very much, guys. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.