From Prompt to Production: Building Governed Data Products on Databricks in Minutes

Summary

Quest Data Studio is a native Databricks App that converts plain-language business intent into a governed, production-ready data product—including Lakebase transactional tables, Delta Silver and Gold layers, Unity Catalog governance artifacts, AI/BI dashboards, and a Genie conversational experience—from a single approved model.
Quest's approach addresses three gaps between raw data and AI-ready data models: Unity Catalog tables lacking semantic context for Genie, manually-designed Delta medallion layers without proper lineage, and Lakebase schemas that require expert database design.
This video includes two live demos showing the full flow from describing a model in plain language to generating a governed Lakebase Postgres schema, deploying it in Databricks, and pushing semantic metadata back to Unity Catalog for Genie to use effectively.

From Prompt to Production: Building Governed Data Products on Databricks in Minutes

Modern data products still require weeks of coordination across business, engineering, governance, analytics, and AI teams. Even in modern Lakehouse environments, transforming business intent into a governed, production-ready data product remains fragmented and manual. In this 20-minute session, we’ll demonstrate Quest Data Studio, a native Databricks App that converts plain-language business intent into a governed data product in minutes. Through a live demonstration, attendees will see how a business user can describe a desired outcome, automatically generate and refine a governed contextual model, and deploy a complete data product within Databricks, including Lakebase transactional tables, Delta Silver and Gold layers, Unity Catalog governance artifacts, AI/BI dashboards, and a Genie conversational experience. All from a single approved model.
Talk By: Adrian Moir, Sr. Product Manager, Quest Software ;

Chapters

FAQs

What is Quest Data Studio and what does it produce?

Quest Data Studio is a native Databricks App that takes a plain-language business description and generates a complete, governed data product—including a Lakebase transactional schema, Delta Bronze/Silver/Gold layers with lineage, Unity Catalog governance artifacts, AI/BI dashboards, and a Genie conversational experience—from a single approved contextual model.

What are the three gaps Quest addresses between raw data and AI-ready models?

Quest identifies three gaps: first, Unity Catalog tables and columns carry no business meaning on their own and need semantic context before Genie can use them effectively; second, Delta medallion architectures require manual design with proper lineage and documentation; and third, Lakebase Postgres schema design requires database expertise that most data teams don't have on hand.

How does Quest's human-in-the-loop approach work?

Before any schema or data product is activated in Databricks, Quest requires a governance review step where a human approves the generated contextual model. This ensures the speed of AI-assisted generation is combined with the discipline and governance required for production data products.

Why does Unity Catalog need semantic context to work well with Genie?

Unity Catalog tables and columns carry no inherent business meaning—they describe data structure, not business context. For Genie to correctly interpret natural language questions and generate accurate SQL, it needs semantic metadata such as business definitions, ownership, classification, and entity relationships, all of which Quest Data Studio adds automatically.

Full transcript

[00:10] All right, good morning, everyone. Welcome. Uh my name is Adrian Moya. I'm a senior product manager at Quest Software. And today, um we're going to be looking at uh turning plain language intents
[00:23] into a governed, production-ready data model on Databricks in minutes, not weeks. We're going to cover um
[00:35] who we are as Quest. We're going to talk about our approach, and then we're going to finish with two live demos. If we have time, it'll be two live demos. So, this is all about discipline and
[00:49] governance, not just speed. We can do things really quickly, but you have to do it in a level of uh with a level of discipline and governance. So, as we're going to be showing you something that is brand new and is not
[01:03] released to the market yet, I have to show you this. So, you all have to read this now. You got it? You ready? You can take photos, read it at your leisure, but please read it. Basically, it's a brief
[01:16] legal note to say things might change from what you see today. Okay, so let's get into this first piece. So, Quest has spent decades helping enterprises model and understand and
[01:29] govern their data. We bring that discipline to the full Databricks ecosystem, lakehouse and lakebase. So, we're going to be doing we're going to be showing you the lakebase stuff
[01:41] today. Three pillars we're going to look at, so there's data modeling, industry-and-domain-aware, data intelligence, semantic context for AI. Really important to get that so you are
[01:54] kind of AI-ready. And governance, when you thinking about lineage, ownership, classification by design, all of those aspects.
[02:07] This is built on modeling intellectual property. It's not improvisation, but it's going to be looking at modernizing the work that still takes modeling teams weeks to
[02:19] do. You can do this in a few minutes now. So, we all know that Databricks is a powerful platform, but there are three distinct gaps that
[02:31] sit between raw data and governed AI ready uh data models and data products. So, we have the Unity Catalog and GenieP. So, tables and columns carry no business meaning.
[02:43] They can't do that alone. So, you have to add all of that in. They're only as strong as the semantic context behind them. So, if you have no context, it doesn't help Genie work out
[02:55] what it needs to find. Delta medallion, trustworthy bronze, silver, gold layers with lineage, documentation, all has to be
[03:09] intact. That's its own discipline in its own in its own right. So, and it's still largely manual. So, that's another problem. Lakehouse, it's new.
[03:22] It brings transactional workloads to the platform. It's a little bit different. The challenge then is going to be helping the business design for that. It's a different skill set entirely.
[03:35] And that's where we're going to help with this. So, we have three challenges. One team that has solved them before, and we're now going to accelerate them
[03:48] on Databricks, is Quest. So, a genetic AI is there to help accelerate things.
[04:00] But when you're looking at governance, you still need the human in the loop. You can't just rely on it to say everything's fine. So, we have to take uh something very simple to start with. So, it's going to
[04:12] start with plain language intent. Now, we all type into chatbots and AI sites and all sorts of things. But the intent that comes back from your request is the really important thing.
[04:27] So, we have to describe the outcome in business terms. So, until you have those business terms, entities, relationships, and rules,
[04:40] no schema work is there to be able to be to be begun. You can't start until you have all of that information. And that intent we get from the prompt that we enter in. So, an an agent is going to draft
[04:53] structured governed model in seconds, grounded in your schema, and your context. And that's the big, big difference. Without that context,
[05:05] you're modeling on thin air. And we then have the governance side of things. So, we have the human in the loop governance. So, stewards can review, refine, and approve your
[05:17] classifications, lineage, ownership can all be applied before the schema is activated.
[05:31] And the agent we use is not improvising. It's guided by that decades of our modeling experience. And we encode all of that as a reusable best practice.
[05:43] That's the difference between a model that's plausible and one that's governed and correct. So, if you understand you went out to an AI site and asked it to create a model, sure, it'll do that for you.
[05:55] Is it usable? Probably not, because it has no context of your business. And that's where we differentiate. We use your business context.
[06:13] So, what we're going to show you is going to be inside a native Databricks app. So, this is going to be running in your Databricks ecosystem. So, your business intent becomes real, becomes a governed transactional schema
[06:27] is what we'll end up creating. So, you describe that intent in plain business language, generate the model with AI. We can then refine it,
[06:39] use the human in the loop to govern it, and then we can deploy it to lake base directly. And we'll do all the branch work for you, as well.
[06:51] It's all automated. So, you don't have a mock-up anymore. You have a real schema.
[07:03] So, the payoff at the end of this is that you get a real Postgres schema in lake base, generated, governed, and ready to evolve through your business.
[07:19] We don't change your data. We add meaning. So, the meaning that we're going to add, Genie and AI and business intelligence stuff, it can depend on that.
[07:31] We can push things back into Unity Catalog, as well. We'll even show you the DDL that we're going to push back in, so you can go and inspect it and find out what it's going to be doing. So, we can enrich what you have in a very surgical way. You can pull the
[07:44] tables out, you can enrich them, and then you can push that enrichment back. And it's going to help Genie find and use your govern data sets a lot easier.
[07:57] And again, we still have the human in the loop. The steward can still approve, review as we have this real really nice kind of cheap triage uh box approach. So, the result from the same Genie questions can vary.
[08:10] Uh and but the more context you add and we can enrich with, the better your answers are going to be. So, the bottom line of this is bronze to gold, governed end-to-end,
[08:24] context for the whole lakehouse that it can rely on, and accelerate Genie in your business. Right. That's enough of me now. I'm going to hop over now.
[08:36] Uh I'm going to introduce Rodney. Well, he can introduce himself as he's a grown-up. Um and he's going to run a demo through for you. >> Good morning, everyone. My name is Rodney Willis. I'm a director of engineering for Quest. Uh I am leading
[08:49] the initiative to bring Hold on. Here we go. I'm leading the the initiative to bring this built-on Databricks app to you.
[09:01] So, what I'm going to show you today is um portions of what we are building. Uh this is still a prototype. It is not ready to be released, but we are working with um some of our friends here um who I see in the audience from Databricks.
[09:14] So, we're Yeah, hey Ashway. We're excited uh to be working with them uh and helping to bring our modeling experience, our AI-first modeling experience into the Databricks marketplace. So, first what I'm going to show you is
[09:26] the lake base side of things. Lake base, what has been built here is very exciting um by Databricks. They've they've built this a branching workflow that is kind of ingenious, honestly. Uh and the way you can switch back and
[09:38] forth and do live testing on your branches, it's it's fantastic. So, what I want to show you first is how we would take this natural language statement, bring it to the model. So, I'm going to come down here to our chatbot and just do a a
[09:51] standard thing. Uh create a model that shows customer profitability for a regional bank.
[10:05] All right. Um we're going to say transactional. And that way it'll it'll send it over to Lake Base. So, it's going to go Oh, no. >> Internet?
[10:20] >> Um you know, I didn't pray to the demo gods first. I'm sorry. >>
[10:33] Are we there? Had a little internet issues with the Wi-Fi here, one or two.
[10:49] Let's see if we can get this to work a second time. >> Uh we will see it uh >> We have an ethernet port that's direct. >> It seems uh
[11:04] I'm hitting a 403. >> Connect you directly through the ethernet port. We have a direct connection. >> Okay. >> That way. >> Okay. >> No. That'll make it better. >> Any network connection? >> Yeah. >> Allowed. >> All right.
[11:16] >> But >> Have a network connection now. >> Huh. >> Yeah, but I'm getting a 403. Which is an off issue.
[11:30] All right. I apologize for this. This was working just a few minutes ago. Um, I'm getting a 403 right now, which obviously indicates uh, trying
[11:51] Yeah. Okay. Um, tell you what, let's switch back to him for a second. >>
[12:03] >> There you go. >> All right. Meanwhile, back at the ranch. So, what we're going to once we get our networking sorted out, which is probably what it is, um,
[12:15] what we're going to show you, hopefully, is our ability to take that initial prompt and then go and create that schema directly. And this will happen before your eyes. It will just kind of start magically
[12:28] creating for you. And that includes not only just the entities, but also the attributes as well. So, that will then allow you to have that schema. And then to that point, you get to the ability to then go through a triage
[12:41] inbox to look at all the things you now need to go and and verify as a steward. So, you get looking at relationships, domains, categories, all the other contextual information that we can add
[12:53] to that as well. And then there's a level of enrichment, too. >> Is yours working? Huh? Oh. Okay. >> So, the level of enrichment that we can then add
[13:06] stays with the schema. >> Um, >> And that's the important thing, staying with the schema. How we doing? >> Give me 1 second.
[13:19] >> All right. We're just going to check again. I need to be careful I don't fall off here. I like to wander when I'm talking. It's not a It's not a good thing, gray on gray. All right.
[13:35] No. Okay. So, give me his. Okay. All right. I apologize for that. We are going to switch the screenshots as the backup.
[14:03] Let's switch to two. Yeah, okay. But, um >> Switch to two. >> Mhm. >> There you go. You're up. All right. So, backup plan. Thank you, Yatin. >> Okay.
[14:16] Yeah, um >> We're going to walk you through some screenshots. >> Great. So, um this is going to go in reverse order, but that's okay. Um so, first what this is showing here, um is an import of an existing schema from lake
[14:29] house. So, data warehouse side. We were going to show lake first, first, but we'll go lake house. So, lake house we can bring in the schema. We read it. Everything that you see in green there is accepted schema. Stuff that we know
[14:41] about that is working in your environment, right? We trust it and that's why it's green because it came from your environment. Everything that you see in yellow, the dash lines, those are AI inferred. So, in this case here, this schema that came in is actually a
[14:54] bronze level schema, but we are inferring relationships for you. We're going to infer primary keys for you to help you understand and structure your your data better. So,
[15:06] that's why it is in yellow. Um next, we are going to go through an enrich an enrichment phase. The enrichment, it will go off, it will uh infer context based on a semantic
[15:20] catalog that you configure. So, you can import your data, you can set up your glossaries and bring that data in. We will use that as grounding for the LLM. We will help infer the the semantics, the synonyms, the descriptions, and then
[15:33] we we propose that for you to review and approve. That would get routed to a steward in that situation. So, they would review and improve uh that routing.
[15:45] This is an example of the semantic catalog. We can bring in industry models uh as reference. So, if you were using FIBO in your financial services or Accord in your insurance, uh you can bring those models in. We will help use
[15:57] that as grounding to make sure that we are aligning with your terminologies that you're you're already using. This is more of the semantic catalog showing vocabularies. We see this with
[16:09] with all of the standardized uh ISOs style vocabularies. You can also though do a re uh a search, have it look at your uh your schema. Anytime we find a field like a status field that should be
[16:23] a a vocabulary, we will suggest to you that you bring that in as vocabulary. It would show up here as suggestion. You can then edit it, fill out the values of what it would be, and then that would be used as part of the
[16:35] semantics as well. So, when you're looking at the schema, the the items that I showed you that were in the dashed yellow lines, um those are proposals. If you want to
[16:47] review those proposal proposals, you can go to the triage inbox. Triage inbox will surface all of those recommendations and it will give you the opportunity to decline them or accept them or edit them right in line. So, you
[17:00] can go into each one of these and and edit. One thing I want to point out in this particular screen, if you look at this, you'll see that those are domains. In this product, we are making domains a first-class citizen, which means
[17:13] everything is gets related back to the domain stewardship. That becomes a boundary. So, the domain itself is sort of the ownership boundary of that data. So, any entity that's attached to that domain has the steward, and that is the
[17:27] steward that has to approve the work related to it. So, we suggest the domains for you, and we give you the opportunity to accept them. All right. When you have that that
[17:39] accepted, enriched, curated schema, you can then push that back into Unity Catalog. It will show you the DDL right here that is going to be executed. You can publish it, you can copy it out, put it into your own CI/CD pipeline,
[17:52] whatever you want to do. But, the thing I want to point out there is that you'll notice the semantics are getting embedded directly into the schema itself. That will help Genie pick that up instead of having to do data data profiling on your data. It will help pick it up and use your terminology
[18:04] right there. You'll get faster answers. Okay. Um So, the Hop.
[18:16] All right. So, this uh did not show the Lake Base. Let me explain to Lake Base uh how this works for you. Um and I'm sorry that we're having the the authentication issue right now. The When you create the
[18:29] schema, so you do the AI propose, it builds the model for you. You can see that same model on the screen. You can curate it, you can enrich it, you can do everything that you want to do. At that point, when you have the known good schema, you can deploy it to Lake
[18:43] Base. But, when you deploy to Lake Base, it goes through the the Databricks branching strategy. It's hooked in automatically. So, when you click deploy, the first thing we do is based on your project, we will create the
[18:55] production branch, the staging branch, the the development branch, and the ephemeral branch. So, we create all of those. We automatically deploy to the ephemeral branch. We give you the option to review that schema in Databricks. So,
[19:08] you can bring it up, you can use the Databricks own tools to to look at that schema, you can see the structure, it's all great. When you are ready, you come back into our tool and you you promote that into the development branch. Now,
[19:20] what's cool is once you're in development, you can, you know, start using it, you can start filling it with data, start doing your testing, everything's great, right? Now, you come back into the tool and you're like, you know what? I need to modify the schema.
[19:32] I need to add a new table, add a new attribute, whatever that happens to be. Perfect, no problem. You add that, you deploy that now into a new ephemeral branch, you can go back into Data Bricks and you can compare the two schemas, the
[19:44] development schema that you had and the new ephemeral schema and you can see the DDL that's going to get pushed into that environment, which is great. This gives you the control. You can now then do testing against the ephemeral branch if
[19:57] you want or you can promote that to development and then, you know, continue to do your testing. You can always come back in here and then promote to the next stage all the way to the through to the production and once you're in production, you can then do your
[20:10] finalization and your clean up of of those ephemeral branches. So, we are unfortunately out of time. I'm sorry we took a little bit extra. If you have a question, our booth is literally right there. Between the time I walk over
[20:22] there and here, I'm going to try to get the situation resolved so I can show you and answer more questions. But if you have questions, please see us over there. Um I would be happy to hear your feedback and take your questions there um because unfortunately we are out of
[20:35] time. But thank you all. I appreciate you coming. >> Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.