Skip to main content

dbt and Unity Catalog: A Well-Architected Framework for Governed AI Apps on Databricks

Summary

  • Verified context: dbt provides tests, documentation, lineage and semantic definitions for explainable AI results.
  • Machine-readable context: reduces wasted tokens and makes results explainable with governed, testable data.
  • dbt MCP server: enables semantic queries and chat with dbt status, tests and lineage for AI applications.

dbt and Unity Catalog: A Well-Architected Framework for Governed AI Apps on Databricks

Watch: dbt and Unity Catalog: A Well-Architected Framework for Governed AI Apps on Databricks
Talk by Rachael Friedman, Solutions Architect, dbt Labs
AI apps on Databricks earn trust when the data underneath them is tested, documented and governed: dbt supplies models with tests, documentation, lineage, contracts and semantic definitions, and that metadata flows into Unity Catalog where any AI application can read it. This video walks through the Databricks Partner Well-Architected Framework for AI apps, why apps end up abandoned when users cannot verify an answer, and how shared machine-readable context makes results explainable while reducing wasted tokens.
The talk explains why AI apps land in an "app graveyard" when users cannot verify answers, and why a shared, machine-readable context makes results explainable and reduces wasted tokens. It shows how dbt model documentation and metadata land in Unity Catalog, and demos a live analytics control center app on Databricks with dbt run status, data quality test results, failed records, lineage, a semantic query builder and a chat interface built on the dbt MCP server.
Sponsored by dbt Labs.

Chapters

FAQs

Why do AI apps end up in the app graveyard?

Users cannot verify AI answers, so they lose trust and abandon the app. This video shows how dbt supplies models with tests, documentation, lineage and contracts, creating a shared source of context that makes results explainable.

How does dbt metadata help AI results?

dbt creates machine-readable context: tests validate data quality, documentation explains what data means, lineage shows where it comes from and semantic definitions help AI answer questions accurately without hallucinating.

Can dbt work with Databricks Unity Catalog?

Yes—dbt model documentation and metadata flow directly into Unity Catalog, where any AI application can read it. This video demos a real analytics control center app using the dbt MCP server for semantic queries and chat.

How does the dbt MCP server help AI applications?

It lets AI agents and chat interfaces query data with semantic understanding, reducing hallucinations and wasted tokens by providing governed, testable data that agents can reason about reliably.

Full transcript

00:00 Introduction: dbt and the Databricks Well-Architected Framework

Hello everyone. It seems we are on air. Can you hear me? Perfectly. So, congratulations. Today we'll talk about DBT and the Databricks partner architecture framework for building AI applications. This is me. I am a solutions architect at DBT. My name is Rachel Friedman. I live in Denver, and before coming to DBT Labs, I worked in data analytics consulting for almost a decade. I was an active practitioner of DBT , as well as a big supporter of it.
And now I'm excited to work on the DBT team, helping other teams achieve the best results with the DBT platform. Before we begin, I'd like to get to know you a little. So, who here is using DBT on Databricks today? Can I see your hands? Okay, very cool. Have any of you already created an AI application on Databricks? Okay, a few people. Perfectly. You are in the right place. So, today I'm going to talk about what a strong data foundation means in this age of artificial intelligence.
We are at a turning point. We no longer provide data only to people. Agents are quickly becoming the primary consumers of our data. Agents have the ability to amplify any weakness in the data base. Therefore, things like vague definitions, inconsistent metrics, or weak governance —AI results will only be as good as the data they are based on. So the real problem we're solving now is building a robust, managed data infrastructure that makes AI results accurate, explainable, and reliable.

01:25 Why AI apps need governed, trusted data

How can we avoid situations where artificial intelligence confidently invents or hallucinates answers? This is what we will look at in this session. We'll walk through how DBT and Databricks create a managed foundation that supports AI applications. Databricks provides best-in- class scalable computing, a modern lakehouse platform, and for DBT, creating robust definitions, data quality, and clear consistency is part of our DNA.
Together, they unlock opportunities for true self-service and AI readiness with better governance, more reliable responses, and a more efficient AI experience. And finally, I will demonstrate a ready-to- use AI application on DBT and Databricks . So you can take this template and use it yourself. I would like to start with a very common question we get asked at DBT: why use DBT with Databricks? Our answer is that Databricks and DBT serve different purposes.

02:39 Why use dbt with Databricks?

So, Databricks is a lakehouse platform. DBT is a contextual layer that makes data suitable for analytics and use in AI. So, DBT does this in a few basic ways. First, it is standardization between teams. Instead of having transformation logic scattered across notebooks, SQL queries, and pipelines, DBT gives you a single framework for building models with tests, lineage tracking, documentation, and version control. This means that a wider range of professionals, from analysts to engineers, can work in Databricks with a common language for transformations without isolation.
Also thanks to scalability. As your organization grows, there is a need to scale transformation logic between projects. Therefore, DBT is great for transforming raw data into reusable business models that teams can build on. And the DBT catalog makes it easy to find out what's available. Our DBT mesh architecture allows teams to securely share trusted data across departments and domains. And finally, it's efficiency.
DBT has optimization features built into the platform to help organizations run only what is needed when building pipelines. This way, teams don't pay for extra work. And ultimately, standardization, scalability, and cost- effectiveness matter even more now when you're building AI solutions. DBT provides a framework for creating and managing transformation logic, transforming it into a manageable and machine-readable context.
Therefore, AI applications can justify their answers, rather than just giving them out. Applying this same approach to applications in Databricks: if you are building AI applications in Databricks, why add DBT? Databricks Apps allows you to incredibly quickly build and deploy AI applications that your stakeholders need. With built-in AI capabilities, you can create, for example, an executive dashboard for management, an application for monitoring data quality, or a financial application with predictive analytics.
There are countless use cases for Databricks, but almost every such application faces the same question: what exactly is the verified and managed version of the data that supports this, and how do I prove it? So, when users of the executive app ask, "Where did that number come from?" Or users of a data quality app say, "Which models don't work and why?" Or the finance team asks, "Is this data current and complete?" These are all different applications, but the requirement is the same—a proven context .

05:40 Why every AI app needs a shared source of context

Without a common foundation, each of these applications can obtain context independently. Some can take objects from notebooks, and some from pipelines with completely different structures. And that's when things can start to go wrong. Indicators are duplicated or become contradictory. Sources can become outdated, and that's when AI starts giving answers that are inconsistent or that can't be explained. And it might seem like it would take two years to track down the correct definition of the metric among all these different sources.
So once the interested party cannot verify the response, trust disappears and the application ends up in the " app graveyard". And this is where DBT comes to the rescue . When all of our transformation logic and metadata are stored in DBT, we get a single, managed layer of context that any application can rely on, with consistent definitions, provenance history, test status, and auditing. And if you use the DBT MCP server, we can present this context in a structured and queryable way for everything we create or use with AI.
So, in this case, all our custom programs tell the same story from the same source of truth, and we can trust what we get. So, digging a little deeper, why is DBT trusted? What makes our metadata so uniquely powerful for this AI use case? DBT is where teams standardize the meaning of their data. DBT applies software development best practices to analytics. This means that transformations are defined as code. Dependencies are explicit, logic is documented, and business rules are tested.
And this creates a reliable level of context for how the data is created and what it means. DBT users have been using this framework for the past decade for transparency of the level of transformation. And now it is this structure that makes DBT an AI-centric infrastructure. Metadata in this format provides agents with the right context up front , in a structured way, so they can retrieve the right metric from the right model with the right definition.
For you, this means less clarification from the AI, less manual verification of the correctness of the answer, and less wasted tokens. Finally, because changes to DBT go through Git, version control, and pull requests, this context is managed by default before any AI tool touches it. So, if you look closely, what does this architecture look like from the inside? On the right we see a typical reference architecture for DBT on Databricks.
DBT is great for building transformations on Databricks. You can use medallion architecture by creating intermediate and target models ( or bronze, silver, and gold) in DBT. And while you're working in DBT, the documentation and metadata of the models we define go into the Unity Catalog, adding rich context that powers data intelligence in Databricks. So, taking this idea further, we based our application on the Databricks architectural framework, as it already covers the key pillars that development teams need.

08:45 How dbt metadata flows into Unity Catalog

This is robust management for applications governed by the Unity Catalog. And also, um, a customized user experience and AI-based acceleration. So Databricks accelerates the creation of AI applications, and DBT provides the necessary management context. Thanks to this, the work of AI becomes transparent and reliable. And together you get AI applications that are user-friendly and scalable. OK. Now I'm going to show you something really cool that we just created at DBT Labs.
This is our first native Databricks application built on DBT. We developed this application using DBT MCP server, LLM from Databricks, um, and Streamlit. The idea behind the application I want to show you is to create an analytics control center . This will be an intelligent self-service application for an organization's internal teams, providing full data visibility from a single interface tailored to their needs. The app displays data quality and provenance, um, catalog information, semantic-level metrics— all in one place, and also uses AI to answer questions in plain language.

10:05 Demo: analytics control center app on Databricks

So, let's move on to the demonstration. I'll switch to this screen. Give me a minute. Um, okay. I have a small note. I didn't trust the conference Wi-Fi, so we'll go through this together. This is our Analytics Management Center application . Um, again, this is test data for an e- commerce site, so there's information here about orders, customers, and suppliers. This application is intended for internal teams to use to view and explore their data assets.
There are a few different tabs here that I'll walk you through. We immediately see the status of the last task execution in DBT. And right below that—the KPIs of all your data quality tests. You can see which tests passed , which failed, where there are warnings and data validity violations. Below we can delve into this information in more detail. You can apply a filter if you want to select a model for detailed analysis, or by tag if your assets are organized that way.
And what can be found in each of these sections. These are specialized views that help your users dig deeper into the results of data quality tests. So, we see the table where the test is performed. We can see what this test is, but some special features we've added to this application allow us to view the SQL code that the test is based on and then describe its meaning in plain language. This makes it easier for users to understand why the test failed.

11:58 Data quality tests and failed records

You can also see the test execution status on the right. We can review the results when we move on to the warnings. Not only that, we also built in a way to run and display failed records and view data that violates our test results in one place without having to switch to another platform. You can imagine that you will continue to improve this further and simply add more custom views to make exploring data quality results even easier.
Now let's move on to the data asset discovery tab . Here you can select a DBT model to review and learn more about it. So, we can choose a model. It will show us the different columns, their data types, and descriptions. We can check the tests that are running for each column and see their status again. You can also look at the data lineage. So you can see the full data lineage from source to outcome for the asset you are looking at.

13:14 Data lineage from source to model

You see the entire path from source to the final analytics-ready result, so you can understand the full structure, materialization types, and any other metadata you want to add. And finally, you can see the underlying SQL code that this model runs on. Next, let's take a look at our query builder tab. This is another example of a custom view for how internal data teams can consume information. There may be people who don't want to interact with their data.

13:51 Query builder on the semantic layer

They might prefer to come here, select a metric from the list, and choose the parameters for grouping. A. So right here, let's look at the average revenue per customer. to the client's country. And maybe that's how they want to explore the data, by clicking the run query button. They can see the results here in a table showing the average revenue per client country. They can also see the compiled SQL code that provides this view.

14:28 Chat with your data using the dbt MCP server

So these are just different ways of exploring and querying at the semantic level. Finally, what if they still want to chat with their data? This is the chat tab with LLM that we're showing here: they can choose a prompt or come down here and ask the same question again, but in a chat format with your data. So, show me the average customer size by country. And now the results will appear in chat format. And here, please.
We got a nice table with all the results and key conclusions . And the most interesting thing here, if we scroll down, is how the calculation is done. This shows exactly what our dbt MCP server did : steps the query down to the semantic level to make the response reliable and verifiable. OK. This is our app. This is just an example of what can be done. You can imagine how this solution can be further improved. Our idea was to have a single interface for every internal data user on the team: from engineers troubleshooting testing bugs to the business asking questions about metrics.
This is built on Databricks, DBT, and MCP Server. The data team supports the DBT project on Databricks, and the application provides metadata and an interface tailored to the needs of users . We are now ready to confidently release our AI applications with Databricks. Yes. Together, Databricks and DBT allow you to build your own AI solutions without losing consistency. DBT improves the development experience in Databricks by defining the meaning of data for all users of the platform.

16:23 Takeaways: governed context for every AI use case

The DBT MCP server provides every application and agent with a common context, so results become manageable, reliable, and transparent. Where to start? If you want to try it, it's very simple. Connect your Databricks app to the DBT MCP server, add context to your DBT project, and start developing. What we showed today is just the tip of the iceberg. With Databricks, you can build a variety of different applications right out of the box.
DBT is also great for building pipeline monitoring tools, data quality checks, provenance analysis, or specialized agent applications. There are many opportunities for deeper exploration here, so I encourage you to experiment. Thank you all. That's all I wanted to say today. Thank you for your attention. If you want, come by stand 430, I'd be happy to chat or answer your questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.