Best Practices for Running dbt on Databricks: From SQL Models to Production Pipelines

Summary

This video covers proven best practices for running dbt on Databricks as part of a unified analytics engineering workflow, including how Databricks SQL's price-performance and open-format governance make it the right execution engine for dbt transformation projects.
Lakeflow Connect extends dbt projects to cover the ingestion layer, Lakeflow Jobs provides unified orchestration of dbt and surrounding pipeline steps in one place, and materialized views offer performance optimization over traditional dbt models by precomputing results incrementally.
Unity Catalog provides governance and lineage for dbt projects through tags and attribute-based access control, and the video describes a two-step migration approach—move execution to Databricks SQL first, then progressively adopt native platform capabilities—for teams moving existing dbt projects.

Best Practices for Running dbt on Databricks: From SQL Models to Production Pipelines

Many teams use dbt to standardize transformations, but still struggle with brittle orchestration and inconsistent governance. this video shows how to run dbt on Databricks as part of a unified analytics engineering workflow, from SQL models through production pipelines. We’ll cover proven patterns for building a medallion-style architecture, orchestrating dbt jobs directly in Databricks, and applying governance and observability using Unity Catalog for a faster, more reliable delivery of analytics-ready data at scale.
Talk By: Ramiz Bozai, Solutions Architect, Databricks ; Srilekha Dornadula, Product Manager, Databricks ;

Chapters

FAQs

Why does Databricks SQL work well as the execution engine for dbt?

Databricks SQL provides strong price-performance for transformation workloads and is built on open formats, meaning dbt projects running against it automatically benefit from Unity Catalog governance, lineage tracking, and integration with Lakeflow orchestration. Because it is part of the unified Databricks Data and AI platform, analysts, engineers, and data scientists can all work off the same data without handing off to external systems. This video explains how dbt running on Databricks SQL becomes part of a complete ingest-transform-serve pipeline on a single platform.

How does Lakeflow Jobs improve dbt orchestration?

Lakeflow Jobs allows teams to orchestrate dbt transformations alongside ingestion and serving steps in a single unified pipeline, eliminating the need for external orchestration tools or manual handoffs between systems. This video includes a live demo of creating and deploying dbt jobs directly within Databricks. The approach reduces operational complexity and provides a single monitoring and alerting surface for the entire analytics engineering workflow.

How can dbt projects be extended to include data ingestion?

Lakeflow Connect can be integrated into a dbt project to handle the ingestion layer, bringing raw data into the bronze layer before dbt transformations process it through silver and gold layers. This video shows how extending dbt with Lakeflow Connect enables a complete medallion architecture—ingest, transform, and serve—without leaving the Databricks platform. The result is a single pipeline that analytics engineers can reason about from end to end.

What is the recommended approach for migrating an existing dbt project to Databricks?

This video describes a two-step migration approach: first, move dbt execution to Databricks SQL to immediately gain the performance and governance benefits of the platform; then incrementally adopt native capabilities such as materialized views, Lakeflow orchestration, and Unity Catalog lineage. This staged approach minimizes disruption while progressively unlocking the full value of the Databricks Data and AI platform. Materialized views in particular offer performance optimization over traditional dbt models by precomputing results and refreshing them incrementally.

Full transcript

[00:07] If you're in this room, you already know what DBT is. It has become the standard tool for data transformation. So, we are not going to we not here to convince you why you should use DBT. You perhaps already know it. So, the more important question is where you run DBT and why
[00:20] you run it. A lot of you in this room are already running it on data bricks and this session is about getting the most out of that combination. making your DBT on data bricks fast, cheap and easy to run across your data life cycle.
[00:34] So we will start with why DBT and datab bricks fit so well together and then spend most of our time on actually how to do it well. Sounds like a plan.
[00:48] Quick intros first. This is Shrika and I'm a product manager at data bricks. I'm joined here today by Ramse who is a solutions architect at datab bricks. So here are our goals for the session today. Our goal by the end of the
[01:00] session today is for you to walk away with specific best practices that make DBT on data bricks fast, costefficient, and easy to run. We will go through the whole life cycle and we'll also show you
[01:12] a few demos along the way. All right, so first let's start with where data bricks SQL sits in the platform. It's our data warehouse built
[01:24] on the open lakehouse and it's where most DBT runs and because it's part of one unified platform your DBT projects running against data brick SQL come with everything around it all the goodies your data stays in open formats Unity
[01:38] catalog handles the governance and lineage for you lakeflow runs your orchestration and your all your analytics BI workflows and more importantly your AI use cases can run right on top of it as we go through the
[01:52] session you'll see dbt lean on each one of these across your data life cycle. So let's look at the whole pipeline and where dbt actually fits in. So every
[02:05] pipeline typically does three things right in just ingesting your data transforming it and then serving it. On datab bricks all three can be done on the same platform and with dbt you can
[02:17] ingest transform it through bronze silver and gold layer gold layers and feed it into your BI analytics AI and ML use cases. So this is one platform for your analysts, your engineers, your data
[02:30] scientists all working off of the same data and you can accomplish this pipeline in data bricks without handing off uh over to external systems.
[02:44] Now we will also very shortly show you how you can actually expand your dbt project to also include injection part of your data pipeline and not just transform which most teams I'm sure typically use uh dbt for. So we'll shortly show you how you can do this.
[03:01] So there are primarily two key themes for you today. Um simplify and unify. Everything we are going to show you today will tie back to one of these two. Simplifying with fewer tools, less tuning and less to manage and unifying
[03:13] with one platform for your entire data pipeline. So we'll cover the data life cycle primarily in four parts. Build, where
[03:26] you develop your models and pipelines. Orchestrate, running and deploying your pipelines in production. Optimize, keeping your DBT faster, cheaper, and performant. And lastly, governance,
[03:38] securing your data, and being able to attribute the spend back to your DBT models. But before we get into any of these and deep dive into these sections, let's answer an obvious question.
[03:51] Why even use dbt on data bricks? You can run dbt in a lot of places, right? So why data bricks? Uh glad you asked. So two reasons that really matter. One, it's open. So you're never logged in and
[04:05] the price performance is hard to beat. So let me double click into each of these. All right. So, I'm sure you've been hearing open, open, open everywhere, right? I'm going to kind of repeat the
[04:18] same thing here. So, first, it's open and there are really three parts to it. Your DPT generated data sits in open formats like Delta and iceberg. So, any engine that can read and write in those formats can read your data. Secondly,
[04:32] Unity catalog is open and any external engine can read your tables directly without any copies or exports. And your logic is just standard SQL. You're not looking at a proprietary dialect that you'd have to rewrite if you ever decide
[05:01] And secondly, what about price performance on DBT? I'm sure that's the one you all care about the most. I'll briefly walk walk you through how we actually ran this. Uh unfortunately there is no industrywide standard benchmark for a DVD. So we did the next
[05:15] best thing. We started from TPCDI the closest industry standard ETL benchmark. But the caveat is it is typically built for it actually has uh heavy batch loads and it's pretty light on incremental
[05:28] which is how most DBT projects actually run in production. So we wanted the benchmark workload to be representative of DBT uh models in production. So we modified it for it for dbt converted it into a dbt project with incremental
[05:42] daily batches. So we are going to publish the detail methodology the code and the benchmarks out for you to like look into and let let's take a look at the results on this benchmark. So on this benchmark
[05:55] dbt on data brick SQL has the lowest total cost of ownership compared to the other major cloud data warehouses in the market. And I'm sure you can probably guess who the other three are, but they but they came in at 2x, 3x, and almost
[06:09] 5x the TCO on uh for DBT on data bricks. So it's open, it's very cost efficient to run dbt on data bricks. So now knowing all of that, how can you
[06:21] actually get the most out of dbt on data bricks? I'm going to hand it over to Ramse to take it from here. All right, thanks for your All right, so to actually achieve that simplification and unification process,
[06:33] we need a strategy that's going to cover the entire data life cycle. So we've broken this down into four pillars across the traditional workflow. And we're going to start with the building process.
[06:45] on data bricks. But to truly unlock that power, we have to talk about the friction in the status quo. And so if you look at the reality of our pipelines today, the typical analytics life cycle is completely fragmented. So first the source data typically lives everywhere.
[06:59] Teams spend too much time managing messy connections to different databases and external systems before they can even start modeling the data. And the second piece is having ingestion break the DBT workflow. So we often have to move data into standard tables first which creates
[07:12] brittle pipelines and splits that logic across different tooling. And finally metrics tend to drift. I'm sure everyone's experienced this and experiences this today. So when definitions aren't managed centrally, teams end up rebuilding the same KPIs in
[07:25] different places. So with that in mind, let's walk through some of the solutions we can layer on with DBT to chip away at this. And so one of the biggest pain points we hear about is bringing data from a laundry list of source systems. And
[07:37] historically to handle this, we've wrestled with multiple DBT profiles, projects, and even custom pipelines. But with Lakehouse Federation, you can expand the practical reach of DBT and map that foreign source or database
[07:49] directly into Unity catalog. And suddenly your DBT project has become incredibly simple. So you get all the power of DBT transformations in with that source system without any adapter headaches. And so I can still model that data in DBT, document it, test it, and
[08:03] materialize a trusted output on data bricks. And so one thing I want to mention though is that in certain cases of high volume or concurrency needs, you'll want to explore ingesting data from the foreign catalog in a structured manner in a structured manner with
[08:16] connectors available say in LakeFlow connect. All right, let's talk about ingestion. Um, which can be a serious bottleneck in most modern data platforms. So no secret that standard SQL is great for
[08:29] transforming data, but it really wasn't designed for ingestion. And because of this, we used to split our pipelines. Maybe we built something with pispark for ingestion and then handed it off to DBT for the rest of the transformation process. And what that usually meant was
[08:42] a broken DAG split lineage and several tool sets to manage. And so with the streaming table materialization, you can actually handle complex ingestion directly inside of your DVT models. So under the hood, we're actually using the mechanics of a streaming engine to solve
[08:55] the pain that comes with batch processing. So now your entire medallion architecture ranging from ingestion to gold lives inside of one unified DBT DAG allowing for a complete view and lineage.
[09:12] All right, the last piece is trust. So does anyone here work with teams that define every metric the same exact way? No. Okay, I'd be concerned if you raise your hand. Um
[09:24] catalog semantics, you can actually define your metric logic just once centrally instead of rebuilding it across dashboards, notebooks, and other BI tooling. And so that gives you three things. That gives you reusable definitions, govern trust through UC and
[09:37] richer business context for AI agents like Genie. And so whether someone's looking at a dashboard asking a question using natural language, they're going to be working off of the same govern definition.
[09:51] And at the core of this framework are metric views. your single source of truth for measures, dimensions, and relationships across the entire organization. So, we're going to define the metric view within code using familiar DBT patterns like the ref function and point at the underlying
[10:03] model. From there, the metric logic lives with the rest of the DBT project. So, it's version controlled, reviewable, and deployable through the same exact workflow that you guys are used to using. And as a bonus, this actually feeds into Genieon ontology, which we
[10:16] just unveiled earlier this week, having the ability to add cross-domain entity relationships, giving Genie a full semantic graph now to reason over rather than just isolated metric logic.
[10:32] All right, we're going to jump into the fun part, uh, the live demo. So, let's go over, actually, we have a backstory first. Um, so we have a St. Bites medical center. Emphasis on the name St. Bites. Um it's a health system
[10:46] that just acquired a small specialty clinic. And so the problem is that clinic has allergy data that lives in a legacy Postgress database. And so we're not interested in making additional connections or additional pipelines. And
[10:58] then the operational data for the the actual health system comes from a nightly dump in CSV form. And so the first problem after the acquisition was how do we get access to this data? Right? We've got this nice DVC project
[11:10] and we can't migrate this over. And so again, instead of a separate pipeline, we're going to use Lakehouse Federation to query it in place.
[11:23] All right. So if we go over to UD catalog and you'll notice that the Postgress database has actually been added as a foreign catalog inside of UC. And so it looks just like a native native object inside of data bricks. And so you've got
[11:36] additional details on where it came from. I'll click into the allergy data over here. And what I think is awesome is once you go to the lineage here, it's going to treat this just like it was a native
[11:48] object inside of data bricks. And so as we'll walk through the rest of the dbt project in the later parts of that demo, you'll notice that this started in raw form coming from a completely different source outside of data bricks. We've
[12:00] read that in in our dbt project and it feeds through to the rest of the modeling layers. So you've got your marks layer and then there's going to be additional layers on top of that as well with the metrics and then my favorite part is even tracing back the column of
[12:12] lineage as well. So that's uh that's how they got access to the Postgress data um for the allergy the all important allergy um data. All right now they've got that they also need access to their
[12:25] operational data though the main bread and butter that they've been working off of. So they used to use uh just a scheduled batch job that would reprocess every single line of um information that they'd get. And so what we've done now
[12:37] is built the staging layer for that using streaming tables. And so that's going to incrementally process each new row. And data bricks is going to take care of that. And we're going to express that using dbt.
[12:49] And so if you notice when we go over to our staging layer and if we pull up some table that actually feeds into like a revenue cycle use case you'll notice this is a streaming table and the actual catas or
[13:02] not so much cats in this case uh is actually sent through by dbt so you get the same metadata and you'll actually notice that under the hood for those who are curious it actually triggers a spark
[13:14] declarative pipeline. So data is responsible for managing the ingestion process in here. So figuring out what data is new, what isn't, and incrementally processing that.
[13:26] And inside of our dbt project, we've written the same code that you guys are used to. So a select statement and this can be a from stream read files. What we've done in this case is
[13:38] just kind of wrap it with a macro here. But this is essentially what you're looking for. And so we just dropped in the path for the Unity catalog volume which is super robust in terms of adding different types of data types and given it from here. And so when I do a DBT run
[13:52] on this staging layer, it's actually going to trigger this build. And from there, every subsequent build is going to be taken care of by data bricks. All right, so that's the staging layer.
[14:05] Now the one problem on the serving side was the metrics drift that we talked about. And so we've all been there, right? People have different analyst groups, executives, everybody has a different like connotation of what
[14:18] something is expected to be. And so if you actually notice, we've got this metrics layer schema here related to the revenue cycle use case. This was actually built using that DBT materialization metric view. And so all
[14:31] the metadata is actually carried forward to Unity catalog. You can actually look at all the dimensions and measures that have been built. The tool tip will actually show you the actual expression. You can click into the lineage as well.
[14:44] But my favorite part of this is how you can feed this as a source inside of a genie space or a genie agent. And so what we've done is stood up a genie space for the revenue cycle team.
[14:57] And so when we configured this, we had it talking to the gold layer, so the marts. But if you notice, we actually had the metrics here as well. And what's super powerful about this is you can give it instructions to pull from metric
[15:09] views if they have access to it. And so it'll cut back on that processing time and even give you better answers in that case. And so
[15:21] it's actually going to give us a source that it pulls from. And so it gives me a nice chart. And with Genie, it's going to site its sources. I can pull this up and you'll notice that it's all pulling from the measures that we expressed
[15:33] earlier. So, it's going to figure that out based on the instructions that you gave it. All right. So, we're going to go back and we'll be back with same bites. Um, but that's it for for the demo. All right. Sorry.
[15:50] There's more. Uh, all right. So, now that we've simplified how we build our models and unify our metrics, we have to talk about taking that code into production. Right. Right. So that brings us to our second pillar of orchestrating and deploying at
[16:03] scale. So the reality is is that I've had this happen, right? DBT models run amazing on your local machine and then the friction starts as soon as you start to put into production. You're like, "Oh no, it's the wrong schema. It's the wrong
[16:15] database. Nobody can look at it." Um, and so we usually see teams run into a few common roadblocks because of that. And one of the biggest things is adding extra tooling to just schedule and trigger those DBT runs. And so that adds
[16:28] costs, administrative overhead, and more systems to manage. And another big issue is visibility on that. And so pipelines usually just aren't limited to normal SQL. We all know that we're going to often run DBT with dependencies on a
[16:40] Python job or a file drop somewhere. And when those live in separate tools, debugging can get messy pretty quickly. And so finally, there's a manual setup problem. And so this concept of click ops where you go into a UI and make some
[16:52] changes to a schedule or a trigger super dangerous because as soon as you do that and try to replicate it in some other workspace see a lot of head nodding um it's harder to version and review and then obviously you want to take that over different environments as well. And
[17:05] so the goal here is pretty simple again going back to those main themes bringing orchestration closer to the data layer and managing the whole deployment in a programmatic fashion. And this is where lakeflow jobs comes
[17:18] in. Obviously you guys saw a lot of it. The keynote and other sessions. A common problem that we see is dbt just runs on an island really. Um Python ingestion is in one tool, streaming is somewhere else and then a cron job is just sitting there. Hey, am I ready to go? Like
[17:31] what's going on? And so with lakeflow jobs, dbt becomes part of that same exact workflow. So you can have a data bricks notebook pull in data from an API somewhere and then trigger dbt to build the downstream marts all in one unified
[17:45] DAG. And you can also get the operational pieces built in as well. So you get serverless execution, monitoring, alerts, retries, and built-in dependency management as well. And so today, LegFlow jobs support DBT core and DBT
[17:59] platform as native operators. And we're going to soon be supporting DBT core v2 as well. All right, so we just talked about how coordinating those different systems
[18:12] usually forces you to adopt a totally separate orchestration tool. And we're actually going to help you figure that out and eliminate that overhead entirely. And so coming to private preview in Q2, teams will have access to native external integrations directly in
[18:25] LakeFlow jobs. So instead of jumping between different tooling, you can actually trigger queries in other platforms, execute your DBT models, and fire off a Slack alert upon completion, all within a single unified pipeline.
[18:37] And if you actually want to learn more about how external integrations work, there's a session happening at the Blue Shield Theater at 3:00, uh, modern orchestration for the data platform. And so they'll talk a little bit more about external integrations.
[18:56] All right, so we've solved the code problem with DBT, but what about the versioning issues? So the data bricks infrastructure that's needed to run that code, right? Super important. We can't you can't run dbt without infrastructure. And so that's where declarative automation bundles dabs come
[19:09] into play. So instead of manually configuring the jobs in the UI, you define the lakeflow job serverless settings in the DBT project in YAML and when the codes merge in git dabs will deploy them. So the DBT code and the
[19:22] data bricks infrastructure as one singular package and so teams can now use devs to eliminate that concept of click ops for dbt deployments. So your jobs configurations scheduling infrastructure as you go from dev to CI
[19:35] to prod form.
[19:48] So we built the pipeline, right? We we passed that part, but now we have to figure out who's going to run this and how do we run it exactly? So, St. Bites had a complex airflow DAG. So, they have custom operators for DBT, Postgress, and Slack notification. They've replaced all
[20:01] of this with a single lakeflow job. And so, we'll go over to our workspace and we'll pull up our production ELT job. And
[20:15] just as you know 30,000 foot view, you can see the different tasks associated with this. And so I'm extracting my data, running the DBT project, a couple commands associated with it, and then refreshing a revenue dashboard somewhere
[20:28] downstream. You'll notice that for this particular job, there's the scheduling and information associated with it. We'll get back to that in just a second. And if I go to most recent successful
[20:40] run, my favorite, you know what? Let's let's actually save it for later. So again, you get the same view earlier. And then my favorite view is this kind of Gant chart of the individual tasks in your lakeflow job. And in particular,
[20:53] and you can actually get really fine-tuned with this, the exact steps as part of each task as well. So the timing on that as well. Now, this isn't the easiest to sift through and and Tria is going to talk about governance and
[21:05] monitoring a little later and how you can make more sense of this um within data bricks using the same data. And then for this particular job, you've got obviously the environment that it ran in, how long it took to run, those types of things. Now, I kind of skipped over a
[21:18] little earlier, but the reason why this is grayed out, right, and you actually says it right at the very top, is we've connected this to an automation bundle. And so once I deploy this, and we'll walk through the
[21:30] code for that in just a second, this is going to be like handsoff now. So you don't have to worry about somebody fat fingering like a schedule change or moving anything back and forth. This is like locked in a sense. And where we did
[21:42] that was inside of two YAML files. Pretty simple. And so we started with the data bricks.yaml. Just added a couple variables, placeholders. Um, in
[21:54] my very complex example, I've got the same workspace for both dev and prod. Um, hopefully you guys don't do that in real life. Um, but the actual cookbook or the recipe for the job is in here. And so you'll notice that this actually
[22:07] makes more sense in terms of where we can control the configs for that particular ELT job that you guys just looked at. And so as we do this, you use a data brick CLI. There's a couple different ways for you guys to deploy dabs.
[22:19] This is going to make its way throughout the environments and workspaces that you guys have. So you don't have to worry about drift across different scheduling or configurations.
[22:45] All right. So at this point our code has been built, deployed and safely orchestrated in production. But getting our pipelines to run successfully just isn't the finish line, right? So they actually need to run efficiently as well. And so that brings us to the optimization phase and how we can improve performance while also keeping
[22:58] compute costs under control. I know everybody cares about that. You may not care about it, but the person above you cares about it. So we'll definitely talk about that. So keeping queries fast and cost low usually takes a lot of manual
[23:10] tuning, right? It's it's status quo, right? Do you got to put work in to get some sort of efficiency out? And the first issue that we generally see is wasted compute. And so too often teams rebuild large tables from scratch just to process a small amount of data. And
[23:23] that will drive up runtimes and cloud spend as well. And so the second issue kind of related to it is data layout. And so engineers spend a lot of time choosing partitioning keys, super nitty-gritty things like tuning layouts,
[23:35] but the query patterns are going to change like underneath you. And so when they do, something that you optimized yesterday is is not even relevant today. So let's look at how data bricks can help automate some of these optim optimizations directly from dbt
[23:49] and we'll start with materialized views. So the biggest avoidable cost in dbt kind of alluded to it earlier is probably the full table rebuild. And so to be clear up front though materialized views are not a blanket solution for
[24:02] your entire project. We believe they're often best utilized at the serving layer. And so those expensive aggregations that power dashboards and need to stay fast as data changes. And so the nice part about this is how simple this is in dbt. So by leveraging
[24:15] the materialized view strategy, data bricks will handle the incremental refresh behind the scenes. Very similar form to how it was happening with the streaming table as well. And so this tracks dependencies
[24:27] um updates only what's changed and avoids recomputing the entire table each time. And in our benchmarks, we actually saw a massive 17 times improvement compared to a standard full recomputee.
[24:41] All right, this is my favorite question. So, anyone here love partitioning? Like finding columns to cluster on? No. I saw a hand raised. If you if you like, we can talk later. Like
[24:54] I I didn't mean to put you on the spot. Sorry. Um, so we believe it's time to let data bricks take care of all the data laid out related decisions, problems, you name it, right? So, we don't want you guys to worry about manually picking those partitions. So we
[25:08] had liquid clustering or even you know automatic or manual liquid clustering by specifying the cluster keys and so again going back to earlier like rigid partitioning thing of the past we believe people used to use post hooks to keep those dashboards fast and liquid
[25:20] clustering simplified that so we brought you know more flexible organization of the data under the hood how it's queried without locking you into a specific strategy but with automatic liquid
[25:32] clustering data bricks can choose and adjust that clustering for you so engineers are going to spend less I'm actually picking those partition keys and tuning the layout and then more time building models. Now, the best part about this is that
[25:45] DBT is already managing the entire model life cycle and object creation. So, we can set a global configuration at the project directory, even model level to ensure all new objects are automatically clustered. And so, you've just got a
[25:58] simple config there in the lime green text. Now, one thing I think is important calling out is liquid clustering is fundamentally a read optimization, but there are benefits when building and running DBT operations as well, like using an incremental
[26:11] model. And so, you've got merge statements that run under the hood. So, if you're looking for assistance and migrating objects with legacy partitioning, we're actually developing an easy button um for those migrations. And we're happy to chat more about that
[26:24] offline if you're interested. And now I'm going to pass it back to Shrieka to talk a little bit about governance.
[26:40] Thanks Ramy. So we have built our pipelines. We have orchestrated them and we have made them performant. And there is one last piece left which is governance and I believe which is perhaps the most important one. So let's talk about doing governance with DBT. Governance
[26:54] is where things usually typically fall apart because governance gets treated as an afterthought and it really shows up in two ways. One in your data governance in your access and protection rules. And here is how it usually goes wrong. On
[27:06] most teams, data governance lives outside your DBT projects. Permissions, tags, your column masks get set uh separately disconnected from the code that builds your very data and none of
[27:18] it is versioned. And because it's separate from your DBT models, the governance can silently drift. The other side is cost governance. So let's say when finance at the end of the quarter asks you why what your pipeline is
[27:31] actually costing, you're kind of stuck because a query sitting in your usage data really has no clue what DBT model it came from. So let's try and tackle these two. First, data governance. The fix is to
[27:44] pull governance inside your code where your uh where your inside the code that actually generates your data which is basically inside your DBT project. So right in the model configuration you can say what your data is and who can access
[27:57] it. For example, you can add a classification tag so it's easy to find across the catalog. Let's say the classification tag can be something like labeling it as PII information. You can add a row filter so people can only
[28:10] access their region. You can also grant select to a certain group like analysts. And lastly, if you have a sensitive column like uh SSN, you can obviously add column masks as well. And all of
[28:22] this is just part of the model. When DBT runs, Unity catalog enforces it, enforces all of it for you. The tags become searchable across the catalog and the masks and filters apply on every
[28:34] query. And once your data is tagged, Aback, which is attribute-based access control, takes it even further. You write one tag based policy and it applies to every table or column that is tagged automatically. So you're not
[28:47] really doing it like model by model. And the best part, it's all versioned and reviewed alongside the rest of your DBT code. So you're basically pulling all the governance in into your production pipelines.
[29:01] All right. Now let's talk about everyone's favorite topic, which is cost. How many of you here have watched your bills spike but couldn't quite place your finger on why? I see a lot of heads nodding. So let's say when finance
[29:14] asks like why your pipelines bill has gone up, you're stuck because a query in your system tables has nothing that tells you which DBT model it came from, right? So query tags exactly fix that. Datab bricks automatically stamps every
[29:28] DBT model with a certain set of tags like model name, DBT version and materialization type. On top of it, you can also specify DIY custom tags like team and cost center. So every dollar in
[29:42] your usage can now be tied back. You can also group your usage by models or business domain through custom tags to get to a even granular cost breakdown. you can uh you can see exactly which
[29:55] models are also costing you the most. And if you'd rather not write the query yourself, you can obviously just genie it. So this gives you like a great picture of where your costs are coming from as far as DBT goes.
[30:13] All right, now let's get into the demo. We'll be building of the same premise as uh Ramsey's just mentioned the send byes data center, send byes medical center. So okay, let's say a new analyst joins the
[30:26] company and it's the first week. So they really have no idea of the data landscape just yet. So they go to catalog, click on your prod tables, open up MS and click on one of the M tables,
[30:38] let's say them patients.
[30:50] here like first name, last name which are obviously PII data and these get tagged as PII. And these tags also come with a policy to mask those PII information.
[31:02] Now I'm going to quickly click into the sample data to show how it looks like. So the as you can see the first name and last name here are redacted because we applied PII reduction policy.
[31:14] And let me actually go into the code now to show you that all of this governance just lives inside your DBT project. So here in this piece of the code, all the
[31:26] descriptions that you have specified in your uh DBT project YAML will flow into Unity catalog through DBT versus docs configuration. And if I go into mods.yaml YAML file.
[31:38] This is where you actually specify the descriptions of all your columns which flows into Unity catalog. Now let me go back to the project.yamel file. Here I'll quickly show you that the
[31:52] PII tags along with the policy are also set inside your DBT project configuration. And as you can see here we also specified query tags. Uh we specified custom tags here like domain operations
[32:04] or revenue cycle or clinical etc. Now I'm going to click into dashboards here.
[32:17] Yes. So uh we already preset this dashboards. Obviously we have couple of dashboards that look into the cost. So here the chart on the left has cost by DBT model where you can break down the cost from all of your models and the
[32:29] chart on the right goes even further and breaks it down by your domain and layer. So now you can quickly see, okay, my clinical silver layers are costing me the most. So perhaps that's there's something that I need to look into there. Cool. Uh now let's get back.
[32:50] So that's the full life cycle across build, orchestrate, optimize, and governance. Which leaves us with one very practical question. So if I'm not already running DBT on data bricks, how do I get on to it? Great question. So the good news is it's easier than you
[33:02] think. So migrating dbt is really two steps. One, you basically translate your uh SQL code into datab bricks dialect to account for any minor dialect
[33:14] differences between various data warehouses. And secondly, you just swap out your connection from whatever warehouse you're using to data bricks. So that's it. your whole project uh just carries over your DAGs, your
[33:27] documentation, your metalizations and the rest of the project structure just carries over and you're not looking at a full rebuild here. So during uh data bricks also offers a free migration tool called Lakebridge which handles the
[33:40] migration end to end. During DBT migration though the main thing to watch out for is obviously the reconciliation and the validation. You'll want to make sure that the results coming out of data bricks match uh with your existing
[33:52] warehouse. So you'd want to start with a point of uh with a batch of queries, repoint them, and once the outputs add up, then you'd want to go all in. You can use uh Lakebridge for validation,
[34:04] but you can also rely on a couple of external partners that we have to fasten up this migration process. Well, and plenty of teams have already made the move. 3,000 plus customers already run dbt on datab bricks and the
[34:16] dbt datab bricks adapter sees close to 9 million adapter downloads per month. So we want to leave you with the whole picture in one place. So this is
[34:28] everything we have walked through today across build orchestrate optimize and governance and the highlighted ones in greens are the new ones that we have added since last year. So the space is moving fast and there is more on the way
[34:41] with features like support for dbt core v2 in jobs granular dbt dag where you can actually expand the dbt dag in the jobs UI and rerun specific subsets of it and as Rome mentioned external
[34:53] orchestration where you can read systems beyond data bricks from your lakeflow jobs. As you can see, we are investing pretty heavily in DBT experience on data bricks and you can expect this list to keep growing.
[35:09] Well, that's it. That's it from us and thank you. So if you have any questions, we have I believe couple of minutes and Rams and I are going to hand back us.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.