Sponsored by: dbt Labs | Empowering the Enterprise for the Next Era of AI and BI
Summary
- dbt Fusion Engine: dbt Labs introduces a Rust-based SQL compiler that transforms how data pipelines work on Databricks and other platforms.
- Governed self-service: AI enhances developer workflows while SQL comprehension generates detailed metadata, boosting efficiency and ensuring data quality.
- Centralized control: dbt's data control plane provides repeatable, scalable methods for organizations to succeed with the Databricks Data and AI platform.
Sponsored by: dbt Labs | Empowering the Enterprise for the Next Era of AI and BI

Talk by Elias DeFaria, Staff Product Manager, dbt Labs (Sponsor Speaker)
The next era of data transformation has arrived. AI is enhancing developer workflows, enabling downstream teams to collaborate effectively through governed self-service. Additionally, SQL comprehension is producing detailed metadata that boosts developer efficiency while ensuring data quality and cost optimization. Experience this firsthand with dbt’s data control plane, a centralized platform that provides organizations with repeatable, scalable, and governed methods to succeed with Databricks in the modern age.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What is the dbt Fusion Engine and why does it matter?
The Fusion Engine is a Rust-based SQL compiler that dbt acquired through SDF Labs. It enables dbt to compile and execute Databricks SQL dialects locally, bringing powerful data transformation capabilities to dbt Core workflows.
How does dbt improve data quality and developer efficiency?
AI in dbt enhances developer workflows by suggesting optimizations, while SQL comprehension generates detailed metadata. This combination helps teams catch errors early, understand data lineage, and optimize costs automatically.
Can I use dbt with Databricks today?
Yes. dbt's data control plane integrates with the Databricks Data and AI platform, providing centralized policy management, governance, and the ability to manage both batch and streaming workflows at scale.
How does dbt help with governed self-service analytics?
dbt's control plane lets data teams establish policies that downstream analytics teams follow, enabling self-service without sacrificing governance. Automated metadata and quality checks ensure reliable data for all users.
Full transcript
All right, folks. Good morning. How's everyone feeling? All right. Welcome to Data Summit day two and we have who's here to give a quick presentation on DBT side of things and empowering the enterprise for the extra half analytics and AI. Well, I'll let you take up the show. Have a quick round of applause for him. Oh, wait. Ah, there we are. Amplified. Um not the event ingestion library. Okay. Um How's it going, folks?.
Today I'm going to talk about empowering the enterprise for the next era of analytics and AI. Particularly about how we're going to do that with the new DBT Fusion Engine, which is something very close to my heart that I'll tell you about today. Um so, who am I? Uh I'm a staff PM at DBT and I was one of the co-founders of a company called SDF Labs. Um and SDF, we built a lot of stuff over the course of about 3 years, but one of those things was a Rust-based SQL compiler that took uh uh dialects like uh Spark or Databricks SQL and enabled you to compile and even ultimately execute them locally.
And this is a very, very powerful component of um of a data transformation layer, which is why in January um uh DBT acquired our company and that's why I now work in product and I'm standing here in front of you talking about what we built together. Um Cool, this is a forward-looking statement that um I didn't add, but I think I think we're all supposed to read it. Not really sure. I imagine this will be recorded, so you can read it later.
Let's move on. And here's one more thing that I didn't add but that you should also do which is complete your surveys. Um, yeah. I don't know if it's a star-based system but it is. sure that all of you complete the survey here before leaving the room. So, You're all set. So, if it's a star-based system, I'm expecting five or 10 if it's out of one of those numbers. Anyways, let's move on back to what I was talking about.
So, DBT Core, this is a framework that many of you probably use today to author SQL data pipelines, right? Or Python even. They have Python models too. So, rather just generally this is how many of you probably do data transformation your day-to-day jobs. Hands up if you use DBT in your day-to-day jobs. Awesome, probably why you're here. Um, great. So, what did DBT do? Well, about almost 10 years ago now it introduced software best practices to analytics workflows and it did that by creating a new standard for data transformation, right?
Y'all are familiar with the concept of a model, probably. And with refs that sort of stitch together our models to construct a DAG and tell the system what order to run our models in. DBT introduced documentation. Introduced a testing framework for data pipelines like this, right? And it really took off for these reasons. Um, but like I mentioned a second ago, that was almost 10 years ago, right? And the world has changed since.
AI is a thing. Yes, if you're not on LinkedIn, I have news for you and that is that AI is a thing and it's here to stay and a lot of the things we we do with data now go to serve some AI purpose or AI use case. Um, Iceberg came along and we see ourselves slowly inching closer and closer to the commoditization of compute, right? Um, and pluggability and modularity in these systems, right? And you combine that these new sort of innovations in in data with the fact that many of the things that DBT introduced, um, like modularity in SQL, versioning those pipelines as code, a lot of that is now par for the course for these systems, right?
And you have these two sort of conflicting things. Um, and I think it's fair to say we need an update to this framework and to the engine that powers it to meet the data industry um, in this moment, where it is today, right? What does that update look like? I think it looks like something that keeps developers in their flow, um, or even helps them sort of vibe code by integrating natively with AI systems. Um, I think it looks like something that speeds up your inner development loop, right?
So, when you make a change, you need to figure out if that change is valid or not. And today, you have to go run that up against the warehouse and incur a compute cost just to check if a string is valid SQL, right? So, these these aspects of the development framework that I don't believe are are are are where we need them to be for where the industry is going. And that is why DBT acquired my company, SDF. Um, and that was in January.
Um, and now I'm here. And that was cool. Uh, and what we did together, what we did together actually is is is the far more important thing to discuss. And that is we we we built this new engine. We took the technology that we built at SDF, right, a multi-dialect SQL compiler, and we embedded it into into the the engine that runs DBT projects, right? Um, and it looks like that. It looks like an engine in a car, right?
It's putting that thing on like it's like nitrous oxide, right? Um, and uh uh yeah, there's a few things that does that are actually fundamental sort of step function improvements to the developer experience. So, starting on the left, um, we rewrote DBT in Rust. I'm not even kidding. We did that in 5 months. Uh because we we sort of got to this problem um where the ST F stuff was written in Rust, right? And the DBT stuff was written in Python.
And I don't know if you've ever tried to build Python bindings in a Rust system, but it makes it really tough to get right, you know? Because uh if Rust ever has to call out to Python, um it needs to ensure that the right sort of Python version, the virtual environment is set up on the system, all of these things that make it very very difficult for these two types of architectures to communicate with one another. Um and the result of that sort of initial research was, "Hey, you know what we're going to do?
We're going to actually write the DBT part in Rust so that it can natively integrate with this SQL compiler at the core." And what that also did is it made the whole thing way faster, like 30 times faster to parse a project. Um uh much much faster to compile a project. And uh and that's really cool. And speed is cool for speed's sake, like it's just fun to say things are fast and some people buy things because they're faster.
But I think speed also serves a critical purpose in a developer framework, which is to keep you effectively in your flow. Every time you make a change, if the engine has to validate that change you made, right? And that's your inner development loop. I make a change, the engine checks that change, right? In this case I would be editing a SQL or Python model. Um that engine needs to be fast, otherwise you are bottlenecked.
Um the next primitive or sort of key capability we introduced into the engine is deep SQL comprehension, right? Um that is quite deep SQL comprehension. Uh we now take SQL models in DBT, we resolve them down to logical plans, and in the process of doing that we actually validate the query according to the dialect specifications. Um and then we we statically analyze that logical plan to do things like produce column level lineage or automatically propagate metadata, right?
And extract tons of valuable metadata about exactly how your DAG is transforming each column, right? The logical plan lets us lets us understand that a column might be a normal string here, and then it gets aggregated with a count function and results in a number, right? And we can analyze and understand all of that, produce valuable metadata with it, empower developer experience with it. And I'll show this a little bit in a demo later, but I think that's a really really fundamental.
Step up. And just to speak to what sort of DBT does today with this, DBT takes Jinja, as many of you know, and it resolves the Jinja down to raw SQL strings. Then it defers that understanding to the cloud, right? So it actually it it goes, "Great, we produce SQL. I have no idea if it's correct." Now I'm going to go run this up against Databricks' SQL Warehouse, let's say for example, right? And then then Databricks' SQL Warehouse is going to come back and tell us if that's correct.
Now that incurs a compute cost. That is network latency that developer has to wait for, and so I I really believe in this as as a huge level up to the developer experience, and I'll show that in a demo in a little bit. Last thing is state awareness, right? Um so the new engine understands the freshness of the data every time it runs the DAG, which means it will skip reruns of a model if the upstream data selecting from is no longer fresh, right?
And so that means that orchestration becomes as simple as "Hey, I want to run this thing every X minutes, maybe every 15 minutes." And I can trust that every single run it's not going to do unnecessary work. It's only going to run the models that have new data right to process. So this is a very fundamental optimization. Um and a new primitive of the system. Cool. So those are the key capabilities of the new engine. Um and we're straight to the demo time.
I thought there was a slide in between. Okay. And now I'm going to show them to you. Because um I think they're really cool. How many of you work in VS Code today or Cursor when developing with DBT? Nice. Yeah, it's the majority of the audience. Um you may have also heard that we launched a VS Code extension. Right? Um that VS Code extension was built to pair natively with the Fusion engine. And I think it it is the best thing to illustrate what this this engine brings to the table.
So, I'm going to switch over here to this view. I'm going to go full screen on that. Great. How's that? Good? Okay. So, I have a project here called DBT Dugout. Right? And DBT Dugout is just a normal DBT project, um except it's baseball-themed. And uh all of our analysis here are about different aspects of the sort of the team um uh that we're analyzing, our team, right? Imagine I'm like a technical analyst trying to deliver insights to my GM, right?
Um so, I I want to call out a few things to start. DBT F, this is the new engine running the way Even the help output is a little quicker, you know? Um and so, uh why is that the better I want to quick quick TLDR on like Rust? Rust produces compiled binaries, so we can just invoke them as executables in the system, right? A a Python tool typically requires Python on the system to run it. So, I don't need Rust in the system to run this, which means you don't need to to figure out installing Python, pip, a virtual environment, and setting that all up just to get DBT installed anymore.
It's as simple as just downloading this thing and running it, right? Or curling it, as we put on our doc site. Cool. I can parse my project with the new engine. And like I mentioned, it's lightning fast. It's about 200 milliseconds to do that. Um I can compile. Same thing, lightning fast, right? And every operation in DBT depends on a parse, and the large majority depend on a compile, which means everything gets snappier with this.
Um and uh uh I like to run those commands first because it's worth calling out that the Fusion engine is a separately sort of installable thing, right? And the VS Code extension is is also is sort of a separate app, and I can go actually look it up here on the marketplace. Um and there we go. There's our new DBT VS Code extension. Um we actually just Now, this whole thing is in beta, but we did just launch uh uh Databricks support today, right?
No coincidence. Um but it's worth uh it's worth giving it a go after this. Um and trying with I would say our starter project, which I can show you in a second. So, let's explore the VS Code extension um with the Fusion engine. Great. So, I've brought up a model here called staging players. Staging players, um nothing crazy, typical sort of staging model here. Um and I can do a few things. So, first I'm going to preview this model.
Um and nice, there's our preview. I got data back. Cool. Basic things like that. Um but you're probably not impressed by that. You've probably seen that in a lot of IDEs, right? Let's go with something a little more interesting to illustrate the um stuff we understand about the SQL. If you'll see there, I just pulled up the command palette, and I looked up show column lineage. And I'm going to select the column lineage for the full name column.
Let's take a look here at what's going on because I think this really aptly um illustrates how much we now understand about the SQL. So, let's go through each layer. At the bottom here, it says this is the full name column in our staging players model. And it's labeled transformation. And that's because full name, as you can see in the code, is the result of a concatenation of first name, a space, and last name. It's a reasonable transformation.
Um but as we look downstream, models uh uh uh select full name and don't do anything with it. That might be a select star, or it might just be select full name, right? And that we call those pass-throughs because they're not being the the actual value of that column is not being transformed. And then we even know one more layer downstream when full name gets aliased to FN, we know that's a rename, right? This this is all coming from the analysis of that logical plan.
And if y'all are familiar, maybe you've run like explain plan in a database before? Yeah, and it spits out that like sort of weird structure. It's like projection, column, you know? That's the logical plan, and that's what we're we're sort of bringing SQL down to with a compiler, and that's what we're analyzing to produce this valuable metadata. So, all of this is happening locally. If I were not sort of streaming online, I could literally turn off the Wi-Fi and it would still work, you know?
Um, which is quite cool. Um, but I think I think this compiler really shines when we show it in a developer experience, right? So, uh that's not what I wanted to do. I'm going to open up this model called pitcher salary analysis. And pitcher salary analysis is a set of chained CTEs, right? Starting with the salary data one. And this is sort of selecting uh data about our pitchers, as you can see by this filter, um uh in in the years since 2019.
Uh it's also uh aggregating these sort of pitcher totals here. But let's say I want to add one more aggregation in a CTE. Let's say I want to figure out the yearly median salary of our pitchers, right? Like the yearly median salary here. Great. As um let's do select star from salary data, which was our first CTE. Um, but you'll notice something. When I save this file, ah, magic happens. A red squiggly line. I'm not sure I've ever I've ever been so excited to see a red squiggly line in my code.
Um, but I want to call out that we just found a SQL error uh without having to run this up against the warehouse, without incurring a compute cost, without doing anything. We just saved the file and you saw how instantaneously that sort of popped up. Now, this is a relatively basic error. Can anyone tell me what's sort of missing? Comma, yes. I heard that. Line 23 is missing a comma. So, add that comma, save the file, we're back in a good state.
And so, that's the loop we want to power now is that sort of like type, save, instantaneous feedback, right? Um, if I continue transforming this model, I would probably select the year column and then I would start typing maybe median, right? Cuz I wanted to find the yearly median salary. You'll see we start to recommend um functions uh uh uh as I'm typing that, right? And this is just like the sort of tip of the iceberg with the IntelliSense.
It's called IntelliSense or autocomplete that we can power. You can imagine we have like column level autocomplete as you're typing the columns, right, within a from statement when we already have the from set. Um, we have sort of uh uh uh function IntelliSense as we're seeing here. We could have CTE level IntelliSense. Um, so this is just just scratching the surface, but it's quite powerful and it even embeds the descriptions of the functions from the warehouse um being worked with, right?
So, all all of this is dialect specific. Um, so I can come in here and I can actually aggregate that salary. I'll do median salary. When I save this file, we get another issue, um an even deeper problem, actually. Um, and I call it deeper because before we missed that comma, right? And some of you intelligent software engineers could probably figure out how to write like a regex parser to catch that, right? This one you probably couldn't, though.
Um, and this this is um what we call a compiler error, which actually requires us to understand that median is an aggregation function in the dialect we specified. So, that's like the first thing we turn in. Then, we have to understand all columns being projected upstream of that, and we have to figure out if any of those columns are being grouped by, or if they're also present in aggregation functions, right? And that's how we report this error, which is that the year column is present in an aggregation context, and we say either must be aggregated, or it must appear in a group by clause.
That's really cool, right? It's a very, very direct and very straightforward error to now resolve, right? Um I would say this is even better than many of the errors that most databases give back to us when we miss a group by, right? Um and what's great about a good error like this is that it's it works very well hydrating context to a copilot system. And that's why when I press enter there, the recommendation is perfect.
And how could it not be perfect? I mean, if an AI could not figure out that this error message, where it says the year column must appear in a group by, right? Then, uh we've got work to do on the LLM side. Um but it figures it out, and I save the file, and we're good. So, you can you can imagine even that flow, right? Of like having AI generate code for us, and then having the compiler check that immediately. And that's why I'm not sure if you've seen this, but but many reports claim that things like like the languages like TypeScript or like Rust uh produce significantly better and higher quality code with AIs because they've got a compiler in the loop, right?
And I think this makes a ton of sense because AIs are they're bottlenecked by the same things that we are as developers, right? They spit out code, they need to check it, and then they need to iterate on that. And if you have a compiler that rapidly tells you what you need to iterate on, and tells you the error that's present, it's going to produce better output. Um yeah, that's quite cool. This is all powered by um uh an LSP, if any of you're familiar.
Raise your hand if you're familiar with an LSP. Wow, that's awesome. We should talk after. Um an LSP stands for language server protocol. Um it's a protocol introduced in the '90s. It's just a standard, that's all it is. But it's an important standard. It's a standard that powers code editors like this. It powers like when I press a new token, it recommends the next one, right? And it's just like a standard protocol for doing that.
Um in the same way HTTP is a protocol, right? So um the reason why it's important that we conformed to the LSP spec is because this now naturally integrates with all editors and with the AI experiences in those editors. That's why GitHub Copilot works with this. That's why DBT Copilot works with this out of the box, right? Um and in short, it was the right way to build it, you know? Um The next thing I'd like to call out about compilation is that it doesn't just work with one file at a time, right?
It doesn't just look at, hey, did my edits to this file break anything? It actually looks at the entire project context as I make a change. This is really, really powerful. How many of you have had an experience where you make a change only to later figure out that it broke something downstream, right? Like some downstream dependency. Let's raise hands, yes? A dashboard, a model, yes, seeing lots of hands up, yeah. I also developed DBT models earlier in my career and found the same thing, right?
And the way to mitigate this is typically you run that model, run all its downstreams, incur compute costs for all of that execution, only to realize, hey, my change here broke something way down there, you know? Well, now, let's say I bring up the column lineage for the player ID column. He's got some downstream dependencies as we can see. If I alias player ID to something that um breaks these downstreams, you'll notice something.
The minute I save this file, all these turn red. So that's a real-time impact analysis. The minute I make the change, the compiler runs a compile downstream and looks at all downstream models relative to the change that I just made. And that really, really helps us improve quality and confidence in the decisions that we make as developers. Um because the alternative of well, pushing up to production only for that production run to fail is horrible.
And even the alternative of pushing up to a CI system, which many of you probably have, that runs the downstream models, is still long and can take like an hour just to get this feedback, right? So, you can see the larger theme here of speed and why features like this put us on hyperdrive and allow us to make changes with confidence and iterate rapidly. And even better, I don't know if anyone's used So, I I started my career as a JavaScript developer early in college and then I switched to TypeScript at one point.
Did anyone go through that same transition? Yeah, nice. And do you remember how much easier it was to make refactors? To like because you made a change in the project and then everything would light up red and you'd be like, "Oh, I know everything that's wrong with my project because of this change now, you know?" Uh whereas before, at least for me with in the JavaScript dev side of things, I I would like make a change and then I'd I'd have to like go to every page on the website to figure out if it broke anything, right?
If I broke some some shared utility. This is exactly I think what we introduced here. We've introduced some like TypeScript for SQL, right? Is some way to describe it for the for the subset of you that went from web dev to data engineering, but um probably a slim subset. Um so, this this real-time impact analysis also works with macros, right? Um and specifically, all of our LSP features sort of natively integrate with macros.
If I I can see that sort of the split pane here, which I really like to look at. If I click on this macro, it it highlights the SQL that it produces on the right there. This also works for refs, right? So, I know it's quite common, at least for me when I'm developing DBT models, uh to write a ref and then go, "Oh gosh, like what target am I in?" And uh what fully qualified name is that kind of produce? Well, now it's sort of sort of you can see it side by side and the minute I I make this change, like let's say I select a different ref here, game stats, I save that file and it just updates rapidly, right?
Um so that can be very powerful in scenarios where you're using sort of complex macros to construct your statement. Um but if I actually press this to I can well, I can first I can hover it and show you the sort of sequel it produces, also kind of cool. But I can command click it to go to that macro definition and then I can introduce an error here and you'll notice something again when I save this macro. It shows me all the models that use that macro and whether or not they're now their sequel is valid.
The macro produces, right? So imagine this at the scale of thousands of models that all share macros to produce some utility. And let's say I I change that macro, right? And now I know immediately the subset of those models that break as a byproduct of that change. So again, common theme, real-time impact analysis, right? Working macros, working with sequel. Um and again, just working with all the dependencies of the project and checking them.
So I can revert that change, save the file um and we are back in a good state. So you saw me command click to go to that macro. There's lots of other developer ergonomic things, like command click to go to column and that just takes me to the CTE where I was selecting that column from. If I hover on this star, I can even see the full schema that I'm selecting from, right? Um recent feedback that I'm very excited to add is uh if we uh like have like a little utility here to actually change that select star to a you know, to all the columns it's selecting, you know?
Because it's just better practice, but I don't know. Um I was sort of gauging the reaction to that to see if we should build it. Um yes, yes. Okay, okay, okay, okay, okay, we'll we'll build it, we'll build it. Um so command click can also take us to the sort of selection of the star, right? Um and uh all of that just to kind of keep you agile moving through the project. Um if I do other things like in the realm of developer ergonomics like rename this model staging teams to staging team like let's say I'm just analyzing one team.
The minute I press enter the extension prompts me it says I want to refactor changes on your behalf and I say cool. Uh show me what you want to do. Show preview. And it shows me that it wants to make these ref changes for me, right? So I'm sure I know when I was developing DBT often times it felt like refactors like this were daunting and not worth it, right? Even though it might be like the right thing to do for the cleanliness and the readability of the code.
Um we aim to make all of that way easier. We aim to like you know extend things like this you know you can imagine even renaming a column, right? Propagating the rename of that column to all downstream usages of it. All that's super possible when you have the entire project in context of a compiler and available locally in the file system. Right? Um so I can sort of discard that change. I'm going to rename that back.
Cool. Um Yeah. Rename column I talked about. Go to column. Hover on star. Yeah. It's There's a lot that we've built in here. Um but maybe I want to sort of finish it off with a just a build, you know, to show you just how much faster everything is. Nice. Yeah, there it goes. Um And we just built our project. Oh yeah, this was introduced in the most recent build. That's a small thing, but clearly terminal. I got really far in that live demo without anything going wrong.
Um and those were just warnings though, come on. No no errors. Um No big deal. We are in beta. We are in beta admittedly. Um but yes, you can build your project. That was just to sort of show that and again runs the models in order runs the tests as you would expect. Um And uh there is sort of one last thing I can't demo today, but like that build ran all those models, right? And it it it didn't respect if the models had fresh data sort of upstream because that freshness requires sort of like a a Redis state that we offer in DBT Cloud.
And so that's sort of the benefit of now orchestrating in DBT Cloud is that state awareness that respects model freshness that only runs models when the code has changed and I'll talk about that a little more later. Um but it's a cool feature. Oh! I totally forgot one. I knew I was forgetting one. This is like one of the best ones. Okay. So go back to our our developer experience here, right? With this yearly median salary.
Um we are sort of selecting this year column up from salary data, right? We're also selecting the salary column from salary data. Um and we have no dependencies on other CTEs. I want to call that out. Like we don't depend on the pitcher total CTE or any other refs. Um Now how many of you today when you want to just see the output of this CTE? Like just this part. How many of you what? Like comment out the rest of the file and then select just this this one, right?
Um well now the utility is quite simple. You just press preview CTE right there. And it shows you that output. So it's kind of like a Jupiter notebook. Yeah, yeah, yeah, yeah, yeah, yeah. It's cool. I I knew I shouldn't have forgotten that one. I was like that's the best one. Um and it's kind of like a Jupiter notebook or a SQL debugger, right? Where where where it's it I love this feature because it actually it sort of pairs with DBT style guide of like using these chain CTEs to create like sort of modular structure in the models.
Um and it it helps us just like preview the state at each layer, right? Like sort of a debugger the state of the data. Um so that is quite a that's quite a cool one. And I'm so glad I didn't I didn't miss it. 9:40 Cool. I got 10 minutes? 10 minutes? Okay, cool. Let's go back to the Let's go back to the Oh, okay. I'm just going to speed run the deck. Let's do it. Uh okay, so that was awesome. We showed you a demo. Um the DBT Vision is in public beta.
You can install it by a curl script and give it a go on your project. Um uh it admittedly we are in beta and certain projects that are more complex may not work right off the cuff. That's okay. We're moving really fast. Um and I would say about like 80% of net DBT features are supported, but you might run into one some niche feature like custom materializations or microbatch models that might not be there yet, but that we're working on very quickly.
Um and I hope what I demoed probably shows and proves the orders of magnitude faster developer experience. I think it does. Um like I mentioned, we can optimize the compute costs with avoiding unnecessary model runs. And coming later this year, we can improve data governance. Um and this is a really important point that I like to harp on, um which is that earlier I mentioned our understanding of column lineage and you and you saw how much we understood about each layer of the DAG, right?
You saw how much we understood This was the transformation. Um and we actually we only we render transformation there, but we actually know the exact function that's being called at that layer, too. And I think we should just change it to show the function. But um the reason this is important for data governance is that imagine we're we're propagating data classifiers with our column lineage. So Unity Catalog does this today, right?
It like based on the column lineage, it can propagate the classifiers through each layer. Um um but what's powerful is to actually enable metadata to react to the transformations at each layer. So if I label sensitive data PII and it's like an email column, and then I count that email column downstream, I produce an aggregation. I just say like count of emails and it's like a number. That's no longer PII, right? And we can enable you to to write rules to tell us how you want us to propagate at each layer.
So if like a the PII column the tag reaches a layer where it gets counted, we can reclassify that to PII aggregated or we can just remove it all together, right? So, you can create a full sort of automated data classification suite to power governance workflows, things like that. Very excited about that. We intend to launch that um in the next sort of 6 months. Um the DBT VS Code extension also in public beta, you can install it.
It only works with the Fusion engine. Um uh you just type in DBT the Visual Studio Code Marketplace, like I showed you. Um state of orchestration, this one is worth taking a little bit of time on. Um I mentioned to you it a few times, but essentially imagine sort of new data comes into raw orders there, right? And it only reruns those models downstream that depend on raw orders. It doesn't rerun the other ones that depend on raw customers, payments, or products.
Um early estimates on our own uh internal uh data projects look at about 10% cost savings by just skipping those model runs that don't have fresh data. Um you can imagine even extending this to uh uh like on a CI job when you do a merge job and you need it to deployment, right? Like it should only run the models that have been modified and the downstream models of those modifications, right? That's what it does today, actually, with state modified.
But, we can take it a step further. We can look at the exact columns that have been modified and only rerun downstream models that were impacted by those column modifications, right? Through sort of a what we call like sort of a a breaking change analysis, right? Um and so all of that, I think, right now with just these features that are there today about 10% cost savings, but we want to drive that up to 20 even 30 um in on many projects based on the features we're launching.
Um and this guy, he rocks. He's really cool. I wonder if he's at the conference. Maybe. I'll find him. Anyways, he has this quote. The Fusion engine with VS Code extension is how folks will want to develop with DBT moving forward. This is awesome. It's the experience we've all been waiting for. Raise your hand if you agree. Yes? Yeah. Okay, pretty good. Um I agree. Cool. Thank you. Let's do it. Let's do questions.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.