Sponsored by: Coalesce | The Age of Data Exhaustion

Summary

Coalesce Field CTO Douglas Barrett describes a state of "data exhaustion" in which data engineering teams are overwhelmed by thousands of tables, brittle transformation logic, and AI pilots built on untrusted foundations, arguing that AI amplifies existing data disorder rather than resolving it.
This video explains how Coalesce's metadata-driven transformation layer on Databricks standardizes pipeline patterns, connects governance to build-time, and shifts data quality from reactive to proactive—reducing fragility and restoring team productivity on medallion architecture deployments.
Coalesce addresses the longstanding tension between business urgency for data and the engineering team's need to deliver it correctly, offering a path to continuous improvement through pre-production data quality gates and catalog integration that let teams build quickly without sacrificing trust.

Chapters

FAQs

What is data exhaustion and how does it affect data engineering teams?

Data exhaustion, as described in this video, is the state data teams reach when they are managing thousands of tables, maintaining brittle transformation logic, and trying to support AI pilots built on data foundations that no one fully trusts. The exhaustion comes not from data volume alone but from code overload, data overload, and the erosion of confidence in the accuracy and reliability of what has been built.

Why does AI amplify data disorder rather than fixing it?

This video argues that AI tools accelerate development velocity, making it faster to generate transformation code and build pipelines—but when those faster-built pipelines rest on poor requirements, inconsistent patterns, and ungoverned data, AI multiplies the inconsistency and technical debt rather than cleaning it up. The result is a larger set of fragile, hard-to-trust assets than existed before AI was introduced.

How does Coalesce use metadata-driven transformation on Databricks?

Coalesce uses a metadata-driven approach in which transformation patterns are defined as reusable templates rather than written fresh each time, standardizing how pipelines are built across a Databricks data warehouse. This video describes how connecting governance to build-time—rather than checking quality after the fact—makes pipelines more consistent, easier to understand, and less prone to the kind of drift that causes business users to lose trust in their data.

What is the difference between reactive and proactive data quality approaches?

Reactive data quality means catching problems after data has already been loaded or consumed, typically through monitoring alerts that fire when something breaks downstream. Proactive data quality, which this video presents as Coalesce's approach, means enforcing quality gates before data reaches production—checking rules at build-time and pre-production stages so issues are caught and resolved before they affect business users or AI models.

Full transcript

[00:08] Welcome to the the age of data exhaustion. Sounds a bit grim. AI is not all roses and sunshine, and we're going to take a quick look at what that what those problems are. Now, this is primarily aimed at data
[00:21] data engineering team that's using AI for development.
[00:35] Okay. My name is Douglas Barrett. I'm a field CTO at Coalesce. Uh Coalesce is a software company. We build um tools for data engineers to build, manage, deploy
[00:47] data warehouses on Databricks uh in the age of AI. Now, my job as a field CTO is to talk to customers and prospective customers. Basically, find out what their problems
[00:59] are and see if Coalesce can help them. And it's really those conversations that are form the basis of this talk.
[01:14] So, we're going to have a look at what data engineers do to build data warehouses, what issues they they hit. Uh we're going to have a look at why AI can sometimes make that worse, not all of it, but some of it.
[01:26] Um we'll have a look at how Coalesce tools might be able to help out in that situation. And then, obviously, that leads to nirvana where we get the to use AI um but without any of the drawbacks.
[01:45] Okay, so as I'm talking to um customers, there seems to be a recurring set of issues. So, building a data warehouse is complicated. Uh there's a natural tension between the business and between uh IT or the the data teams
[01:59] delivering the data warehouse, uh the business want the data now. Uh the data teams wants to deliver the data right. And there's a natural tension there. Story as old as time.
[02:11] Well, since the 1990s anyway. Um and really most of the problem is caused by poor requirements. The business give us poor requirements because they don't know what data they've got to work with.
[02:25] Uh it's common for them to say, you know, "Give me what I had on SQL Server. Rebuild it on Databricks and I'll be happy." Or they'll just say, "Well, just give me everything. We've got SAP. Just give me
[02:37] SAP data." Uh well, there's 100,000 tables in SAP, so they probably don't want that either. So, really the the best way of engaging with the business has always been to build a working proof concept. Take
[02:51] their requirements as a starting point, look at what data is available in that area, and build something. Build something quick. Build something with production data and socialize that with the business. Find out uh what they
[03:04] like, what they don't like. Make iterations to that based on their feedback. And the key to that is doing things quickly. The business has a short attention span. And so, we need to build things quickly,
[03:18] change things quickly, get it back in front of them, and get to the best possible design based on their requirements and uh based on the data that's available. And AI is great in that regard because,
[03:30] hey, it's going to speed things up. It's going to generate a whole bunch of code. It's going to generate a whole bunch of stuff for us nice and quickly.
[03:42] Supporting a data warehouse is also hard. Uh in any kind of environment, we're going to have hundreds or thousands of tables. We're going to have thousands of lines of code. We're going to span multiple
[03:55] technologies, ETL tools, Python, SQL, dashboards. Uh we're going to encounter old code, code that's over 10 years old in a lot of cases that I've come across anyway.
[04:09] And the problem with that is not necessarily that the the code is old, but it's the people that wrote it have been lost to the mists of time. So, we don't know why they built it that way. Now, when it comes to support,
[04:22] humans need to understand what was built. Uh when someone rings up with and says, "Hey, we've got a funny number in here in this dashboard." A human has to go in there and figure out how the data got to
[04:35] that dashboard. Uh and that involves digging through well, source data, the loading mechanism, the transformation logic, all the way through to that dashboard.
[04:47] And really AI doesn't help as much in that regard as it does in the development of code. So, what we find is that when we're using AI,
[04:59] the expectations are that we can deliver projects much, much faster. We can get more products more projects into production quicker. Uh and we'll probably have more projects as well because hey, AI needs data. So,
[05:13] we'll be preparing uh data for AI projects. So, the business expects that we can deliver these projects fast and that means more code, more tables, more
[05:25] reviews, more deployments. And this is what starts to grind the team down is you can't just generate code with AI and deploy it to production. You kind of got to check that that code looks good. You
[05:39] got to review all the changes that that that uh AI made to all the code. You got to test it. There's all that workflow that needs be done in order to um approve that that movement into
[05:51] production. And as you do that, as you move stuff into production, hey, you own it now. It's like you had the consultants come in, all care and no responsibility,
[06:04] build something and then skedaddle. When it goes wrong, it's your problem. It's not AI's problem, it's your problem. So, pressure builds up in the whole support around the the development of code. If we take the
[06:19] cost of developing code down almost to zero with AI, it doesn't take all the other surrounding parts of managing that code down to zero as well. And AI AI does have issues.
[06:33] Um it's not all roses. Um it can hallucinate. I've trouble shot plenty of SQL statements where it's imagined a column that doesn't exist in this table. It says it should, but hey, it doesn't.
[06:46] It doesn't have institutional knowledge. It doesn't know that we upgraded the uh loan origination system last year, and so the way that we look at these the the parts of the loan origination cycle
[06:59] is changed. So, we can't compare data today with data from last year because the data's different. It AI doesn't really know that. It's non-deterministic. When it's generating code, it can generate code of
[07:12] one pattern today and then tomorrow a different pattern to try and achieve the same stuff. That's all pretty okay. All of all of that is okay. We can carry on using AI for that.
[07:24] Um it also doesn't have this accountability. Kind of you assume the accountability. But one of the biggest problems I've seen with AI code generation is this unbounded scope,
[07:38] which means you can say something simple like make this pipeline faster." Like a three- three-word prompt. And it can literally change thousands of lines of code all throughout your pipelines.
[07:51] And that's no fun to then find out what changed, retest, review, redeploy thousands of lines of changes just from that little prompt.
[08:09] So, the problem really is this opening gap between the ability to generate lots and lots of code and the ability to support and manage lots and lots of code without the right tools, of course.
[08:21] So, that whole deployment management around that code generation is where the teams get exhausted, which is the sort of the the theme of this talk. And without the right tools, when you're
[08:33] generating code with AI, you're going to end up with these differing standards. You're going to end up with duplicated pipelines because, hey, it decided it's quicker and easier just to build a new pipeline than change an existing one.
[08:51] It's potential for growing cost computational costs on the platform as it's duplicating logic. Um one of the biggest problems with all of this is this compounds over time. You build a a pipeline, move it into
[09:03] production, next project you build on that pipeline, and so you're compounding the issue with more and more code. And that can lead to trust erosion. Uh trust erosion where the business doesn't
[09:16] fully trust that you're on top you're in control of the the platform. And that's a feeling a lot of data engineers have is that they're losing control of the environment. And when you lose trust,
[09:28] the users the business will lose confidence in the data. If they lose confidence in the data, they're not going to use it.
[09:41] Okay, this is Bob. He's a He's a data scientist. There's no data scientists here, is there? Maybe. Well, this is a real world customer story that I've just come hot foot from. And it's basically a a retailer, a big
[09:54] retailer over in Europe. And they have a sales data mart where they pull in their transaction logs from the point of sale system. They pull in inventory, they pull in
[10:06] price and cost information, they pull in competitive information. It's a big, complex, mature environment. And they use that data to calculate and try and forecast their sales for up and coming promotions. It's
[10:21] a Black Friday, they're going to discount the price of a vacuum cleaner. They want to know how many more vacuum cleaners they're going to sell. Now, they had an existing process. It was about 70% accurate based on price
[10:34] elasticity, season, so forth. Uh implemented in a pipeline. And they thought that, "Hey, we could try and improve this if we got Bob along, the data scientist, to try and improve that with some um some AI
[10:46] generated magic." And he did. He spent 3 months, improved the the forecasting accuracy probably by about 8%. And of course, the business were overjoyed. Um Bob gets a pat on the
[10:59] back. There's pressure to move that into production because, "Hey, we need that 8% improvement so we know have we got enough stock for any promotions we're going to run."
[11:11] But that 200 There's a 200-line stored procedure that was originally calculating these forecasts has now been replaced by 4,000 lines of Python in about 40 notebooks. And most of those
[11:23] have been AI generated and glued together with uh with Bob's help. And he's like, "Here you go, fellas. This is This is the work." And uh now the data engineering team has to take that and run that every day to
[11:35] to look at the forecasting. Now when that landed on the data engineer data engineer's desk, they thought, "Okay, well, how hard can this be? Let's look at the code. Let's um
[11:48] try and understand it, document it a bit because of course uh Bob's not great at documenting anything." Um when they looked at it in 4,000 lines of code, they thought, "Well, let's try and rebuild this. Try and simplify it." But it was um it was dreadful code. It
[12:01] had hard-coded dates in it, hard-coded um product lines, hard-coded products in it, hard-coded everything everywhere in it. Well, the sort of thing that data engineers stay keeps them awake at night. Data scientists not so much. They
[12:15] sort of used to doing a project and then scarpering. So, the only option to get this into production was to keep it as a black box. Just call it, run it, hope the output looks good, and kick the can down
[12:28] the road a little while. But of course, almost straight away, the business start saying, "Well, this number looks a bit high. Uh why is this number so high?" And now the data engineering team, who
[12:40] now have assumed ownership of this project, going, "Don't know." And that kind of eroded the trust of the business and it's not a great situation to be in. But that's their their situation right
[12:52] now and they're basically in a world of AI hell, just trying to figure out how the thing works and asking Claude, "Well, why is this number high?" It's a bit bit broad. You've got to give it some data to work
[13:04] with and and really try and work that through. So, not a happy place to be at the moment. And that's really exemplifying the loss of control that those engineers were feeling.
[13:19] Now, Coalesce, we try and put the data engineers back in control. What we're trying to achieve is the ability to use AI, get the benefits from AI, be able to build quickly, solve complex problems quickly,
[13:34] get something built quickly, but we want to leave the the data engineers in control of that environment. And we do that with these three tools. We've got three tools, transform, catalog, and quality. We'll dive into those in a little bit
[13:46] more. And they're all AI enabled. So, we have command line interface the AI likes if you're working with files. We've got MCPs. We've got AI built into the tool.
[13:59] But basically, these are built on deterministic building blocks. There we go. So, transform is a tool that data engineers would use to build out a data
[14:11] warehouse. Build out the pipelines to go from bronze to silver to gold to to um platinum. Build all the transformations in between. But each one of those boxes is um is
[14:24] basically a a SQL node that embodies a whole bunch of standards. So, we've got standards for building
[14:37] materialized view node. And each one of those node types has these standards built in. Naming standards, storage standard, coding standards, DDL, DML standards, uh
[14:49] as well as model standards. So, those are sort of encapsulated in there. So, when AI goes to build a pipeline, and it thinks, "Okay, I can break this down this pipeline down into these nodes." Each
[15:01] one of those nodes follows the standard rigidly. Also, there's a a whole load of rich column level metadata as well. And the goal here is to make it so a data engineer can easily understand a
[15:14] very complex environment. Really, its superpower is being able to scale up so that we can support very large-scale environments without incurring massive overhead in in support
[15:27] costs. We also have catalog. This is a way of trying to make the business self-sufficient. So we point catalog at
[15:39] the data environment, at the dashboards, at the ETL tools. It's going to suck that metadata in and try and present it to the business in a nice easy-to-consume way.
[15:51] The way it does that is it's just a search engine over this rich metadata. And really its superpowers are it's really quick and easy to set up. It's also very engaging with the business. Like for example, in this
[16:04] diagram here there's a a report, a dashboard. Has this little pop-out that comes out on the side of the ETL in this in the um in the BI tool, like Power BI or Sigma, whatever tool you're using. It pops out
[16:18] and you can ask context-sensitive or context-rich questions about this dashboard, like what does COGS mean? Where does rider volume come from? What does buyer mean?
[16:30] That kind of information it's getting from all of the the lineage that it competes together from your ETL tools, your SQL pipelines inside Databricks. Pulls all that out and enriches it.
[16:47] And then we have our quality tool, which is really a trust layer that we can build in. It's a data quality and a data observability tool that monitors the data. And it's used by data engineers and product owners really to try and find
[17:01] problems with the data before the business does. It's monitoring the data and looking for anomalies in any sort of time series distributions that it can find in the data. And then it when it goes outside of what's an expected value, and you've got
[17:14] some control over how wide that the band might be as it's looking at trends over time. But when it falls outside that band, it pieces together a nice English description, natural language
[17:26] description of the problem, and also it can have a look at try and figure out the root cause as well. So if this started to happen after we did a deployment to production, it's going to flag that as a potential root cause.
[17:40] Um so that's pretty handy for trying to provide trust to the business about the data that we're providing. So all of these things together are
[17:52] really trying to put the data engineers back in control, even if humans are building stuff, or the AI is building stuff, it's using these building blocks which make it much more um manageable from a data engineering
[18:05] standpoint. Adding these standards, adding these building blocks, adding this metadata to make it much more easy to support at scale.
[18:19] And that's the end of my talk. So if you wanted to come along and see a demo of any of these products, we're on booth 561, which is buried a million miles that way, and um feel free to wander along. We can give you a demo in any of
[18:31] that if uh if that was interesting. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.