Skip to main content

Lakehouse Observability at Scale: Dashboards, Alerts, and Automation for Cost and Performance

Summary

  • DoubleVerify, ingesting 20 billion rows of advertising data daily on Databricks Lakehouse, built a three-pronged observability blueprint using System Tables that surfaces hidden storage costs including orphan files, Delta transaction logs, and Parquet metadata overhead invisible to standard monitoring.
  • The blueprint delivers three dashboards tracking total cost with DBU and cloud breakdown, table-level physical versus logical storage ratios, and column-level cost analysis via Parquet footer inspection, paired with smart alerts for cost anomalies, budget thresholds, unused clusters, and query performance.
  • Centralizing optimize, vacuum, analyze, and retention policy operations into a governed automation layer replaced decentralized chaos and yielded double-digit storage savings and a 30% performance improvement across the platform.

Lakehouse Observability at Scale: Dashboards, Alerts, and Automation for Cost and Performance

Watch: Lakehouse Observability at Scale: Dashboards, Alerts, and Automation for Cost and Performance
Lakehouses offer unlimited, inexpensive storage and decoupled compute, but that flexibility hides a cost problem: visibility gaps. Hidden beneath the surface of organized data tables lie orphan files, Delta transaction logs, Parquet metadata overhead, and redundant copies waiting for the 7-day vacuum window. DoubleVerify, processing 20 billion rows daily across petabytes of data, built an observability blueprint on Databricks System Tables to surface real costs, link them to usage patterns, and empower teams to optimize their spending.
This lightning talk reveals the infrastructure that transforms lakehouse chaos into governance: three dashboards tracking total cost (DBUs plus cloud), table-level physical vs logical storage, and surprising column-level cost breakdown achieved through Parquet footer analysis. Pair dashboards with smart alerts (cost-effectiveness anomalies, budget tracking, unused clusters) and centralized automation (optimize, vacuum, analyze, skipping, retention policies). The result: double-digit storage savings, 30% performance improvement, and teams that own their data costs rather than watching budgets balloon quarter to quarter.

Chapters

FAQs

What are the hidden costs in a Databricks Lakehouse that standard monitoring misses?

Common hidden costs include orphan files left behind by failed jobs, Delta transaction log overhead, Parquet file metadata that grows with each write, and redundant data copies that persist because of the default 7-day vacuum window. DoubleVerify discovered these through Parquet footer analysis and System Tables, finding that column-level data duplication was a significant but invisible driver of storage spend.

What are the three dashboards in DoubleVerify's Lakehouse observability blueprint?

The first dashboard tracks total cost by combining Databricks DBU charges with cloud storage costs at the workflow level, including tag-based cost attribution. The second compares physical versus logical storage at the table level to surface inefficiency ratios. The third performs column-level cost analysis by parsing Parquet file footers to identify which columns and tables are driving the most storage overhead.

How does centralized automation improve Lakehouse governance and performance?

Centralizing operations such as optimize, vacuum, analyze, and retention policies into a single governed automation layer prevents individual teams from running inconsistent or conflicting maintenance jobs. DoubleVerify moved from a model where each team managed its own tables to one where the platform team enforces policies uniformly, which was essential for achieving double-digit storage savings and a 30% performance improvement.

How does DoubleVerify use Databricks System Tables for cost attribution?

System Tables provide query-level and cluster-level cost data that DoubleVerify joins with storage metadata to produce workflow-level cost attribution across its 20-billion-row-per-day ingestion pipeline. This linkage lets teams understand not just total spend but which specific pipelines, tables, and users are driving costs, enabling targeted optimization rather than blanket budget reductions.

Full transcript

[00:07] Thank you for coming today to my perform my presentation on scaling Lakehouse observability and performance. So, I'm going to go through these slides very quickly. I have a lot to talk about. Please bear with me. This is a lightning talk and you will be able to see it
[00:24] a recording of this later on. So, I hope that I pique your interest but I'm going to move quickly. So, a little bit about me. My name is Saul Tawil. I work for Double Verify. I've been about 30 years in the data management and development space.
[00:42] I've spent a lot of time in financial services, Cisco, last 3 years at Double Verify. And I've been involved in data engineering and big data for the last 10 years. So, a little bit about Double Verify. So, we're in the digital advertising space. Our vision is to make digital
[00:58] advertising stronger, safer, more secure. And our mission is to bring better outcomes for advertisers. So, I work on the measurement data platform team. We have a wonderful team and our scope is basically in taking the
[01:16] impressions from the ads from all over the world and ingesting them into our measurement data platform. We use Databricks Lakehouse for this platform. As you can see on the bottom, we've got multiple
[01:32] data platforms. We use two other leading cloud data platforms. That's outside the scope of what we're talking about here. So, the scope of our platform is about ingesting 20 plus billion rows of data per day, petabytes of data, thousands of
[01:47] tables, hundreds of users. The scope of the team is about putting together the tooling so that the all the different teams that work with the platform can do their job. Uh we we're in charge of governance and
[02:03] most very importantly of the last couple years has been cost management and performance and maintenance. And I'm going to be focusing today What? It flipped by too fast. Um the focus of this session is going to be on these last two responsibilities.
[02:20] Okay, a little bit about the our data lakehouse architecture. So, we made a successful migration recently from uh the Vertica data warehouse to Databricks. Um we stream in uh these billions of rows that I talked
[02:37] about uh with Kafka into our lakehouse using your classic uh medallion architecture uh going from bronze silver to gold. And we serve multiple personas, uh including data engineers, machine learning engineers, and BI analysts.
[02:54] So, my agenda for today is to cover talk to you a little bit about what the challenges are of lakehouses, especially when it comes to visibility. And then I'm going to talk about the solution, which is in a nutshell vis- visibility, action, and automation. I'm going to be focusing mostly on visibility.
[03:11] So, what is the lakehouse challenge? So, we all love lakehouses, right? We all use them. Um we we love their unlimited scalable storage, right? We could We can We know the storage is cheap and we can expand it as much as we want. However, with
[03:28] that comes lack of oversight, right? People just throw a lot of stuff into the lakehouse without thinking, and there becomes these cloud billing shocks all of a sudden, especially like us after using it for a couple of years, all of a sudden you see your storage costs, you know, go through the roof.
[03:44] So, we're going to talk a little bit about how we handle that. Uh the second benefit of a lakehouse is the decoupling of storage and compute. Okay? The Uh that's a that's a great advantage, right? We can scale each one independently. However, with that comes
[04:00] a little bit of loss of visibility in terms of cost, right? You've got a We get a bill from Databricks, we get a bill from our cloud provider, and we really don't know our total cost uh the total cost of ownership here. And thirdly, we have the advantage of advanced table formats.
[04:16] Um we're working with immutable files, object stores, um that you can't modify them, right? So, every time we need to make a change, we need to create new files and delete the old files. Okay? Now, that leads to lots of uh orphan
[04:31] data, which we'll talk about. And we've got metadata logs, right? Uh table formats work with metadata in order to manage which files are current. Okay? And that takes up some storage as well. The bottom line is we have a lot of visibility gaps when it comes to a data lakehouse, much more so than in
[04:48] other data stores. So, I want to talk to you a little bit about the orphan files tax. Um this is what we call uh the orphan files. Now, if you take a typical scenario, we've got a table, it's based on uh you you you you keep on
[05:04] adding to it every day. It's a time-based table. And let's assume for a second that it's mostly uniform, maybe it grows a little bit every day, okay? And you've got some kind of retention policy on it, right? Maybe 30 days, 45 days, 90 days. Um and every day you need to delete the data that goes over that
[05:21] retention, okay? So, you're deleting one day's worth of data every day. In addition to that, right? If you're following best practices, you're optimizing that data, right? Especially with streaming data, it's coming in, you've got a lot of these small files, right? You've got to convert them to bigger files, right? To avoid the small
[05:37] files problem. Okay? So, in order to do that, you optimize. Now, optimize, what is it doing behind the scenes? It's deleting the the today's data, one day's worth of data, and you're create a whole new set of data. So, basically, you've got two days worth of data, which is basically dead. You're not using it on an active
[05:53] basis. It's a safety net, but you're not using it. Now, Databricks, by default, has a vacuum window of 7 days, okay? So, basically, that data is not getting cleaned up for another 7 days. So, you've got 14 days of data just sitting there, just in
[06:09] case, and maybe that's important, but you need to understand the cost and if that's worth it to you, okay? So, that's 14 days of data. It doesn't matter what your retention is. If you look at the chart on the right, uh the low Actually, it turns out, ironically, the lower your
[06:25] uh retention period, the actually the more significant the problem is because uh the 14 days is fixed. It doesn't matter what your retention window is, right? So, if you're 45 days, it's 30%, 30 days, 50%, 15 days, it's almost 100%.
[06:40] So, you're paying 100% tax on top of the data that you have. And if this is a large table, uh which we have many large tables like that, this can be quite costly, okay? So, you have to make basically be able to surface this these costs and make some decisions about whether this makes
[06:56] sense. Now, about a year ago, Databricks gave us the capability of controlling this vacuum window, and now you could actually customize it. You could say uh like like in the case that we did, we reduced it to 3 days, okay? And that helped us save thousands of dollars a month, okay? So, it's important to look
[07:12] at this, to understand your costs, and make those decisions. Okay, so, here's why why I like to look at lakehouse storage, okay? I like to think of it as a container ship. Picture the ones that are stuck now before they go through the Straits of Hormuz, right? You've got that container ship. It's got
[07:29] these beautiful containers on it, all different colors, all different sizes, and they're organized beautifully, right? And that's what you see above the surface. However, below the surface, there's a lot of data there, a lot of storage that you don't you can't see. It's below the the water line. It's in
[07:44] the hull of the ship, right? That's the orphan files that I just talked about, as well as Delta logs, parquet footers, parquet overhead. There's lots of data there, lots of file storage that's being taken up. This is the fact of life of lakehouse storage. So, what is our solution?
[08:02] We have a three-pronged solution, uh observable what we call an observability blueprint, where we go from the chaos of a lakehouse and we try to bring it a little bit more governance. Okay? So, we're talking we're going to talk about visibility. I'm going to focus most of my solution session on that. We're going to
[08:17] talk about actions. We're going to talk about automation. Okay? Now, this requires a bit of a cultural shift. And that's what I want to talk about now. The true visibility is about surfacing your physical storage costs, as well as your use usage metrics to see if that it justifies that
[08:33] storage. Okay? And for this, we're going to need today, unfortunately, to write custom jobs and dashboards in order to bring all of this all of this together. So, the cultural shift that you need to take as an organization is going from let me just keep on increasing the budget the storage budget as opposed to
[08:50] let me see how I can optimize my spend. Okay? And the third part of it is really empowering your teams. Okay? Each team should be in charge of their data. They should own their cost, but you have to give them the information. You have to give them the data. You have to tell them how much
[09:05] their storage is costing. And that's the important part here. And then, finally, we need to alert them if there's any cost anomalies. We can't depend on them to go to the dashboard proactively. We need to also alert them. And that's what we're going to talk about smart alerts, and we're going to
[09:21] talk about how we automate a lot of the operations to make it again easier for the teams to focus on what they do best. And a little spoiler alert, if you look at the red box, the results uh for us speak for themselves. Uh we've been able to reduce our storage costs by double-digit percentages.
[09:38] We've been able to improve performance by 30 over 30% and we already made a lot of life cycle management processes for our teams so they can focus on building business value. So, let's talk about the first step, visibility and dashboards,
[09:53] as well as the jobs that go into them behind them. Okay, so the first dashboard we're going to talk about is the the Oops, did I skip the slide? Fine, one second. A little too fast there. Okay, so we've got three main dashboards that I'm going to talk
[10:09] about. Total cost. This is where we bring together the cost of Databricks and your DBUs together with your cloud cost and your storage cost, okay? That's the main thing that's going on here. Table cost is about showing you
[10:25] your table sizes and the costs, okay? But, showing you also the physical storage cost behind those tables, okay? That's the important part here, as well as the usage patterns. So, we can understand what's the value of these tables are bringing. And then the third third step is the column cost table.
[10:41] This is a much more difficult uh uh metric in order to uncover because, as we know, Parquet is semi-columnar. Not so easy to find out what the column sizes are, but I will show you how to do that.
[10:56] So, total cost, like I said, bringing together uh DBUs and cloud. That's the main issue point here. The other important point is to remember is that you uh are going to want to tag all of your assets, okay? In order to do that,
[11:12] you're going to use now uh Databricks' govern tags, which is great and helps us uh manage the tagging process. Once you've got all your assets, especially your compute, tagged, okay? Now, you can filter your costs by all these different filters here, whether it's environment,
[11:28] warehouse, owner, workflow, etc., okay? This is very powerful. Um I want to point out, for instance, let's say we couldn't do this before. Let's say you have a particular workflow, you want to know how much it costs you, right? Uh some are using job clusters, not serverless. You've got it split up
[11:44] between DBUs and cloud. If you filter on your particular workflow in our dashboard, you can see both together and see what your total cost is right away. Okay? So, that's one of the advantages here. So, the important thing is to be able to have that kind of granularity and use govern tags and use cluster
[12:00] policies to make sure that you're applying those govern tags all around. And then, this is standard on any cost dashboard you'll find. You have your top computes, your top workflows, and your top warehouses, and we do all of that
[12:15] together as well. So, the second dashboard is the table cost dashboard. This is again where we bring together the logical table sizes with the physical sizes. Okay? And I want to show you this chart out that's on the right. I'm going to zoom in. If I can get this thing to work. There
[12:31] we go. Okay, so uh I'm showing you a trend here of how a table grows over time. And if you look at the blue, that is the actual size that you will see if you open up your Databricks UI and you look at the size of your table. Okay?
[12:46] However, the yellow is the actual physical storage that it had takes on disk. Now, that physical storage as you can see, sometimes is is is a lot larger than the actual logical. And this helps the our users
[13:02] uh be able to visualize it visualize this and and spot any issues. Um So, this is very very useful, very helpful. Uh and this is kind of like the table format how we show that, and we show them the storage percentage, which basically shows them the the how much
[13:18] the storage is adding as an overhead to the table. So, you can see in our case, and this is a little mocked up data or anything, but but it's about 30%. Okay? So, it's about 30% more in this case so you could track that as well. Um the next piece I wanted to show you here is how we integrated whoops went
[13:36] too fast there. I want to show you here how we integrated a unique metric which I'll go through in the next slide in more detail but called cost per read. This allows us to really understand what's the value of this table. Okay, if we take the number
[13:51] of reads and we break it down into uh all different kinds of ways of looking at the reads but we take the total reads and and divide it into the total cost, we understand what's the real value of this table. Now let me go to that a little more detail. Okay, so the uh cost per read is is is
[14:10] like I said it's taking all the reads. Now we're looking at the downstream read count which basically requires us to look at the lineage table and to get all of the reads that are associated with the source table. Okay, and then we divide that into the physical storage cost and that becomes a cost per read that helps us identify cost tables that
[14:27] are basically not cost-effective. All right, if they have a high cost per read, they're not cost-effective. Then we have the column dashboard. Now this dashboard uh like I said is not so simple to create. You need to be able to inspect your parquet footers,
[14:43] look at your the what it says is the size of your column chunks and aggregate them all up on a petabyte size uh warehouse like ours, um it's it's it's basically not a easy task to do. I'll talk a little bit about how we did it.
[15:01] About one other thing I want to mention here is that this allowed us to see duplicate data in terms of what columns are just copied from brown from bronze to silver to gold and without any concern whether they're the data is really needed. Select star and that's it. So if you've got very complex
[15:18] uh uh columns with a of nested fields, this can become very very expensive. As soon as we put this dashboard out there, within a month, we had our storage costs go down by thousands of dollars because they saw this duplication and they were we highlighted it for them. So, that's
[15:34] the power of visualizing and surfacing this data. And then parquet views, I'm going to go over that. I'm going to run through this, but this just allows us to inspect the parquet uh metadata and see if we have any issues there. And we can look in uh
[15:50] how big our data log delta logs are, parquet photos, etc. and see if we have any issues. Okay, so uh in order to do all this, of course, we need system tables. I don't have time to go through all this, but how we use them, but I'm going to call out the lineage and the history which are
[16:06] particularly useful. Okay, and then under the hood, we're using a lot of libraries in order to calculate the storage cost storage sizes. Okay, so we use cloud cloud
[16:22] libraries from FS spec uh they're available for all the different clouds. Uh delta rust uh libraries, pyarrow, etc. We used a lot of bunch of parallelization techniques, otherwise, you cannot get this done. Uh and of course, AI agents help us along the way
[16:37] very much. Genie code is your friend. So, now I'm going to talk about the second area which is alerts. So, we have a cost-effectiveness alert. This is the alert based on the cost per unit that I talked to you about. And this uh we we push this out to the teams once
[16:54] a week and we threw this put this into their channel a list of the least cost-effective tables that they have for them to take a look at and see what they can do. Uh I have an example I'll show you. We have an alert on a developers are using the wrong cluster type if they're using maybe SQL
[17:10] warehouses for uh jobs uh instead of a uh job a job compute. We have alerts on queries that are running for a very, very long time, and we have alerts uh to show teams how they're tracking
[17:25] to their budget. So, here's an example of one of our Slack alerts on the cost effectiveness. Okay. So, the third area I told I told you I was going to talk about is automations
[17:41] uh and optimizations. So, we've come to the conclusion that uh we had eventually given to our teams uh some tools to be able to do optimize and vacuum on their own, and we found that it was kind of spotty.
[17:56] We decided to take it all into the data platform as a responsibility and to centralize it, okay? And we found that this really helped a lot uh and in in fact, it improved our performance by 30%. So, something that I highly recommend is take control of it in a centralized
[18:12] fashion. Um as well, from learning from that experience, we also centralized, analyzed, and data skipping. And as well, we created a framework for centralized retention. And then finally, we've got some
[18:27] advanced optimizations, which we've done where we uh calculate what are the most frequently used columns in the where clause and the predicate. And this helps feed into our analyze, okay? And our dealt and our data skipping, so that
[18:44] we're uh doing the uh doing these operations on the proper columns, and this is dynamic. Uh we recalculate this all the time, um so it's always changing, okay? And this again helped us improve performance greatly. And then finally, we we just did this uh
[19:00] recently, we did some uh testing around uh expanding the Parquet row group size. If you're familiar with uh Databricks' default, it's 128 uh megabytes, and where we we're expanding that row group size because the bigger the row group
[19:17] size, uh the greater throughput you can get. Okay? And this has helped us improve performance by 15%. So again, this is something I'd recommend doing it on all your large tables across the board.
[19:32] Okay, so that's all I have and uh some of the key takeaways uh I'd like you to take away from this. I hope you uh gain some insights here uh is visibility. Focus on visibility. Focus on getting that visibility down to the proper granularity. Okay? Um alerting.
[19:49] Don't just rely on visibility. Push out the alerts on the most egregious cost anomalies. Send it out to your teams to deal with. Give them the uh empowerment and the ownership over that those costs. Then we have centralization. Centralize
[20:06] as many of the uh optimization processes and maintenance processes as you can. And use your system tables. They're there for a reason. They're they're great. They're really useful. Uh highly encourage you use those. Finally, um
[20:21] here's a a engineering blog that I wrote which goes into a lot of detail in terms of how I wrote the jobs behind some of these dashboards, especially the table size dashboard and the column size dashboard. So you
[20:36] could scan that QR code if you'd like. It'll take you to my the blog page for DoubleVerify's engineering blog on Medium. Bookmark that. Read it later. Uh and you can go to a lot more detail. Okay? So like I said, everything here is uh recorded. You can review that. I just
[20:53] wanted to highlight and touch on a lot of these different ideas to pique your interest and I hope you took away uh some useful uh insights from here. And um there's any questions, I'll uh be happy to take any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.