Top Performance and Cost Optimizations for DLT

Summary

• Performance Tuning Strategies: Optimize DLT pipeline performance through practical strategies including autoscaling configuration, Photon optimization, and selection of appropriate node types for the Databricks Data and AI platform.
• Cost Optimization: Monitor resource usage and determine when serverless DLT is the optimal choice, enabling cost-effective pipeline execution while maintaining performance standards.
• Best Practices from Production: Learn from real-world customer implementations covering performance enhancements, monitoring approaches, and optimization techniques for reliable, efficient data pipelines.

Top Performance and Cost Optimizations for DLT

Talk by Steven Yu, Principal Solutions Architect, Databricks
DLT simplifies pipeline development and management — but how do you optimize for performance and cost? In this video, we’ll explore practical strategies for tuning DLT pipelines, including when and how to use autoscaling, Photon and different node types. We'll also cover how to monitor resource usage and decide when serverless is the right choice. You'll learn best practices drawn from real-world customer implementations, along with an overview of the latest performance enhancements available in serverless DLT.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction
• 2:20 Key Factors that Affect Performance
• 6:59 Changing Instance Types
• 8:47 Memory Performance
• 11:55 Tradeoff Triangle
• 14:56 Liquid Clustering
• 18:57 Project Lightseed
• 22:39 Autoscaling History
• 24:06 Performance vs Standard
• 26:01 Summary

FAQs

What are the key factors for optimizing DLT pipeline performance?

This video covers practical strategies including when and how to use autoscaling, Photon optimization, selection of node types, and monitoring resource usage on the Databricks Data and AI platform.

When should I use serverless DLT?

According to this video, serverless DLT is optimal when you want to eliminate infrastructure management overhead and let Databricks automatically scale resources based on pipeline demands.

How do I monitor DLT resource usage?

This video covers best practices for monitoring resource usage in DLT pipelines to identify optimization opportunities and make informed decisions about infrastructure choices.

What performance enhancements are available in serverless DLT?

The video highlights the latest performance enhancements available in serverless DLT, derived from real-world customer implementations and industry best practices on the Databricks Data and AI platform.

Full transcript

0:00 Introduction

Good morning everybody. Welcome to Data Plus AI Summit. U. Um so before before COVID uh I used to do this thing where I'd have you kind of get up and high-five the person next to you just to make things not awkward for the next 40 minutes. So if you to do that, you can. Uh but if not, you can just kind of listen to me kind of ramble on for the next 40 minutes. So um who am I? Uh who is this person standing in front of you talking about performance?
Well, I'm an essay here at Data Bricks. I've been with a company now for just under eight years. Uh and throughout that time, I've seen and worked on a lot of really interesting performance problems and a lot of interesting streaming use cases. So, I've tried my best to kind of distill those things, those lessons down into some generalizations that I'm going to share with you today. Cool. So, before we begin, uh you will probably see this slide uh in every Oh, there's live dictation.
That's cool. You're probably going to see this slide in every uh presentation that you're here uh that you're that you visit at Summit. And it basically says that I can't make any promises. Everything in the presentation is informational only and subject to change, namely the product name during Thursday morning's keynote. So definitely attend that. Okay. So I like to start my presentations with a quote. Uh this one is by a fictional character in a 2006 comedy about a race car driver named portrayed by the legendary Will Ferrell.
I want to go fast. That's what we're gonna talk about here today. So, agenda is very simple. Uh we're going to talk about some key factors that affect performance. Uh we're definitely going to touch on cost optimization. Talk about some of the knobs that you have, some of the controls that you have there. And then finally, what we've built into Lake DT to kind of make all of this easier for you. Now's your chance to escape.

2:20 Key Factors that Affect Performance

Just kidding. You can leave whatever you want. All right, so let's get started. Key factors that affect performance. Now, years ago, I bought an Apple Pencil to, you know, make these illustrations and drawings for presentations. And I finally got around to actually using it. So, the illustrations are going to kind of suck. Apologies. Um, did you know that Apple pencils die if you don't keep them charged? All right. So, we're talking about DT.
We're probably talking about ETL workloads. So you can oversimplify and kind of break down an ETL workflow in the three core components, right? There's a read component, there is a do something interesting after I've read my data component, then there's a persist this data somewhere, all that hard work. And for those of you that are familiar with how Spark works, it's really, really good at processing lots of data because it's able to take a big problem and break it down into a bunch of smaller problems.
And for the Spark experts that are in the room, you might recognize that there's a Spark configuration that kind of controls how Spark parallelizes things. It's called max partition bytes. Um I think it's 128 megabytes. The default size has its roots from the Hadoop era for those of you who remember. Uh that was the HDFS block size uh for most commercial Hadoop distributions. 64 megabytes for open source. Anyway, um we're able to take big block of data, break it up into units of work that we call spark partitions.
Okay. And then we go and assign a CPU to go and do some work with that partition of data. All right. Then uh do some interesting things. Um I don't consider decompression an interesting thing, but it's often a very necessary thing. Um you're probably shuffling some data. You're probably doing some joins. You're probably doing some math, probably doing some kind of aggregation. And then you take all of this stuff and you write it back out to cloud storage or to disk somewhere.
Okay? And you're probably coalesing that while you're doing so. So what are all the things that can possibly go wrong in this process? No, I did not draw that picture with my Apple Pencil. I wish. Uh that is a AI prompt generated image. Apparently when you ask it to draw you a picture of a computer bottleneck, you get four very sad components staring at an empty wine bottle. Maybe that's why they're sad. Um so um things that can go wrong.
Uh well compression and decompression are CPUheavy activities. You might be working with uh deeply nested complex data types, oddly specific, which in turn causes things like disk spill. Um you might be, you know, running into network latency issues. Maybe you picked an instance type that was kind of small, didn't quite have the same amount of network bandwidth, or maybe you just didn't have enough parallelism. Maybe you're working with smaller amounts of data and that 128 megabyte default max partition bytes is too big.
So I mean you can basically classify all of these things as resourcebased issues. So to start with the first one uh we have you know the concept of a CPUbound workload. Can anyone take a guess what that means? Nobody knows. I'm just kidding. Um, so this is obviously where your CPU is your bottleneck. Um, you know, where you're doing, you know, lots and lots of work on the CPU. You're doing lots of math basically. And that is a terrible illustration of a CPUbound graph.
Just pretend the CPU is pegged at 100%. The next type is the memory bound workload where surprise surprise this is where you don't have enough memory and you're spilling the disk or you're hitting out of memory issues. Typically happens when you're working with data types that compress really well on disk but as soon as you read it it kind of blows up like 10x in memory. And then finally uh the third type would be an IO bound workload. not surprisingly limited by input output operations.
So reading data from cloud storage, reading data from disk, spelling data to disk. How do we classically address these sorts of problems? Well, the easiest thing that you can do is change the instance type, right? That's the lowest, easiest hanging fruit. Um, and in every major cloud provider, you have these different classes of VMs. Okay. Um, I want to mention something about the compute optimized one. So, we tend to forget that, you know, since you're just staring at a list of options that says this machine type has four cores or eight cores or 16 cores, we have this tendency to forget that not all cores are created equal.

6:59 Changing Instance Types

So, who remembers the CPU clock cycle or clock speed wars of like the 90s and the 2000s? Yeah. Yeah. My first computer was a I think a 486 SX2 66 MHz processor with 2 megabytes of RAM. And then in middle school, uh my family upgraded to a pennium with MMX technology. That was awesome. 166 megahertz. And the performance was amazing. screen. The point is like not all CPU cores are created equal. If you're dealing with a aggregationheavy CPUbound workload, it might behoove you to use better CPUs.
Uh there's also general purpose. That's a really good starting point. U these ratios, it basically means for every one CPU core, you have two gigs of RAM or four gigs of RAM. In a memory optimized instance type, you have eight gigs of RAM. And in an IO optimized instance type, you have basically your memory optimized instance with SSDs. So, let's talk about improving memory performance. I I mentioned disk spill several times.

8:47 Memory Performance

Um, and as an essay here, over the last eight years, I've seen this is probably like the number two most common problem that I see uh that customers have to deal with. Now, you can historically deal with disk spill uh a few ways. The obviously the easiest one is increase the amount of core memory that you have on your machine. But you're like, well, I I've already done that. I'm already using a memory optimized instance type.
I'm still running out of memory. What do I do? I'm still spilling the disk. Well, what else? The other thing that you can do is basically again like think back to how uh what we were talking about how Spark does work. It breaks things up into units of work. So I'll give you an example. Let's say you're reading a one gigabyte file. Okay? Uh and it's compressed really really well. That one gigabyte file when decompressed blows up 10x.
But you are working with a node that has let's say eight gigs of RAM. So if you were to go and have Spark read that 1 GBTE file, it's going to go and break it up into eight partitions, okay? 128 megabyte partition 1 gig. And then it's going to go and let's say you're running this on a VM with eight cores for the sake of example. It's going to go and blow up this one gigabyte file. It's going to become 10 in memory. It's going to spill the disk and then you're going to be sad just like this clip art.
So what can we do? Well, max partition bytes is configurable. We can reduce the amount of work that you're doing at any given time. So if you say change max partition bytes 128 down to 64, then instead of eight units of work, you created 16 units of work. 16 partitions but you have a fixed number of CPU cores right fixed number of Tesla so at any given time you're now processing half the amount of data that's one way you can address it now the number one problem that I see when dealing with performance problems u IO performance um and it kind of pretty much just boils down to how do we minimize the amount of work that we're having to do And I have this conversation often if you want your if if big data performance is all about like how can I reduce the amount of work I need to do and still get the correct answer.
So classically we would solve this using strategies like partitioning or clustering using Hbert curves and KD trees or some kind of a indexing strategy like bloom filters. Um, we'll talk about how this has changed in DLT. Um, but first, let's talk about cost. So, I originally had a a dial that had like cost on one side and it had or no, it was a cost dial that had latency on one side and performance on the other. And then marketing took a look and they were like, "No, that's terrible.

11:55 Tradeoff Triangle

You got to use a triangle." So I'm sure you've all seen the infamous trade-off the trade-off triangle. Uh you have cost, latency and performance is three dimensions. You basically pick two. So what are the what are the controls that you have? Uh first you can control the cost of compute and secondly you can control the amount of work to do. What does that mean? So for compute costs duh use cheaper instances. That's definitely one thing you could do.
You don't always have to use the most expensive instance type. You don't have to use on demand. Um, you kind of make these trade-off decisions based on what the SLAs's are for your workload. A lot of customers use spot instances. A lot of customers once you know what instance type you're going to be using for that job, it's probably not going to change. You get reserved instances. You're paying less. um you're using things like autoscaling uh and basically using only the compute capacity that you only that you need at any given time.
Uh one thing that I want to sort of point out here uh is that a common uh pitfall for people doing this for the first time is that they don't realize that just simply reducing the amount of compute capacity without changing the amount of work that you have to do actually increases latency. It doesn't save you any money. It just makes things slower. What does that mean? Well, let's go back to my poorly drawn 12 squares representing Spark partition.
So, let's say uh we have 12 units of work and it takes one minute to process each partition. Now, let's say you have a 12 core VM. Okay, you have 12 CPUs. you can process this entire chunk of data in one minute because we're doing 12 things in parallel. Then you might be thinking, okay, well, you know what, a 12 core VM is too expensive. I'm going to use a one core VM for the sake of example, one core VM. So, it's going to take me 12 minutes to run this.
It's okay. My data can be 12 minutes late. No one cares. The problem is the way that costs are structured in most cloud providers. Running a one core VM for 12 minutes is the exact same cost as running a 12 core VM for one minute. So don't forget you have to reduce the amount of work along with it. Okay. Now you might be thinking this sounds complicated. Um I hope I don't know. I'm judging by the faces I'm getting from the expressions.
I see some snickering. All right. So, um, that's where Lake Flow DLT comes in. Okay. So, the first thing that I'd like to talk about, um, I'm sure you've heard of this if you've been with data bricks for any period of time. The first thing that we've done to address IO performance is introduce this concept of liquid clustering. Now, let's let's talk about what the traditional problems were with uh, Hive style partitioning.

14:56 Liquid Clustering

It was very rigid, right? like partitioning is great. It's really good at like eliminating large chunks of data when you're filtering on uh in your query. Uh but it's it's rigid, right? So in this example, let's say you have a customer table that's partitioned on two dimensions, um customer and date, very simple, very common, but anyone that's worked with real world customer data knows that volumes are different, right? like you'll have varying varying amounts of data depending on who your customer is.
Could be a big customer, could be a tiny one. And so what happens is that you end up with these partitions. Some of them are right-sized. Some of them have no data. Some of them have very little data or the number of files are different. So, it's not really great for IO performance because you might be opening up way too many file handles for one partition and not enough for another. So, how does liquid clustering solve this?
Well, what if I told you that instead of having all of these files that are sort of like sparsely placed across these partitions, we can just go ahead and coalesce them across a dimension. Let's say for all of customer B or customer A we can have one file that spans multiple dates but not for every every customer right and then same thing for the other dimension across a single date we might be able to consolidate multiple tiny cluster uh customers doesn't stop there you can consolidate across multiple dimensions so tiny customers across several dates That's kind of cool.
But as would say, wait, there's more. You can also split this up. You can also split up larger partitions into smaller ones. That's kind of cool. Gives you a lot more flexibility. You have to really think about like, you know, limitations of partitioning. But you might be thinking, okay, that sounds great and all that you've already written all this data out and that already existed. What about for new data? Right? how does this look for things that I'm continuously ingesting on new days.
So in the very beginning, you know, you're going to be writing small files. Um but then as time goes on, as we start accumulating enough data to where it makes sense to start clustering these things, we can split these things out again on multiple dimensions. We have a table maintenance process that will come in and then consolidate all of these tiny files for you. So you end up with right-sized files. And then since we introduced liquid clustering, we also introduced clustering on right.
So if you're writing lots and lots of large volumes of data and it makes sense to cluster on right, we can absolutely do that as well. So we talked about the IO thing. Uh who who here has not heard of Photon? Raise your hand. All right. Well, I'm glad. Uh so Photon is a vectorzed query execution engine for Spark written in C++. Um it's much more efficient at doing map. It's much more efficient doing joins and it also is much more efficient at doing IO operations.
So when you're using Photon and data bricks and DT or not, you're getting all of these performance benefits specifically around CPU and around IO. And then for those of you who have been to or have attended Summit past, you might have heard of something called Project Lightseed, which is uh basically the our effort to improve the performance of our streaming engine. So we introduced the pro the concept of microbatch pipelining.

18:57 Project Lightseed

And again, looking back at history, uh, for those of you that have CS backgrounds or have taken a CPU course at some point, you sort of know that Intel CPUs back in the day were much better than AMD CPUs because they were much better at pipelining, right? You had they had way more pipes. You could do a lot more work and be more efficient about it. So let's take the microbatch example in a in a typical streaming application using microbatch processing.
You go and figure out okay stage you're going to have a job that reads from a source then you're going to do something interesting and then you're going to write it out right our three TL steps. So here stage zero the very first square um on the bottom left that's our read job. Okay. Well, that's our read app. That's our read. And then serially after you read your data, you shuffle it across multiple stages and then you write it somewhere.
Then after that batch is done, you execute the next one. Everything happens serially. Rinse and repeat. Now, what if I told you that in your stage zero in your read your map operation, you needed 200 CPU cores to do that read. But then as you shuffle and reduce this data, you only need like 50. That means you end up with 150 idle CPUs that are sitting there doing nothing until your job is finished. Okay? And then the next batch begins.
Rinse and repeat. In microbatch pipeline, what we've done is we basically said, hey, look, we have idle CPU. We should be using it. And because every microbatch has known boundaries, this is part of the structure streaming's resiliency model. We can say like, okay, we're done. We have 150 CPUs of capac or task slots of capacity. We can just go ahead and schedule the next microbatch to run. So this results basically in your end toend latencies going way down um to 30% in our benchmarking, better resource utilization.
And then who's heard of Enzyme? Who hasn't heard of Enzyme? All right. So, who's heard of materialized views? Any data warehousing people? Okay, cool. So, enzyme is really, really cool. Uh, as somebody with a data warehousing background because let's think about this. What is a materialized view? It's the output of a query, right? It's the persisted stored output of a query. And you typically did this for what? Reporting for things that were generally expensive to run.
So we ran it once and then everybody just looked at my output for some period of time and then every day, every week or however often it made sense, we'd just go and refresh that data. But for some time, you'd be staring at stale data. Enzyme allows you to go and incrementally update the output of your query when your source data changes. So again, we're reducing the amount of work that you have to do. You don't have to rerun these entire expensive queries and then rewrite all of that data.
You're able to go and incrementally update it. And then it's it's kind of hard to have a conversation about autoscaling without sort of knowing the history of how autoscaling worked on data bricks. So I'll start with classical autoscaling. Um way back when for classic jobs and stuff we had the concept of a cluster running in your data plane. We had the concept of a cluster manager in the control plane. And the cluster manager would periodically ping the cluster, see how busy it was, and ask basically, do you need more nodes to do your work, okay?

22:39 Autoscaling History

And then it would do this back and forth. It would add more nodes, and you know, cluster would scale out, scale down. But it was kind of like this this back and forth was kind of slow. So it wasn't great for streaming applications where you needed to be a lot more reactive. And so in DT classic, we introduced the concept of enhanced autoscaling where we basically moved the brains of the autoscaling operation onto the cluster.
And so now you could be a lot more reactive about your scaling decisions. Well, now we have even more enhanced autoscaling where instead of just adding more of the same node, we can actually scale vertically. So we can go from an XL to a 2XL or 2 XL to a 4 XL to address those pesky like out of memory problems or those disc fill problems. Then finally um we have the for TCO optimizations in serverless you can actually pick two modes of execution.

24:06 Performance vs Standard

Uh one is called performance mode other called standard mode. In performance mode, you basically get all the, you know, the instant on hot standby nodes that you would get with that you would expect from a serverless product. So you submit a job, nodes are already running, already have Spark running, executes fairly instantly, comes at a cost obviously. Uh and we recently in the past few weeks have done a lot of sort of like refactoring on what that cost structure looks like.
And so more of that to come in the upcoming keynotes and announcements. But from a performance perspective, performance mode is supposed to be about twice as fast as a classic cluster, classic non-serverless cluster running in your environment. Standard mode is supposed to have cost parcel. Now the advantage here again like standard mode gets you all the great stuff that's in serverless. It also gets you zero infrastructure management.
Now one thing that I didn't touch on when we talked about cost was the whole like human capital aspect of things, right? Like engineering time cost of you know is it it's an opportunity cost, right? You can have your engineers focus on performance tuning. You can have your engineers focus on like what's the best configuration to run my 500 jobs or you can let the system do it for you. That's why we're using a managed service, right?
And we can f we can focus our engineering time on other more important business critical things. So putting it all together, we have to address CPU concerns to address some IO concerns. We have liquid clustering that addresses the how do we reduce the amount of work that we have to do part. We have microbatch pipelining that helps sort of reduce that overall end to end latency and make your uh your compute resource usage a lot more efficient.

26:01 Summary

You have incremental MDs with enzyme again how do we do less work? We have enhanced autoscaling for better resource utilization. And then we have standard and performance modes to sort of help you balance that whole TCO decision. So like to end with a quote by another fictional character that happened to have the same name, the first guy. I'm going fast. All right.

Learn more about the Databricks Data and AI platform.