How to Modernize Your ETL with Declarative Pipelines

Summary

This video explains the fundamental difference between imperative and declarative ETL approaches, showing how attempting to apply step-by-step imperative habits to Lakeflow Spark Declarative Pipelines causes friction that disappears once the declarative paradigm is embraced.
Common frustrations such as dynamic date parameters, restricted for-loops, and unwanted full recalculations are not tool limitations but symptoms of applying imperative logic to a framework designed to manage incremental state automatically.
Lakeflow Spark Declarative Pipelines provide built-in data quality monitoring with row-level expectations, optimized aggregations via Materialized Views, and real-time ETL with minute-level latency on serverless compute.

How to Modernize Your ETL with Declarative Pipelines

Did you ever wonder why Lakeflow Spark Declarative Pipelines (SDP) do not allow dynamic parameters or partial recalculations for specific dates? Or why for-loops are restricted in table definitions? Furthermore, how do you choose between a Streaming Table and a Materialized View? These are common struggles users face when starting. While often perceived as tool limitations, these hurdles usually stem from attempting to apply imperative habits to a declarative framework. In this video we will discuss real-world customer examples to demonstrate how to transition from imperative to declarative approach. Stop fighting the framework and start leveraging the declarative paradigm for more resilient data engineering.
Talk By: Aleksandra Chashchina, Specialist Solutions Architect, Databricks ; Tomasz Bacewicz, Product Specialist - Lakeflow, Databricks

Chapters

FAQs

What is the difference between imperative and declarative ETL pipelines?

With an imperative pipeline, developers specify every step—reading sources, filtering rows, handling failures, and managing checkpoints—manually in code. A declarative pipeline like Lakeflow SDP lets the developer define what data sets and transformations are needed, and the framework handles orchestration, incremental processing, and retries automatically.

Why do dynamic date parameters cause problems in declarative pipelines?

Declarative pipelines like Lakeflow SDP manage incremental state by tracking what data has already been processed, so injecting dynamic date parameters bypasses this mechanism and can force unnecessary full recalculations. The declarative solution is to write date-free incremental logic and let the pipeline determine which new records to process.

When should I use a Streaming Table versus a Materialized View in Lakeflow SDP?

Streaming Tables are suited for incremental ingestion of new data as it arrives, while Materialized Views are best for aggregations and transformations that benefit from query optimization and can be recomputed efficiently. The right choice depends on whether the workload requires append-only incremental processing or full recomputation of derived results.

What are the performance and latency characteristics of Lakeflow Spark Declarative Pipelines?

Lakeflow SDP supports a real-time ETL mode on serverless compute that achieves minute-level latency for streaming workloads. This video demonstrates serverless real-time mode as one of the advanced capabilities available to teams modernizing from traditional imperative ETL frameworks.

Full transcript

[00:08] Hello everyone. Thanks for attending this session. So we're going to have some forward-looking statements in our presentation. So make sure you take your decisions accordingly. Um and uh also please complete the
[00:20] survey after this session. That's really important for us. Welcome to the session around how to modernize your ETL with declarative pipelines. My name is Alexandri Chesha >> and I'm Tomas Batvich. I'm a product
[00:33] specialist. >> Yeah, we are based out in Europe and we're here to present you this topic today. So uh we will go through a specific topic around TTL uh where we
[00:46] will present the imperative and declarative uh approach and what what is it and why it is important. uh all along the session we'll walk through a specific example of a ntail pipeline to showcase how uh we can uh
[01:00] manage the this pipeline in declarative way. Before actually going into this example, we will walk through batch and streaming semantics and agree on the terms that we'll be using. Then we will go into the
[01:14] details about uh this specific example and how to uh map it to the declarative pipelines building blocks and we will finish by some other cool features in SDP and hopefully we'll have time for
[01:28] some Q&A. So let's begin. So um maybe some folks of you have already heard about this declarative
[01:41] versus imperative. A lot of people are talking about it right now. So let's understand why it is important. So what is the difference between imperative and declarative? You might
[01:53] have know that we have different programming languages with imperative and declarative approaches. And this also applies to your pipeline um with to your ETL pipelines.
[02:05] So with imperative approach uh you you specify how uh to uh appro how to achieve the result of your table with specific step-by-step instructions that
[02:18] you have to write your own. So for example you would have a code like in Python Scala where you would write okay I need to read uh this source I need to filter rows I need to join with some
[02:30] table I need to write a target handle failures with exceptions and orchestrate everything uh all the technical aspects are actually written manually like uh
[02:42] orchestration persistent to tables and checkpointing for incremental um processing Uh and what is good is that you have full control over business logic and
[02:54] execution itself. So you start with a blank page and you can basically do whatever you want uh with your ETL pipeline. Common examples of imperative pipelines are for example pispark or scala
[03:07] notebooks orchestrated by lakeflow jobs. Uh we also see a lot of for example airflow orchestrated jobs backed by python or spark or any other language. AWS glue is another example and we also
[03:21] have Asia data factory. Um declarative is a really a shift in the in the mindset of developing the pipelines. We don't uh think about how we arrive to the result but we just
[03:34] specify what we want to have as the result of our pipeline. So that means that we only think about the desired end state of the pipeline and everything else will be figured out by the framework itself.
[03:48] That means that we need just to think about okay I have this source at the end I want to have this source filtered joined and aggregated together that means that only the business logic is
[04:00] controlled. So boilerplate code orchestration repetitive patterns everything is handled out of the box by this framework.
[04:15] You get the full control over business logic but you get limited control over execution since everything is delegated to uh this framework. And uh as an example we can site basically any SQL based ETL is naturally
[04:30] uh declarative to some extent since SQL is a declarative language and also we have spark declarative pipelines in both Python and SQL and this will be our main focus for today's session.
[04:45] uh why we're talking about declarative and why it is important. We believe that the future of data engineering is declarative. Uh you might have faced a lot of situations when you for example have a lot of data engineering teams.
[04:57] Each of them are doing their pipelines. So with their own flavor, their own framework maybe following different guidelines. We believe that declarative approach can actually standardize on many things and will let uh teams focus
[05:10] on what they can control the business logic and everything else infrastructure orchestration any boiler boilerplate code will be handled automatically.
[05:28] So um let's take an example of a imperative pipeline and how we can actually map it to declarative approach because this is one of the main uh challenges that we can face when we want to um pick up this declarative approach.
[05:48] Um consider a following pipeline. There's a lot of information but the thing you might want to recall is that so we have classic things like bronze silver gold we have two sources of data uh one is uh just files another one is
[06:02] direct access to a database source uh this job is a scheduled daily job which uh picks up the new data from the files
[06:14] uh inserts it uh appends it to the orders uh bronze table does some cleaning and then enriched it with a dimension table to get some aggregations at the end. So we have orders data as a
[06:27] fact and customers data as dimension and at the end we want to get uh some daily revenue by customer. So what are the challenges with this approach? It's working but there are
[06:40] some caveats. Um the first one is that when we want to even if we want to process the orders incrementally, we still have to scan all
[06:54] the files uh in the bucket even if we provide that current date filter which we pass at every daily run which makes it very inefficient.
[07:10] Um for the dimension table we want to um to track the changes with the slowly changing dimension and for this we have to uh provide a specific merge into statement to uh update the records accordingly and this can become really
[07:22] complex because we need to maintain the deletion of uh of rows in in the source table and also there can be some performance issues if we do it on our own.
[07:34] Um if we have some data quality checks we'll have to do them manually with some work closes. We don't necessarily have all the statistics of the data which were uh which was removed or passed. So
[07:49] we have to do something uh manually. Um you may mention that in this particular pipeline we pass a current date as an input parameter every day and
[08:03] this is a challenge because uh it will not be able to handle any late arrival data. So if data arrives later than the date it is supposed to arrive, our pipeline will not be able to take it
[08:15] into account. And finally the final aggregation of daily revenue by customer we have to recalculate it every time. So if you
[08:28] have years of data uh with different customers and revenue like over time this recalculation will be more and more complex and will take more and more time.
[08:43] So before we move move on with this example to understand how can we actually map all these blocks into declarative approach let's agree on some batch and streaming semantic terms. So uh you may consider that there are
[08:56] two dimensions of how to how to process data. First we have um the incremental or full load which means how actually we process data. Do we process it
[09:08] incrementally? That means every time we process only newly ingested data or full load where we reprocess all the history all the times. And then another dimension is the
[09:20] frequency of processing. Uh do we process it few times a day at a at a specific schedule? This is a batch. Or do we process it in real time? This is
[09:32] streaming. This is typically some jobs which run 24/7 and process data uh accordingly and then you can mix those dimensions. So you can have a batch job
[09:45] which runs let's say once a day but processes data incrementally. This is the case of our imperative pipeline. You can have a batch job which runs daily but processes all the data all the
[09:57] time. This is less performant but a bit more straightforward. Um streaming is by nature incremental. So when we have a streaming job running 247, we specifically ingest only newly
[10:11] data. Uh streaming and full load, I have never seen this. So if you do it, please don't. I think it's really an overkill. So all this you can actually combine and
[10:23] use in declarative pipelines. Um actually the main challenge uh is between the full load and incremental processing. Full load is uh very simple.
[10:37] It can you can just do your pipelines as is. You don't have to manage any watermarks, stateful operations and checkpoints. Uh and you are sure that the results are always correct. Remember
[10:49] for example our current date parameter is a problem. It's it makes our pipeline incremental but actually whenever there is a late data our pipeline will not take it into account. The the cons of full load is that it's
[11:03] not efficient means every time we will reprocess the whole uh amount of data. For some volumes it it might make sense but if we're talking about big big volumes it might become a problem. Um you still can do for loading data
[11:17] bricks you have things which are called materialized use. You can also use uh some standard spark operation like spark read and spark write in order to do this.
[11:29] Uh incremental processing is better. It's more efficient because we only process the new data and this is actually one of the common access of performance improvement that we can already do very easily in the pipeline.
[11:44] The problem is that processing logic can become very complex. Think about stateful processing when we need to join or aggregate streaming sources. We need to think about things like watermarks and other stuff which makes the pipeline
[11:57] even more complex and also out of order and late arrival data is a very important issue which arrives whenever we want to do incremental processing.
[12:09] Um in lake flow products we have uh things which are incremental out of the box so that all this complexity is hidden from user. uh these products are llo connect so it's specific feature
[12:21] which can connect to uh outside sources like databases salesforce service now dynamics 365 for example and ingest data incrementally
[12:34] and in SDP you have some building blocks like append flow a play changes flow streaming tables and sync whenever you use these building blocks you you may be sure that you are processing data incrementally
[12:48] And finally outside of SDP we have things in spark structure streaming which also um allows incremental processing. >> Okay. Over to you. >> Cool. Thank you. So what I'm going to do
[13:03] now is talk about how we can map that imperative pipeline that Alexander was talking about earlier into a declarative fashion specifically obviously using Spark declarative pipelines.
[13:15] And so first, you know, we want to point out all those different steps where uh Alexander said, you know, there's some inefficiencies in the imperative pipeline. And so the first thing that we want to do is we want to stop telling
[13:27] the pipeline what day it is. And so if you look at the imperative uh code that we're writing, we typically have to do a filter for example. So then maybe we're loading data every day and obviously we want to load only today's data into our
[13:39] pipeline. And so we need to put in that filter in a imperative way to be able to actually you know get the uh information into our pipeline. Um obviously that
[13:51] typically means you're having to manage what is the last day that you've brought in into your pipeline. And if for example you rerun this pipeline twice in one day you might get duplicate data because now you've run you've actually
[14:03] ingested the same data twice. But with the declarative mode you we're using the format cloud files as you see and that's our autoloader functionality which actually keeps track of all the files
[14:15] that have already been read from let's say your S3 or ADLS buckets and so uh then you can it's item potent so that means that you can keep rerunning that pipeline and every time it would only
[14:28] pick up the files that are new in the source system that it's reading from and so it's fully tracking everything for you in terms of the files that it's already read and it doesn't you don't need to worry about keeping track of the
[14:42] date filter anything like that yourself. So makes it a lot easier. And so that's the first thing. The next one is a really big use case for SDP and
[14:55] that's specifically that merge use case that you saw from Alexander's pipeline. That's where we typically look at a dimension table. For example, we have customers that are coming in and maybe once in a while we have customer information that's being updated around
[15:08] their address or their email and we want to track that, right? Whether it's change data capture SC type one or type two, we have to bring in the newest record and make sure we get a current
[15:21] representation of our customers. And so with the imperative uh mentality, we would typically write a merge statement. And in this case I have a very simple example of this merge statement that we have to write. But in reality to get
[15:34] this to be a robust production ready type uh process you have to write many many lines of code to be able to handle things such as out of order events or deletions in a customer uh you know
[15:46] let's say a customer got deleted from the source system. And so ultimately you're just writing a lot of code to handle for a lot of different edge cases and also to make sure that your actual merge statement is optimized under the
[15:59] hood that makes it to make sure you're not reading all the data from the target table whenever you're actually doing this merge. But with a declarative mode it's a simple statement create auto CDC
[16:11] flow and within it you just pass in a few parameters. So specifically the target table, the source table that you're reading from the keys that you're merging on and then you see the sequence by uh parameter that sequence by
[16:24] parameter is used to make sure that we can track the uh the out of order events. So for example, if we have uh customer information that came in today and then now uh or let's just say we had
[16:36] some customer information coming yesterday, today we have more customer information coming in from that s about the same customer but it's actually data that was relevant two or three days ago. We want to make sure we either ignore it
[16:48] when we're using sedd type one capture or we want to make sure that we create a new row for it to make sure we track that change over time but making sure that it's also still an inactive row. So auto CDC is able to handle that for you
[17:02] automatically. And obviously the last parameter there stored as sedd type. You can easily switch between sedd type one and sedd type two. If you want to track the history obviously you would do the sedd type two mode. And the great thing
[17:16] about this is it's also optimized for you under the hood. So whenever there's new data coming in being ingested, we're only reading uh the target table that is relevant to the data that's being ingested into the pipeline. So you don't
[17:30] have to worry about trying to optimize anything yourself to make sure that if let's say you have billions of rows on that target table, you're not reading that entire table every time you do this merge statement.
[17:44] Another great aspect of SDP is the fact that we have automated data quality. So in an imperative fashion as you saw with Alexandra's pipeline, we had to uh determine how do we handle different types of uh let's say dirty data. Maybe
[17:57] you have nles uh for an ID or we have an amount that's shouldn't be negative but is we have to find ways to take care of that through an imperative fashion by writing different types of rules. So in this case we're checking to see if an
[18:10] amount is below zero and if so we raise a value error. So that pretty much fails the pipeline. We can do the similar thing where we just filter out any rows which are null within a null order ID.
[18:23] Um but there's no real standardized process of of uh tracking these types of uh constraints within an imperative fashion. you kind of have to figure out, you know, what works best and there's no
[18:35] out ofthe-box kind of data monitoring to understand how many rows did I drop or which ones failed and so on. But with the declarative mode with SDP, it's a simple decorator that you put in and you
[18:48] can set up different types of behaviors. For example, you can expect or drop a row or you could expect or fail the pipeline as a whole. And there's a third one that I don't show here which is just warning you if let's say there's a certain constraint that didn't you know
[19:01] wasn't met you'll get a notification but it'll still let the row pass through. What's great about this is it's a standardized way to be able to handle these type of data uh you know dirtiness and and finding ways to make sure you
[19:13] clean it in let's say your silver layer. Uh another great thing about this is that it has out of the box monitoring. So if you think about that imperative uh mode if let's say you wanted to
[19:25] understand well how many rows did I drop or how many rows did I ignore because they weren't meeting some constraints you have to write some extra logic to maybe you know actually be able to report on that but with uh declared pipelines there's an out of the box uh
[19:37] way to just be able to see all this information through a table that's automatically generated for you as you run this pipeline.
[19:49] And the next one is a really big deal in production uh pipelines is to make them robust. You want to make sure you handle out of order data uh you know properly right and obviously this is kind of an overarching arching theme across all the things that we were just talking about.
[20:02] But in an imperative pipelines manner if let's say you found some out of order data that was coming in many days later you now have to go in and manually fix every part of the t of the pipeline to include that out of order information.
[20:15] But with declarative pipelines, all the different layers are covered by our functionality with what you see in SDP. Specifically, the autoloader capability always is able to detect late arriving files because it's looking at the file
[20:28] level of what data hasn't picked up yet whenever you're running the uh the pipeline. With the auto CDC function, you saw there's a sequence by uh parameter to make sure that you're handling out of order events correctly.
[20:41] And then the next the last one is I'll talk a little bit more on the next slide which is materialized views. Materialized views are typically representations of your source tables in the materialized view itself. Which means that if let's say some late data
[20:54] arrived in a source table when you do a refresh on the materialized view it will handle that uh late arriving data to make sure it represents it correctly in the current state when you do the refresh. And the way it does that so
[21:08] let's imagine that we're trying to do an aggregation. So what Alexander's pipeline was doing was uh aggregating on customers by date and trying to understand the revenues that we're getting by by customer. For example,
[21:21] in the imperative mode, you either have an option to overwrite the entire table every time because you have to do that aggregation fully and then materialize that table through an overwrite. Right?
[21:33] If let's say you have some late arriving data, you need to fix some old aggregations that have existed in that existing table. There's another way to handle this which is to do a merge statement. We didn't show it here. Uh but that merge statement also can be
[21:45] fairly complicated to do because you have to figure out well how do I actually aggregate the data that's historical and maybe arriving late for example. Obviously an overwrite a full overwrite
[21:57] is a really expensive uh computationally expensive operation. So you want to try to avoid that. So in the declarative mode we have what we call materialized view. I'm sure you guys have heard this term before. We have a similar concept,
[22:09] but what's great about this is as you see, you're using spark readad.table to read an entire source table. And that's what practically it looks like. You're reading the entire source table. But in reality, what's happening under the hood
[22:22] with a materialized view is it actually understands which components of the source tables have changed and then actually updating that materialized view incrementally. Anyway, so you're using batch commands, but it's still using an
[22:35] incremental update under the hood to make sure that you're reducing your computational overhead as you're refreshing these tables. And one other thing I wanted to talk about is a fairly new feature called
[22:48] replace flows. So imagine if we are trying to update some data in our tables, but maybe a specific component. So in this case uh let's say we have data that we want to um everything
[23:02] that's late older than 7 days in our table we just want to keep the way it is and all the all the data for the last seven days we want to update and that's is this is a common scenario where maybe
[23:14] you have like updates in some windows especially that are moving time windows. In the imperative mode, you might have to go and delete that that set of data first and then uh filter out that data
[23:27] from your source table and then ultimately append that append the data that you're reading from the source table into that target table. So it's a multi-step operation to get this to all work. But in the declarative mode, you can just do a replace where and select
[23:40] the filter that you want to replace where on the target table and then that filter will also be actually pushed down to the source tables that you're reading from so that you're only reading the the
[23:52] necessary information from your source tables and then it incrementally updates the target table based on what it read, which is the last seven days of data. The great thing about this is that it's also incrementally refreshed, which
[24:05] means that it's not actually updating all seven days of data. If, for example, only one day has changed within that seven days. So, it intelligently discovers, hey, there was only one day of data that's changed in this 7-day window that the user set, I'm only going
[24:19] to update that one day of data, which again reduces that computational overhead that you typically have when you're trying to do do these types of updates. And it's all doing it in an intelligent manner for you.
[24:31] So with that said, I'm going to quickly show you guys just a quick SDP demo of um what uh SDP looks like and how it uh you know discovers things
[24:43] for you in a declarative mode. So uh this is the IDE that we have. So if you guys haven't tried STP, definitely go and uh into your data bricks workspaces and try it out. We
[24:56] have a very a full-fledged IDE that lets you you know generate these types of pipelines very quickly and easily and this is a typical DAG that you that's created from an SDP pipeline. As
[25:08] you see here we have bronze layer. We're reading in some data into a silver layer and we can see here that for example I can click on this and see what are the constraints that haven't been met as I'm actually write uh running this pipeline.
[25:22] So in this case we had 17% of the data didn't meet the constraint of amount that was uh not negative and so in this case we're just uh we're we're allowing it to go pass through
[25:34] and then we're doing a join and then ultimately we'll see that there's a materialized view at the end where we're doing the aggregation that I just discussed or the replaceware functionality which is also doing a similar aggregation but it's just using
[25:46] replace wear. The whole point is though is that you can see that it's incremental. So it was able to determine that there was some small changes on the source system and do an incremental update instead of a full overwrite again
[25:59] reducing that computational uh overhead of actually doing this refresh. And similarly it does the same thing with the with the replaceware functionality. The great thing about this because it's
[26:11] a declarative pipeline is that all of these dependencies have been figured out for you automatically because what I actually did was I had a Python file for the bronze layer where I you know defined those bronze tables that I just
[26:23] showed you. I had a SQL file for the silver layer when I defined the different tables uh including the auto CDC flow and then a gold uh layer for the materialized views in Python. Again,
[26:37] what's cool about this is you don't have to worry about, you know, trying to orchestrate this on your own. If you imagine doing an imperative framework, you might have three different files and now you got to go in and determine, well, what is dependent on what? Um, and
[26:50] then you have to figure out, well, are is the data in my silver layer that I'm reading uh actually produced by my bronze layer correctly? Are the types the same? Are the columns the ones that
[27:03] I'm expecting to read in the silver layer? Example, declarative pipelines is able to figure all of that out for you automatically. And what's great about this is that you can do something like a dry run here. So, what I'm going to do is it just checks to see if the
[27:16] dependencies all work. Actually, what I meant to do, oh, sorry. What I meant to do was put in a column that doesn't actually exist. So, fake column for example. And if I do this, I'm just
[27:28] going to stop this. Okay, now it's stuck. Apologies.
[27:40] exist in my bronze layer because that's where I'm reading it from. And it'll obviously give me an error saying that fake column does not exist. So it's looking in and making sure that the dependencies actually exist, helping you
[27:52] orchestrate this pipeline on, you know, automatically without you having to really do anything uh on your own. Another feature that I wanted to quickly touch on is, you know, Genie is a big
[28:04] theme for us. Obviously, we want to make sure that AI is embedded in everything that you're doing. So, when you go in and want to start using Genie to create your own pipelines, we have an agent that's able to create your entire
[28:17] pipeline for you just by, you know, prompting it here. What's nice about this is you can customize Genie for your use case as well. So, if I go into Genie here and I click settings,
[28:29] I can now see there's uh, you know, a few few things I can do such as connect to MCP servers, uh, load some user instructions and so on. But I specifically want to show you guys the open skills folder where if I go into
[28:42] here, I've actually set some SDP basics and these this file actually provides some guidance to the Genie framework to make sure that when it's generating code for us, it is following c certain
[28:55] standards and practices that we might be using at or our organization. So in this case I'm just telling it well tables should be named with bronze datab bricks whatever the table name is. Maybe we
[29:08] want to talk about what kind of tables we want to use for what type of transformations. Maybe we want to make sure that genie is also actually writing comments as we're generating these tables. So we can provide those
[29:20] instructions for Genie in a skill. So that now you can get customized code that's being generated for you as you're using Genie instead of you having to fix all the stuff that Genie wrote for you and it's not meeting the standards that you expect.
[29:34] So that was just a really quick demo. Hopefully it kind of gave you at least an understanding a quick understanding of what STP looks like, but I definitely would recommend going in and checking out the um pipeline. If you ever want to
[29:46] uh you know try it out on your own, you can just go to new and select ETL pipeline and then that will get you started with a new pipeline. With that said, I'm going to go back to the slides.
[30:03] And so I had a few other quick things I wanted to touch on uh around the the functionality of SDP and why, you know, it's a really great framework to get started with your ETL pipelines. Like I mentioned earlier on the demo,
[30:16] automatic dependency check. So the dependencies are automatically inferred for you. And that's what's great about a declarative framework is because you have the established understanding or the compute has the established understanding of what it's what each
[30:29] query is actually returning. It's able to determine all those dependencies for you automatically. And like I showed you the dry run function, you're able as you're developing to make sure that it logically makes sense before you actually run it. in an imperative
[30:42] fashion. You might have to create each table first, make sure that the next stage actually connects to it correctly, orchestrate it at the end once you want to put it into production. Obviously, just a lot of steps that we're trying to avoid doing when we're using STP.
[30:56] Another big thing is enhanced autoscaler. So, this is uh honestly a function that's been available for a few years now with STP. But the reason why I wanted to talk about it real quick is because STP is not just built for you
[31:09] being able to use an interval trigger where maybe you run this once a day or once an hour. You can easily decide to run this continuously so that it now you know maybe you have a really fast SLA you need to meet and so you can run it
[31:23] constantly instead uh and change how you want to process your data. What's great about the enhanced autoscaler is that it's built for streaming. So that means that it's not only looking at utilization of your CPUs on the node on
[31:37] on the nodes, but also on backlog. So it can see well, how much data do we still need to process? And maybe if there's a big backlog, it will aggressively scale up to make sure it can handle that backlog. It also heavily scales down
[31:50] once you're not using all the nodes, all the compute within your nodes. This is something that's often lacking with a lot of autoscalers historically. I did want to point out that typically this isn't as important anymore with
[32:02] serverless. Serverless, we're actually adding a whole bunch of other features to autoscaler to really make sure that your clusters are as granularly uh let's say sized to your data volumes as possible.
[32:20] Like I mentioned earlier also SDP gives you an out-of-the-box event log and that event log you can easily query within DBSQL for example or lakehouse now is what we call it. Um or you could publish it to uh Unity catalog and create entire
[32:32] dashboards out of it. And then you can get notifications. So for example, if all of a sudden a 100 rows failed in your pipeline, you can get a notification to make sure uh you can fix that pipeline and you know uh figure out
[32:45] why why you're getting all those errors for example. So all of this monitoring comes out of the box without you having to write any extra code to actually generate these event logs. And then lastly, something that you
[32:57] might have seen mentioned on our uh keynote today is uh SDP realtime mode. So real-time mode we have it in structured streaming we've now also moved it to uh SDP as well and maybe
[33:11] just to quickly like context set here about what real-time mode is. Historically we've been using microbatch mode which means that you know we're taking small batches of your data processing it in streaming fashion and typically that's great for streaming
[33:24] ETL. you're looking at latencies of minutes uh typically uh to or sorry seconds to to minutes but nothing subsecond really but there's often a
[33:36] whole set of operational use cases you might want to tackle as well for example imagine if you're like a payment platform you need to detect fraud on a real-time basis you don't want someone to make be making a transaction on your platform and you need to wait a few
[33:48] seconds uh to be able to understand whether that's a fraudulent uh transaction so you can use real-time mode in those sub-second type of latencies uh using uh using SDP. So
[34:00] that's something new that you should definitely check out if you have some of those types of operational use cases. Um and yeah, that's kind of it. So thank you so much. Uh we have
[34:13] we have seven minutes left. So if you guys have any questions, let us know. Otherwise, uh thank you so much for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.