Declarative Pipelines: What’s Next for the Apache Spark Ecosystem

Summary

Declarative Framework: DLT introduced a declarative framework that abstracts away orchestration complexity, making it easier to build production-grade data pipelines.
Spark Evolution: This represents the next step in Spark's evolution toward openness and standardization, building on a decade of simplifying data processing.
Standardized Approach: Declarative pipelines set a new direction for the Databricks Data and AI platform ecosystem, opening doors to community momentum and cross-platform compatibility.

Declarative Pipelines: What’s Next for the Apache Spark Ecosystem

Talk by Michael Armbrust, Distinguished Engineer, Databricks ; Sandy Ryza, Software Engineer, Databricks
DLT has made it dramatically easier to build production-grade pipelines, using a declarative framework that abstracts away orchestration and complexity. It’s become a go-to solution for teams who want reliable, maintainable pipelines without reinventing the wheel. But we’re just getting started. In this video, we’ll take a step back and share how DLT fits into a broader vision for the future of data engineering pipelines — one that opens the door to a new level of openness, standardization and community momentum. We’ll cover the core concepts behind declarative pipelines, where the architecture is headed, and what this shift means for data engineers building procedural code. Don’t miss this video — we’ll be sharing something new that sets the direction for what comes next.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are declarative pipelines?

Declarative pipelines let you define what data transformations you want instead of specifying the procedural steps, making development simpler and more maintainable.

How does DLT fit into Spark's evolution?

DLT builds on a decade of Spark simplification, adding declarative abstractions that reduce complexity and make production pipelines more accessible.

Why is standardization important for pipelines?

Standardization enables better interoperability, community adoption and long-term sustainability of the Apache Spark ecosystem and data engineering practices.

What is the impact on data engineers?

Data engineers can build more reliable, maintainable pipelines faster using the declarative approach without reinventing orchestration logic.

Full transcript

`0:00` The evolution of Spark

Okay, thank you so much for coming. My name is uh Michael Armbrust and Uh we're here to talk about declarative pipelines in Apache Spark. But before we get into that, uh, there's, you know, going to be a keynote tomorrow where we actually announce this. And since we haven't seen that already, I decided that you guys could get an exclusive preview of the announcements that are coming tomorrow in the keynote. Um, so this project, declarative pipelines, is kind of the culmination of over a decade of simplifying Apache Spark.
So when I when I first spoke on the stage at what used to be called Sparks Summit almost 12 years ago, uh what we had was Spark RDDs, uh which were kind of gamechanging at the time. This idea that you uh basically instead of having to wrangle thousands of machines and deal with failures and all that other stuff, you could just write these simple functional programming constructs map and reduce and you know that would let you process large amounts of data.
This was the technology that kind of unlocked Google using commodity clusters and kind of ushered in this uh era of big data. But when I look at this code today, what used to seem simple is actually pretty complicated. So, you know, something as simple as computing an average uh requires you to think in this weird functional way where you have to kind of map things. You have this you have one for the count and revenue for the sum and then you have to actually write out the equations and you have to think about things like partial aggregation.
And so one of the first things we did with Spark was try to abstract away some of these complexities for very common functions like doing aggregations, doing joins, doing filters, all that kind of stuff. And so the first addition that we had to Spark uh to Spark was Spark SQL. And with Spark SQL, we made the query execution declarative. So instead of having to spell out exactly how an average is computed, instead all you do is say, I would like you to compute an average for me.
And the system has a an optimizer and a query planner that figures out exactly what code is going to run and underneath the covers it all compiles down to RDD still. Um, and so, you know, that was great, but it actually still had a couple of gaps. Uh, it turns out that you really never run a query just once and your data is always changing. And so running the same SQL query over massive growing data sets over and over every single day was incredibly inefficient.
And so the next thing we added to Spark was structured streaming. And so with just the addition of the stream keyword, instead of computing that report just once, we are now incrementally computing that report as the data changes. And now the system is responsible for figuring out what data is old, what data is new, making sure that you transactionally process it even in the case of failures. checkpointing where you're going.
And so that dramatically simplified the problem of doing incremental processing. But once we had structured streaming uh you know what we had a lot of people doing was they were using it for these large scale ETL problems. We were very good at scaling. But the problem was you'd very quickly end up with millions of paret files stored in S3. And that wasn't a great experience. I think some of you might remember those bad old days where you're kind of dealing with millions of tiny files and you're having to collapse them and you're having to try to do that without dropping data, duplicating data.
And so the next thing we added to this stack was Delta Lake where finally storage became transactional. Instead of thinking about files, now you're thinking about tables that have a schema and they have highle operations like update, delete, and merge and optimize that can run on top of them. And so that brings us to today where you can do a lot of things if you're an expert in Spark SQL and structured streaming and delta.
But still taking this code and turning it into a production ETL pipeline is pretty difficult. At the core you have what is you know still essentially just a SQL query. This thing where you're using your domain knowledge to transform data so you can get value from it for your business. But in order to take this and not just run it once, but run it reliably so it can power the business requires you to worry about things like checkpoints.
How am I going to manage them? How am I going to store them? How do I make sure I don't lose them? Uh you need to worry about retries. When the stream fails, how do I restart it? What if the cluster is in a bad state? Do I need to restart that cluster? Uh you need to worry about things about dependencies because it's never just one table. It's many tables, hundreds of them that depend on each other and they need to be created in the right order and they need to be evolved in the right order when new columns are added.
Uh, additionally, when you start having thousands of tables, some of them need to run in a certain order and some of them can run in parallel in order to get the best utilization from the cluster. So, you end up managing things like thread pools. Uh, and then you end up having large pipelines. You don't want it to be just one gigantic notebook. So you end up having lots of different files to manage different parts of the pipeline.
You end up building like a compiler that can bring them all together and that enables you know proper data engineering with things like CI/CD. But as you can see I'm now spending all of my time on this undifferiated heavy lifting that exists around the query which is the thing that's actually allowing me to get value from my data. And so that is why we built this technology four years ago that we were calling Delta live tables.
And this is a you know a testimony from one of our customers where basically what they found was this by using DT they were able to hide the complexity of working with all of these different tools under this simple intuitive declarative model. And so now it's time to take the next step. You know now that we've kind of battle tested this idea inside of data bricks. That's why I'm super excited to announce that we are contributing Spark declarative pipelines to Apache Spark.

`5:47` Announcing declarative pipelines

And so now as you can see here in just a few lines of SQL we're doing incremental ingestion. We're doing uh you know computing gold tables uh which I think is super powerful that this kind of works all just by writing a few lines of SQL. So now we're turning anybody who knows SQL into a first class data engineer. And so with that I'm gonna hand it over to who's been leading the effort to actually put the code into Apache Spark.
Take it away. Thank you. I'm an engineer at Data Bricks and a committer on Apache Spark. Uh, as said, I've been leading the effort to bring declarative pipelines into the Spark project. I'm going to spend the time I have here basically drilling a level deeper uh into what talked about, why we built declarative pipelines, and how it works. If you're already familiar with DT, a lot of what we'll cover in this talk will likely be familiar to you.
But I'll end um at the end I'll talk a little bit about the differences and also happy to answer any questions. Um before we get into the gory details of declarative pipelines, I want to cover a few broader questions. So the first one is what exactly do we mean when we say declarative? Second, what exactly do we mean when we say a pipeline? And third, why make pipelines declarative? How do data engineers benefit by building pipelines in a declarative way?

`6:59` Defining data pipelines

So a declarative paradigm means that developers focus on describing the results that they want, not the details of how those results are achieved. Spark itself is already declarative in many respects. So for example, here's a basic Spark structured streaming program that reads from a Kafka topic, applies some basic data transformation, and writes to a table. There's only a single line in this program that actually kicks off any processing.
Most of what's going on here is describing a data flow. We're saying what table we're populating and what query defines the data that goes into that table. After all the code in this first section is executed. We haven't actually processed any data. We've just built up a plan for what kind of data processing do we want to occur. Finally, when we run start, Spark looks at our plan as a whole and figures out the best way to execute it.
And this works great. It handles all the gory details of how to do this execution so we don't need to think about them. And so Spark is declarative at the level of an individual query. But a single query is not a data pipeline. What's a data pipeline? A data pipeline is probably not a new concept to most of the people in this room, but I'll give a definition of it. Um, uh, so we can have a common language for talking about it.
A data pipeline is simply something that produces and updates a set of data sets. So it might ingest data from an application into a lakehouse. It might prepare tables for reporting or dashboarding. Um, it might create features that'll be used to train an AI or ML model. Data pipelines aren't one-time affairs. They're not just responsible for creating tables. They're also they also keep them up to date over a period of time.
Data pipelines have two fundamental building blocks. The first is the data sets like the table and views that the pipeline is responsible for producing and keeping up to date. The second is the data flows. These are usually queries that describe how to compute the contents of a data set. Often these flows will read from other data sets within the pipeline. And this gives data pipelines this kind of graphical structure depicted here and often seen.
So why make data pipelines declarative? The reason we built this framework is that developing and operating data pipelines is much harder than it needs to be. So it requires thinking about and managing gory details that are often unrelated to a pipeline's business logic. This is the case even with Spark, which was designed to save data engineers from many of the gory details of transforming their data. To illustrate some of these difficulties, I'm going to walk through a basic data pipeline.

`9:21` Challenges in manual pipelines

And so just a warning, I'm going to show a lot of code here. I don't expect you to actually read it in detail. I'll highlight and point out the relevant pieces to that are actually worth paying attention to. So let's imagine instead of our single query that updates a single orders table, um we have something slightly more complicated. We want to update three different tables. We want to store the orders data that we get from Kafka in its fully unprocessed form in a raw orders table.
We also want to maintain a table of customers and we'll pull our customers in from a CSV that's dumped out by our application database. Um, finally, we want to have a clean fact table of orders. Um, which takes our raw orders, cleans them up, and joins them with our customers data. At the top of this slide is the code for populating each of these tables. Um, and at the bottom is a representation of them and the flow of data in between them.
So notice that instead of our entire program being a single plan like before, we've now got three different places where we're triggering execution and this means that we as the developer instead of spark or the framework are responsible for planning when these queries get executed and this is where trouble arises. Um so for now let's assume we want to execute our pipeline in a discrete non-ontinuous way. Uh, by that I mean each time we run our pipeline, we want it to process all of the data that's available in our sources just at that time.
And when it's done, we're going to stop. We're not going to, you know, have it run in forever. The simplest way to do this is just to create a script that executes our queries in order. Um, so first we overwrite our customers table. Then we update um our raw orders table with the latest orders from our Kafka topic. And finally we update our fact table. So, this will work okay, but there are a few things about it that are not ideal.
Um, uh, first, it's definitely not the fastest way to do things. One thing we might notice is that the step where we update the customers table and the step where we update the raw orders table have no dependency on each other. So, this means that it's possible to run those steps at the same time. This could make our pipeline much complete much more quickly. So, we could update our pipeline code to multi-thread this.
So here um we're adding a little chunk that wraps each of these steps in a function uh starts a thread for each of the functions and then we wait until the initial threads are complete before the final step of the pipeline. If we later want to add more steps to our pipeline or change our queries. However, this multi-threading code can get very complicated very quickly. So here's an example of a slightly larger pipeline with the multi-threading code that's needed to get it running in parallel.
Um please don't try to read all of this. The point is it's a lot. It's also very brittle. Uh if we make a change to one of our queries, it can mean we need to shift all of this around. It's very easy to accidentally end up with code that fails to respect our data dependencies. Um and by that I mean which starts the query for a downstream table before we finish updating the data in an upstream table. Um this causes us to end with unexpectedly stale data often in sort of hard to detect um ways where we find out why people getting mad at us.
Um but suppose we get that all figured out. So our pipeline can hum along nicely um as long as all the queries are completing successfully. What if there is an error? There are a few different kinds of errors that pipelines can encounter. So one category is errors that are catchable at query analysis time. So for example, here's a subsegment of our pipeline and it has a bug in it. We're writing a column named customer ID to the customers table and then in a downstream query we try to join on that column but we accidentally call it customer identifier.

`12:57` Handling pipeline errors

So this is a very easy mistake to make but not an easy one to catch often. Um and we won't even discover this issue until it's time to run the downstream query. If the upstream queries take a long time to complete this could be pretty late in the game. We might wake up in the morning to discover that our pipeline has not completed and we need to restart it from the beginning. Theoretically, if we had a way to analyze these two different queries together before starting either of them, then we could catch this much earlier.
Of course, not all errors are catchable at analysis time. Some errors can't be caught until we actually try to run a query. So, for example, if we're dividing by the value in a column and that column has some unexpected zero values, then all our query is going to fail when it encounters those values. Or alternatively, if our query is reading data from a data source like Kafka and the Kafka service we're depending on goes down temporarily, then our query is going to fail.
So at least for this uh second kind of error error um like a transient error, the query will often succeed if we just try running it again. Um which is really nice if we don't want to get paged in the middle of the night. So ideally we'd like to include some retries in our code. Here's our pipeline with retry handling added. I'm not going to go into the diesels of the code, but suffice to say, our simple little script has become quite long.
So that was automatic retry handling, but we often want to do manual retries as well. Um, we wake up in the morning to discover that one of our the steps in our pipeline has failed. Um, it turns out to be a simple fix and we want to restart the pipeline, but only execute the steps that did not complete successfully because the other ones um did fine. And of course, we can add that to our script, too. Um, so let's give it some command line arguments that determine which steps to run and only run those steps.
Um, so our already monstrous code has gotten even bigger. U, at this point we might pause and ask why not use an orchestrator like Airflow to handle some of this. Um, and so this is what some of the code might look like. After defining our queries, we define an Airflow DAG with a task for each query. And then we tell Airflow about the dependencies between those tasks. And that definitely helps us with some of our problems.

`15:12` Limitations of orchestrators

It'll handle um some retries and some parallel execution. Uh but because Airflow doesn't actually understand the queries we're trying to run, there's only so far that it can go. So for example, Airflow doesn't know what kind uh what tables each of our queries uh is reading from. If the query that generates our fact orders table reads from our customers table, we want to wait for the customers query to be done before the fact orders query starts.
Um but we need to explicitly tell Airflow about each dependency. Um, this is very easy to get wrong. I've personally found this to be a huge source of bugs uh that are hard to discover u until someone yells about stale data. Second, Airflow can't help us at all with the kinds of analysis errors that we looked at earlier. It doesn't know what columns each query produces and it doesn't know what columns each query uh expects.
So, we can't do anything to match those up. Um, as a general purpose orchestrator, Airflow can also be fairly heavyweight for res, uh, presenting every transformation within a pipeline. Task startup can be slow. Uh, and the local development experience is rough. There's a reason that lighter weight frameworks like DBT are popular. Um, and then orchestrators like Airflow are an especially poor fit if we want to run our pipeline continuously.
So, so far we've imagined that we run our pipeline on just the available data and then stop. But let's now talk about the other main way to run a data pipeline which is continuously just keep processing new data as it comes in. This allows achieving much lower latencies uh which can be very important for some applications. So if we go back to our pipeline uh it has both batch queries and streaming queries. This raw orders table is populated uh by a streaming query that pulls in data from Kafka and appends to the table.

`16:45` Continuous pipeline execution

And the fact orders table is populated by a streaming query that reads from the raw orders table. Ideally, we'd like to just run both of these individual streaming queries continuously, which is something that Spark structured streaming allows us to do. The query populates our customer uh the query that populates our customers table is a batch query. So, we can't run that continuously, but we can launch it on a short regular cadence.
Um, and so here's some code that implements this. Uh, it starts a streaming query that continuously appends to our raw orders table. Then it starts a streaming query that continuously appends to our fact orders table. And lastly, it starts a thread that refreshes our customers table every 15 minutes. And this seems great, but unfortunately there are some problems. Uh, so let's imagine it's the first time we're running our pipeline and the tables don't exist yet.
Uh we run our script and it first starts the streaming query that populates our raw orders table. Uh this is an async operation. So the script immediately proceeds to start the streaming query that populates the fact orders table. Um the first thing that second query is going to try to do is read the latest data from the raw orders table. But remember we didn't wait for anything to happen after starting the first query.
So the raw orders table it might not even exist yet. Um and this will cause an error. This isn't only a problem on the first run. So it can happen anytime we evolve our schema. If we update our pipeline code to add a new column to the raw orders table and we up uh update a downstream query to use that column, we can encounter the same race condition. So what do we do about this? Uh one option is that we have a sort of step at the beginning of our pipeline that first writes to all the tables to make sure they exist and have the right schemas and then after that's done then we start the streams to run continuously.
Um, but now we're back to this world with lots of code, bloated implementation, uh, brittle. Whenever we make a basic change to a query, we're often going to need to update code in a bunch of different places. Um, and if we forget to do that, we're liable to get errors, uh, when our pipeline fails in the middle of the night for some theoretically foreseeable but hard to catch way. So, we're encountering these challenges at every turn.
Maybe it's worth taking a step back and asking, is there a better way? Remember how simple life was when we just had the single query? We built a plan, we executed it, and Spark handled the rest. Um, and this, I know it took a little while to get here, is what declarative pipelines is trying to do. So, as I mentioned earlier, declarative pipelines is a new component of Apache Spark built on top of Spark SQL and Spark structured streaming.

`19:25` Introducing the solution

The central idea of declarative pipelines is to extend the Spark declarative model a level higher. Instead of just specifying a single query before executing it, you specify the entire pipeline before executing it. So here's an example of what this looks like for the pipeline that we've been wrestling with. Um, again, I don't expect you to read all this code yet. We'll go through it in more detail a little bit later, but at a high level, the way it works is that you write code that declares your entire pipeline.
Uh, and then you can just tell the framework to run that pipeline as a whole. The code doesn't include any starts or save as tables. um or collects. It's just a set of declarations. Declarative pipelines has two main subcomponents. The first is a set of APIs for defining pipelines. And so these APIs are built on top of Sparks APIs. The initial APIs that we're working on are Python and SQL, but we want to add Scola and Java soon after.
Through the magic of Spark Connect, we'll also be able to support APIs in any language that has a gRPC client. And then the second main component is the pipeline runner. So it takes a set of files which define a pipeline, runs analysis across all of them and executes them. So let's start by talking about the APIs and the core concepts that they model. The goal of the declarative pipelines APIs are to enable users to define pipelines.
So earlier we said the two main components of a pipeline are the data sets and then the flows uh the queries that flow data into those data sets. Um and so in this case we basically just need to describe those to describe our pipeline. Um we want to describe the tables that the pipeline should update and the queries that determine what data should live inside those tables. So here's a minimal example which defines a single table and a single query.

`21:06` Python and SQL APIs

The first thing to note at the top is that we're importing from this new pispark. pipelines module. It includes all of the Python APIs that we're going to use here. Um, and because pipelines is a lot of characters to type, we alias it to SDP, which stands for Spark declarative pipelines. Um, the API that we use for defining our table and its query uses Python decorator syntax. So the name of the decorated function is the name of the table, raw orders in this case, and the body of the function returns a data frame, which is the query that's used to populate the table.
So in this case, the query is a streaming query. And every time the pipeline gets executed, the results of this query are going to be appended to the table. We call this combined definition of a streaming query and a table a streaming table. And a streaming table uh is just a table whose contents are produced by a streaming query. The pipeline is going to automatically assign a checkpoint location to manage state for the streaming query um so that across multiple runs um it can pick up where it left off.
If we want to attach metadata to the table, we can lightly amend this code to pass in a couple extra arguments to our decorator. So here we've attached a comment that documents our table as well as a table property, which in this case controls how long delta logs are retained for. Declarative pipelines also supports batch processing through materialized views. So here we're defining the customers table as a materialized view.

`23:04` Advanced pipeline definition

Uh and its contents are going to be the result of this data frame. The main difference between this and the streaming table that I showed in the last slide is how the results of the query are used to update the contents of the table. For a streaming query, the results are appended to the table. But for a materialized view, the contents of the table are fully overwritten by the results of the query. More generally, what exactly is a materialized view?
So, it's a table whose contents are defined to be the result of a particular query. This is in contrast to normal tables whose contents can be the result of a long string of inserts, deletes, and updates. The contents of uh materialized views are often much easier to reason about um because you can just think of them as the result of this particular query like a regular view. Um materialized views also open up the possibility of incremental refreshes.
That means that if there are small changes to the table that a materialized view is derived from, then the materialized view can sometimes be updated to reflect those changes without recmputing all of its contents from scratch. Note that this does not mean that the results are always guaranteed to be up to date. Uh when dealing with big data and arbitrarily complex queries, being always up to date is just infeasibly expensive in many cases.
Um, so the expectation is that um, people who understand more about their queries and their um, cost tolerance can decide when to do refreshes um, to navigate some of those trade-offs. Okay, so back to our APIs. We already showed the definitions of two of the data sets in our pipeline. After we add in our third one right here for the fact orders table, we're done. That's it. We don't need to write any more Python code at all.
We don't need to define a special DAG object. U we don't need to write any multi-threading. We don't even need to put these statements in any particular order. So they could be in separate files um or just in whatever order. Uh the framework can just analyze the data frames that our code returns uh and figure out the dependencies, wire that all up into a pipeline um and then make it available for execution. I focused on Python APIs up to this point, but SQL is just as much of a first-class citizen of declarative pipelines as Python.
You can basically do anything you can do in Python and SQL. So, here are the equivalent materialized view definitions in Python and SQL. In SQL, the code looks just like a regular create statement. Uh, here's what it looks like if we want to create a streaming table. Um, like we did earlier in Python. Uh, again, just a create statement. In this case, we say create streaming table. Uh, and then we want our select statement to be a streaming query.
So, we use this special stream keyword when reading from our table. Um, stream raw orders is the equivalent of using the readstream API inside Python. Um, and so here's the full pipeline in SQL. Note that we don't have to choose between Python and SQL when we're defining a pipeline. Our pipeline definition can contain multiple source files, some of which are SQL and some of which are Python. So, we can fully mix and match.
Okay, so we've written the code that defines our pipeline and now we want to actually run it. The other main component of declarative pipelines is the pipeline runner. It takes pipeline code, analyzes it, and then executes it. The first thing that we'll typically do here is to create a directory to put our definitions and configuration in. So declarative pipelines exposes a command line interface called spark pipelines uh that we can use to set this up.

`26:22` Pipeline runner functionality

Uh the command line interface supports this init command and we run it. Um, it gives us this little directory structure with a pipeline configuration file and some example pipeline files. We can copy in our actual pipeline code. Often it's nice to split things up. So here we've got a file per data set, one for the raw orders, um, and so on and so forth. Um, and here's what our default configuration file looks like. It just says to look for Python and SQL files inside this transformations directory.
Uh if we want to, we could also specify things like spark configurations um or a default database schema where our table should live. After we've done all that, we can just use this run command to kick off an execution. So here's an this is a little small, apologies. Um but so here's an example of the logs that we'll see when we do a run. The first thing that happens is just that it reads our configuration file to find out to um where to look for definitions.
After that, it's going to create a data flow graph and load all our data set and query definitions into it. Um, then finally, it actually starts executing the run and as different queries are executed, progress logs are going to stream in. This could take a long time. We can kind of monitor it uh through the command line. The framework automatically handles a bunch of stuff for us. So, first of all, it respects data dependencies between the different queries in the pipeline.
It's never going to try to execute a query before upstream tables within the pipeline exist and have the data that they are expected to. Second, it automatically parallelizes execution of queries as long as that parallelization doesn't violate any data dependencies. Uh we can control the number of concurrent queries with the configuration option. Um third, it automatically retries failed queries during execution. Um and how often to retry is also is also configurable.
And finally, it does a holistic uh pre-validation. So before executing anything, it analyzes all the queries to make sure that they're able to run successfully. This catches the kind of error we talked about before where a downstream query tries to read columns that don't exist in an upstream table as well as a bunch of others. Um this pre-validation step provides immediate feedback on whether the pipeline can run successfully instead of you finding out after a bunch of stuff has already been executed.
Declarative pipelines is going to support two different pipeline execution modes. Uh and these roughly correspond to these two different ways that we talked about executing pipelines uh earlier in the top uh discrete and continuous. So the first mode which we call triggered processes a bounded set of data executes each query in the graph once uh and then stops. This is similar to the way that airflow or dbt works. It moves through the graph in a topological order, meaning that it's never going to start a query until the queries that update upstream data sets have completed.

`29:05` Execution modes explained

The second one is continuous mode. So it just keeps processing new data as it comes in until a user explicitly shuts it down. For streaming queries, it just starts the stream and runs forever using standard structured streaming execution. Um uh and this allows much lower latencies which can be very important. Um, and for materialized views, it executes them at a configurable cadence. Note that the words continuous and streaming have some relation but mean two different things here.
So streaming means that we're executing a query that processes one record at a time. Uh, we're consuming data from some source that has unbounded data. And so from a logical perspective, the transformations that we run on that data need to think of that data as a stream. So for example with a non-streaming query we can do aggregations over the full data set that we're processing. For streaming queries this would make no sense because the full data set is assumed to be infinite.
So any aggregations need to occur over some sort of window. On the other hand uh continuous is an execution concept that essentially means start and don't stop until you're told to. Um a continuous execution is going to keep processing new data until explicitly stopped. Um so you know a distinction here is that a spark structured streaming query can run in continuous mode but it's not required to. So it's actually common to run a streaming uh query only on the set of records that are in a stream that are available up to a particular point and then stop.

`31:05` Ecosystem comparison

Declarative pipelines is not the first technology to help run a data pipeline. So the second to last thing I want to talk about is where declarative pipelines fit in the ecosystem of technologies that do related things. First off, what's the difference between Spark declarative pipelines and lakeflow declarative pipelines? So lakeflow declarative pipelines um is what was formerly known as DT. The short answer is that STP, Spark declarative pipelines is a subset of LDP. um any code that's written for Spark declarative pipelines as expected to work inside LakeFlow.
Uh this won't be the case in the other direction 100% of the time. We're currently focused on getting the core APIs out there and it's going to take us um some time to get everything uh as well as uh there are some features that we would need to support uh other Lakeflow features. LakeFlow also supports a pipeline management layer where data bricks handles the infra for deploying and monitoring pipelines. Uh this also includes the integrated pipeline editor that we recently released.
What about STP and DBT? So if you're not familiar with DBT, it's a framework for transforming data within a warehouse by combining SQL transformations into a graph. STP and DBT have some overlap. So they both enable users to execute graphs of data transformations that populate multiple data sets. However, they have different architectures, different strengths and different audiences. As part of Spark, STP is able to have deep integration with Spark's analysis system uh as well as Spark's multi- language support and Spark support for structured streaming.
DBT can be used with other engines but doesn't have as deep integration with them. DBT is primarily focused on SQL but STP supports Python as a first-class citizen um and is eventually going to support Scola and Java as well. DBT assumes batch execution but STP has native support for streaming and continuous processing. Um this makes STP especially useful for dealing with huge data sets where reprocessing data is prohibitively expensive.
Um as well as for running in this continuous mode um that's well suited for lower latency use cases. Um, last of all, DBT primarily expects inputs to come from the warehouse. Um, and STP supports those kinds of inputs, but it's support for Spark structured streaming data sources mean that it's well suited to handle ingestion as well. Um, all this means that SCP is useful for a wide set of data engineering workloads. Um, and DBT is more focused and shines for uh, analytics.
Last, what's the difference between STP and general purpose orchestrators like Airflow, Dagster, and Prefect? The short answer is that even though there's some overlap in functionality, um, STP is actually complimentary with general purpose orchestrators. So the overlap is that like these orchestrators, STP executes a graph of computations. In an orchestrator like Airflow, a computation inside this graph can do anything.

`33:47` Status and future roadmap

But in STP, it's always a spark query that updates a data set. In exchange for this constraint, STB can do many things that a general purpose orchestrator cannot. Um, so it can analyze the full graph before execution, automatically infer dependencies, catch schema mismatches, execute continuously, and more. Executing a Spark to clear a pipeline can be a task in an Airflow DAG. And then Airflow can be used to run that task on a schedule as well as execute other kinds of tasks before and after it.
This is also the way that um Airflow and DBT are commonly used together. One way to think about this relationship uh is that STP is kind of like a GPU to Airflow's CPU. You can't run everything on a GPU that you can run on a CPU, but when you have a workload that does fit on a GPU, it's often worth it to use one. Um and you can delegate those workloads to the GPU and then have a CPU handle the rest and um and invoke it.
Okay, so all this sounds pretty cool. When can you actually use it? Um, as of yesterday, we've contributed the basic functionality to Spark and we're finalizing some finishing touches for tomorrow. Um, uh, we have merged includes the Python APIs, uh, the SQL syntax, the command line interface, um, as well as triggered graph execution. If you're brave enough to check out the Spark repo and build on your own, you'll be able to try it out.
Although, I would recommend at least waiting a few weeks. uh while we polish it and fill in some gaps. Um this should all definitely make it into Spark's next release though 4. 1. The next things that we want to work on uh we want to make it possible to run uh sub selection. So this is kind of like a manual retry. Um we want to handle non-mporter views. Um continuous graph execution, external syncs, which is the idea that maybe your pipeline isn't writing to a table.
Maybe it's writing to some arbitrary structured streaming sync. um right into another um Kafka topic or something. Um uh and last of all, we want uh Sparks catalog APIs to understand the concept of a materialized view so it can represent them to systems that have notions of materialized views like the hive meta store. Um we're not making promises yet about exactly when these will land, but we want to move fast. We believe STP is going to become a durable standard that elevates data engineering to new levels of productivity and reliability.
And we are super excited to work on this with the Spark community. That is all I have. Thanks so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.