From Repetition to Reuse: The Evolution of Apache Spark™ Declarative Pipelines

Summary

Apache Spark Declarative Pipelines (SDP), introduced in Spark 4.1, lets developers declare datasets and transformations while Spark constructs and manages the full execution plan, eliminating manual orchestration boilerplate for dependency resolution, checkpointing, and failure recovery.
SDP supports both Python and SQL for batch and streaming workloads, and introduces streaming tables and materialized views as the two primary dataset types with distinct semantics for incremental and full-refresh processing.
This video covers SDP core concepts including the APPLY CHANGES Auto CDC command, dry-run execution for error detection, and how AI coding agents using Spark Connect can autonomously iterate on declarative pipeline projects.

From Repetition to Reuse: The Evolution of Apache Spark™ Declarative Pipelines

Learn how to build batch and streaming pipelines faster while improving correctness and reducing operational complexity with Apache Spark™ Declarative Pipelines. Production Spark pipelines often require extensive orchestration code for dependency management, checkpointing, retries, and execution ordering—surrounding a relatively small amount of transformation logic. As pipelines scale, this scaffolding becomes increasingly difficult to maintain and evolve. Introduced in Spark 4.1, Spark Declarative Pipelines (SDP) shifts this model by allowing developers to declare datasets and transformations while Spark constructs and manages the execution plan. By separating what a pipeline does from how it runs, SDP reduces boilerplate and accelerates time to production. We’ll examine the architectural foundations of declarative development in Spark and how SDP handles dependency resolution, parallelization, checkpoint coordination, and failure recovery. We’ll also cover incremental processing and emerging testing patterns for declarative pipelines.
Talk By: Andreas Neumann, Senior Staff Software Engineer, Databricks ; Lisa Cao, Staff Developer Relations, Databricks ;

Chapters

FAQs

What are Apache Spark Declarative Pipelines and when were they introduced?

Apache Spark Declarative Pipelines (SDP) were introduced in Spark 4.1 as a way to define data pipelines by declaring datasets and transformations rather than writing imperative orchestration code. Spark constructs and manages the execution plan from these declarations, handling dependency resolution, checkpointing, retries, and ordering automatically.

What is the difference between streaming tables and materialized views in SDP?

Streaming tables are designed for continuously ingesting new data in append-based incremental workloads, while materialized views are recomputed results that reflect the current state of the data including any upstream changes. SDP manages both within the same pipeline graph with consistent execution semantics.

How does Spark Declarative Pipelines handle Change Data Capture?

SDP includes an APPLY CHANGES command, also called Auto CDC, which declaratively handles CDC operations including upserts and SCD Type 2 patterns without requiring custom merge logic. Developers declare their intent and SDP executes the correct incremental merge with built-in consistency guarantees.

How can AI coding agents work with Spark Declarative Pipelines?

AI coding agents can use Spark Connect's architecture, which decouples the agent from the JVM, to iterate on SDP pipeline definitions autonomously. This video demonstrates a live workflow where an agent performs dry runs, detects errors, and selectively refreshes pipeline components without manual intervention.

Full transcript

[00:09] Hello everybody. Uh welcome to my talk. Um I'm going to be talking together with my co-worker Lisa about um Apache Spark declarative pipeline. Uh from repetition to reuse.
[00:23] I don't know. It doesn't >> How's everybody's energy level right now? Are we Are we feeling okay? Yeah. We're ready to go? Yeah, pump up. Ready to talk some declarative pipelines?
[00:36] >>
[00:49] I'm My name is Andreas. I've been in the industry for quite a while. I've built data pipelines in many places, and now I'm doing it for Databricks. Um what I'm going to talk about is the new declarative
[01:02] pipelines in Apache Spark. They were only added in uh Spark 4.1, which is a pretty recent release of uh of Spark. And what you can do there is you can
[01:14] define, validate, and execute data pipelines. It's built on top of existing Spark components such as Spark SQL, PySpark, streaming, and Spark Connect. It supports both Python and SQL
[01:28] and batch and streaming. And we will see this in detail on the following slides. So, what are we going to do? Uh the talk has mainly three parts. One is about Well, why are we Why do we care about
[01:41] declarative at all? Right? Um the second one is we're going into what are the components of a pipeline in in SDP. And then in the end, we're going to look at an actual SDP project, what that
[01:54] looks like, and if we have time, we'll do a demo. So, the shift from imperative to declarative, um declarative is not something new, right? Most of us um Who has used a declarative
[02:06] tool here before? Yeah, almost everybody, right? Because we know systems like Kubernetes, like Terraform, or Airflow, they are all systems that um do some kind of
[02:18] orchestration or deployment in a declarative way. Rather than writing code, you write a configuration or a specification, and the system derives from that what it should do. Now, Spark
[02:31] This is weird, my So, what about Spark? Uh Spark is a declarative framework. If you write a query in Spark, let's say a a data frame expression or a SQL query, that is
[02:43] declarative, right? Uh you just tell it what you want, and Spark figures out a physical execution plan, and then runs it for you, right? However, what hasn't been declarative in
[02:55] Spark is the orchestration of multiple such queries, right? And this is where STP comes in. And so, it it extends the declarative model to to the orchestration of an entire data pipeline.
[03:09] Right? So, in uh in STP, what you do is you only define data sets. And uh that's why we also say it's a data flow framework. It's not a control flow framework where you say, "Do this
[03:21] first, and do that, then do that." But, we simply give our data set definitions, the system figures out based on the dependencies between those data sets in what order things need to run. So, what STP automatically handles are
[03:34] things like checkpointing. If you write your pipeline manually, you have to manage the state yourself. You have to find a place to store your checkpoints, and you have to remember where that is.
[03:47] Uh STP does that for you automatically. Um as I said, it it figures out dependencies and figures out execution order. Um, it also does error handling for you. When it finds that an error occurred
[03:59] that might be retryable, it will retry for you automatically. Um, when it detects that two queries can run in parallel, it will do that for you. And um, also after failures, if you
[04:11] retry, it makes sure that everything is item potent, so you don't get duplicate data. All right. So, with that, your code is really clean. Your pipeline code simply focuses on the business logic of your transformations
[04:23] and none of all the orchestration and operational aspects of it. All right. Okay. So, let's take an example. In this um, very simple pipeline, we
[04:36] have two queries. One reads some customer data, another one reads some orders. And then we want to join them and somehow enrich these orders with the customer information. What's
[04:49] instantly obvious here is that these first two queries can run in parallel because they don't depend on each other. All right. If you do this manually, let's say you do it in in Python. Or is this Python? Or is this
[05:01] Scala? No, this is Python. Um, you start adding like threading code to your to your pipeline code, right? And that um, distracts away from the business logic. It's also error-prone.
[05:13] Especially if the pipeline becomes more complex than these three queries, right? Whereas um, in SDP, you get rid of all this extra code and all you do is just declare your data sets. You don't even have to declare them in
[05:25] the order in which they need to be executed. All right. So, here you see that this last data set here is actually defined first. But, that's all fine. SDP can reorder them in the in the correct order.
[05:41] All right. And so, one thing that this gives us is um, that um, SDP will take all your code and run it and because your code doesn't actually execute anything, it only declares what you want to happen.
[05:53] Um STP can now analyze the entire code. It can build a data flow graph and it can find errors. It'll find syntax errors, it'll find schema errors. Um and um it will do all that before any
[06:09] any uh data is written or any uh tables are created. Right? Um Now, um there's actually a mode that you can run STP in. That's called the dry run mode and that will only do this
[06:22] analysis. Right? And that allows you to build your code iteratively and and pretty quickly without really executing the pipeline. You can still see whether all your queries are correct and your schemas match.
[06:37] Now, what else does STP handle? We already talked about state management. It manages all the checkpoints for you. Um it also detects um if you run your pipeline multiple times, right? You won't run it only a single time. You'll probably run it every day or every hour.
[06:51] And now you make a change to your code or your data changes slightly and that leads to a change in the schema. Now, you need to evolve the schema of the of the tables that are generated by this pipeline and STP can do that for
[07:04] you automatically. You don't need to write any logic yourself. Um Another thing that it gives you is uh partial execution, especially while you're developing a pipeline, you
[07:16] probably only want to run the pipeline on the data set that you're currently working on. Right? And so instead of having to put if statements or conditional logic into your pipeline code, you can just use runtime arguments with uh with STP
[07:31] zero lines of code to just do a partial execution. So, um just to summarize, um declarative frameworks like SDP provide higher-level abstractions
[07:45] and they create a focus on the business logic of the transformations in your pipeline. Right? And there's no more distracting code that is about orchestration and and such things.
[07:58] Um, SDP allows you to have streaming and batch in a single pipeline. It provides operational features out of the box such as state management, schema evolution, orchestration, and retry
[08:10] logic, concurrency, etc. And it enables faster iteration and shortens the development cycle. Now, that's not only true for the human user, right? We have a new user in this world since, I don't know, a year or two
[08:24] ago, and that's the agent, right? And I think we've learned by now the better your abstractions, the better the agent, right? The fewer iterations, and um, so it's actually very useful for an agent, too.
[08:42] All right. So, let's talk about what an SDP pipeline actually looks like. So, as I said, in the in the pipeline, you only define the data sets that you want to create, and for each data set, you define it using a query that gives
[08:56] the data that should be in that data set. Now, you have a a few different things here. The two main ones are streaming tables and materialized views. You can also create temporary views and
[09:09] regular views. You can have a sync. A sync would be something that writes to somewhere outside of Spark or somewhere outside of your catalog. And you can have standalone flows. We're going to
[09:21] see, um, all of these in detail. Now. So, the first thing I want to introduce is, um, streaming tables and materialized views. Um, who here is familiar with a
[09:33] medallion architecture? Yeah, bronze or silver gold. Um, pretty standard nowadays in the industry, right? What you do is is pretty much you do somehow a raw ingestion of data into
[09:46] a bronze table. You're going to filter out the bad data. You're going to cleanse some data. You're going to maybe normalize and then enrich it a little into a silver table. And maybe you do some more aggregations and enrichments into a gold table.
[10:00] Right? So, now how do streaming tables and materialized views fit in here? Well, the first two stages are most likely going to be streaming tables because what is a streaming table?
[10:12] A streaming table is a table that has state. And the state is everything that you have processed so far. Next time you run the pipeline, it will only process the new data in its input.
[10:27] Right? And so, with that it is kind of incremental. But it is also stateful in the sense that it reflects the business logic of previous runs. Let's say you change your code.
[10:39] And somehow records are now processed slightly differently. Existing records in this in this streaming table will remain the same. And only the new data will get the new
[10:51] business logic. Right? So, that is important to know. But it's it's a very useful thing. Uh very often the first stage in the pipeline is you're reading from some Kafka or from some staging area where
[11:04] data is only retained for a short period of time. But you want to keep all this data for much longer time, right? And so, you're going to ingest it into a streaming table. And you know, this table will
[11:16] keep whatever I had there because it's stateful, even if it disappears from the input. Now, a materialized view is something that is conceptually recomputed every
[11:28] time you run the pipeline. It reflects exactly the query that defines it. All right? So, it's not stateful. Now, there's optimizations where you can incrementalize some of the computations,
[11:40] but conceptually, semantically, it's always a full recompute. Um Any questions here about streaming tables and materialized views?
[11:52] The difference of the concepts? Yeah. >> It's interesting. I've watched different presentations and read makes sense. It's useful. I like your explanation of streaming
[12:04] tables on bronze and silver. Um and then materialized views on gold. Cuz I've seen I've seen other presentations where they've done materialized views on silver. Never on bronze. And sometimes on silver and not just
[12:17] I don't know Explain that to me and why you I your explanation makes a lot of sense to me, but is there a reason why someone would do materialized views on silver? >> You could. >> It's not a good architecture, though.
[12:29] >> Yeah, the question was why not materialized views for silver and only for gold? I think it's a matter of personal taste. It really depends on the use case. In some cases, it makes sense. Uh in others. Um uh one example I can
[12:43] give you is if you if you call an AI function in order to uh or or a model that's expensive to run, right? In order to enrich your data, um then you want to use a streaming table because you don't want to recompute that every time the
[12:56] pipeline runs. Okay? All right. So, here's a simple end-to-end example. Um and this looks actually very similar
[13:09] to the demo I want to give. Um So, um we have uh a bronze layer here, which is uh a table that simply reads from some CSV files. Um then we have a streaming table that
[13:22] enriches them a little bit. It basically it takes the timestamp and extracts the date from it. Of course this is very simple, but it has to fit in the slide, right?
[13:34] But then in the gold layer we do an aggregation, right? And that is now a materialized view and here we What do we do here? Yeah, we aggregate by um We aggregate by zip code and then we sum
[13:49] up all the sales um for the current year. Now the interesting thing you see here is we can mix and match Python and SQL, right? This first streaming table is defined in
[14:01] Python, the second one is defined in SQL. You also see that we can have streaming tables and materialized views in the same pipeline. So we can mix and match arbitrarily.
[14:18] So here's one more time a a comparison of materialized views and streaming tables. Materialized views are mostly used for batch processing. They semantically represent a full
[14:31] recompute and always represent the results of the defining query. They're simple and predictable. What they give you is basically query acceleration, right? If
[14:44] you have a complex query, you can take a subquery of that or even the entire query, make it a materialized view and then your query speed will be extremely fast. And typically you use it in the gold layer for things like aggregation and
[14:56] enrichment. Streaming tables, however, are stateful, they're strictly incremental, and the checkpoints of these streaming tables are are checkpoints of the of the streaming queries that writes
[15:09] into this table. They are managed for you. It preserves its history and it's stateful. Changing anything in the query or in the schema only affects new rows and not old
[15:21] rows. This you get streaming semantics though. You can do things like windowed aggregations and joins, which you don't have in batch. Um and they are fast and efficient and you can actually run real-time
[15:33] streaming. Has anybody seen the talk about real-time streaming? Uh it's pretty exciting, right? That's super fast. Um and so what do we do this for? Well, we do this A to retain data for a long
[15:45] time. Uh we use it for really large data sets um and we do simple things like filtering, cleansing, normalization in the bronze and silver layer. >> So, these objects will not be available
[15:57] to Iceberg clients outside
[16:09] the open-source version of STP, which is uh part of Apache Spark. Whatever catalog or storage format you use, it'll work. >> Oh, both streaming and materialization will be
[16:21] >> Uh I do think so. I haven't tried it. Um if if it doesn't work, it would be a bug. Okay. So, um one other thing we have is
[16:35] temporary views. And a temporary view is very similar to a temporary view in other other Spark sessions. It gives you something that is only visible within the pipeline. Right? But it allows you to define, say,
[16:48] common subexpressions that multiple of your queries share. So, you don't have to repeat them over and over again. These temporary views are not persisted. They're not visible outside of the pipeline. And what happens is that the
[17:00] query of that temporary view is simply inlined into the query of anything that consumes from it, just like a normal view. Um and here you see both the Python and
[17:12] the SQL syntax. Um Then we have Okay. So, the simple way to define a pipeline is you define for each data set exactly one
[17:24] query. But, there are use cases where you want to have more than one query per data set. And here's an example, for example, um if I want to consume events both
[17:36] from Kinesis and from Kafka and I want to put them both in the same streaming table. Then I can define this streaming table by itself and then I can define two standalone flows that both write to
[17:48] that table. All right, that's the main use case here. Um another use case is sometimes I want to have excuse me, a backfill flow. Let's say I changed some logic
[18:02] and I want to reprocess all data before a certain date, but I don't want to stop processing new data, right? So, I'm just going to run two flows, one for the historic data, one for the new data.
[18:21] And then we have sinks. So, sinks are um are flows in a way um that write to something that's not inside your catalog. All right, so um essentially it's a for each batch and you can do whatever you want in these things.
[18:37] Now, let's talk about schema evolution. Um I mentioned it briefly earlier. It happens automatically when the output schema of some query changes. All right, so this can happen if the schema in the source changes. It can
[18:50] happen if you change the query. And STP will detect that. It will see what the current schema of that output table is. It will see what the new schema is. And it will
[19:02] attempt to evolve the schema of your data set. For a materialized view, that's easy because we say materialized views are always fully recomputed, right? So, we can also change the schema.
[19:14] For streaming tables, it's a little bit difficult because it could be that the new schema is not backward compatible with the old schema. I I I can't convert a string to a float
[19:26] or things like that. Right? So, when it's not compatible, then the pipeline will fail. And it will actually fail before it processes any data because, as I said, STP analyzes the entire data flow graph
[19:40] before it touches um any any data. Um so, what can we do if that fails? Well, in that case, the only thing we can do is a full refresh. A full refresh
[19:52] it essentially says, "Truncate the table and reprocess all data from scratch." Um now, that data can get the new schema, but if your if your source for that data
[20:04] has a short retention period, then you will not get all data back. That's just important to know. Um and for things, there is there is no explicit schema evolution. For each batch, it's for it's it's basically an
[20:16] escape hatch for you to do whatever you want to do that the framework doesn't support out of the box. Okay, last thing um that uh
[20:28] maybe not the last thing I'm going to talk about, but very important to mention is change data feeds. Who here has uh ever consumed a change feed? How How hard is that?
[20:42] Yeah. Do you have to deal with out-of-order data, partial updates, things like that? Yeah, so um it's it's not actually easy. Um in Spark 4.2, there's a new feature um
[20:56] which is which is called select changes from. And that allows you from any table that is registered in your catalog, you can actually get a change feed in a standardized way.
[21:09] Now, this produces an output that it essentially has the content of a row and then the type of the operation that happened, right? So, it's might be an insert, might be an update, it might be
[21:22] a delete. Um and it typically also has some kind of sequencing. It may be a timestamp, it may be a commit ID, but it's something that we can use to compare the order in
[21:34] which things happen. Right? And the the typical use case is that we want to create a replica of this data. Maybe this data is um coming from an operational database on on prem and we
[21:46] want to have a copy of that for our analytics, right? So, um how do we do this? Um It It looks easy at first, but then you realize data can be out of order and the
[21:59] out of order data may actually be across multiple microbatches because as we know Spark computes in batches and um the the boundaries of these batches are
[22:12] very hard to predict. Um now, um here we have an example where we have um I think we have an update um that comes before a delete.
[22:25] All right? So, the delete comes later and we we need to make sure that we account for that. Um so, other things that make it difficult is um incomplete change uh events. Some
[22:37] databases will spit out only the columns that changed and not the entire row when something changed. Um then you have cases when a table is truncated, right? Um now you have to handle that kind of
[22:51] kind of event. You have to make sure that you reset your replica at that point in time and then play from there on. Um some people um prefer an output format that is not a replica
[23:04] of the original table, but it is called slow changing dimensions type two, which actually tracks history for every point in time that any row in the table has had.
[23:16] Right? And another thing that's difficult is you have to make the whole thing item potent. If you have a failure and you retry, item potency is super important. Otherwise, you might get duplicate data. So, in the imperative paradigm, if you write
[23:29] this by hand, I did it once. Uh it's easily hundreds of lines of of Spark code. Um In Spark 4.2, we introduced a new feature called auto CDC.
[23:43] And this is also available in pipelines. Um And what this does is it applies the changes to a target table based on a configuration that you give. So, all you do is you say, "Well, I want to create
[23:56] this table. I want to read from this change feed. These are my keys, right? That's your your primary key. Um this is how I identify deletes
[24:08] because that looks different in every change feed. And the sequence by will tell me how to order things. And here in this example, I'm saying SCD type one, which means we always only
[24:21] have the latest state of the data. Um SCD type one is the only SCD type that is implemented in 4.2. Um we're currently um adding support for SCD type two.
[24:34] That's going to be in Spark 4.3. Um we had a Well, this session was on Tuesday, so it's too late for you to attend that if you missed it. >>
[24:48] to view the recording. Um All right, so that was auto CDC. Now, I just want to give a quick This I feel like this slide is in the wrong place, but when you run
[25:08] Oh. Doesn't matter. I'll just go in this order. So, in order to run a pipeline, you just say spark pipelines run. And but you also have a way to define a refresh selection, which means you define the subset of tables that you
[25:21] want to recompute. This helps you especially while developing. You also have a way to specify a full refresh, as I said earlier. Sometimes
[25:33] you need to full refresh, and you have the option to full refresh all your tables or only only a subset of them. Quick note here, there is a spark
[25:46] pipelines operator in airflow, so you can integrate STP into all of your other workflows. And now we're going to very quickly look at the pipeline structure, because I think I have to speed up a little.
[25:59] So, basically this is the structure of a pipeline. Everything is in one folder. You have a pipeline.yaml or spark-pipeline.yaml, which defines what belongs to the
[26:11] pipeline, and then you have your transformations in a subdirectory. This is sort of the standard way that we recommend to structure it. And you can configure in this YAML file, you
[26:25] can configure a catalog and a database. And you can also configure extra configurations. What you see here, you can mix and match Python and SQL. Um
[26:38] Okay. So, when you run it, yeah, this gets a little repetitive, so I'll just go through here. The the workflow that you get now is So, so remember, we have two features that are super are One is called the dry run, and
[26:52] the other one is called the refresh selection, right? So, the way you develop a pipeline now is you start writing the code for the first data set. You run a dry run to validate whether it's good. If you get an error, you go
[27:04] back um and um and and make some changes until the dry run succeeds. If the dry run succeeds, you run the pipeline for real. And if it's all good,
[27:17] then you go back and you start over with your second data set. And so, now you can iteratively um develop and always only run like the the the minimal subset of the pipeline that you're currently working on.
[27:29] Um yeah, one thing I want to say about this, of course, this is great for people, but this is also great for agents. Identic workflows, they just map right onto this. And uh Lisa is going to talk a little bit about that.
[27:41] >> Uh my turn to speak. So, who here uses agents to code? Okay, that's a lot of people. Uh who here loves the idea of writing Spark pipelines by hand for the rest of their
[27:53] lives? Not a Not a single person? Okay, well, I thought I was in a room of open source peers. >>
[28:06] there's a lot of API surface area. There is just a ton of things that have been involved with creating this project, a ton of modules, things like that. One of the worst things about that, though, is that it's a lot of surface area for an
[28:18] agent to hallucinate, uh which is not great. And so, when we think about developing a pipeline as a If you're a senior open source engineer, if you've been around Spark for a while, you know exactly what you want to configure in
[28:30] your Spark deployment and pipelines, because you know your stack incredibly well. You've been working with this cluster for a while. The version awareness of Spark lives in your head already, cuz it's your sad reality every
[28:42] single day. Sorry about that. So, when we think about getting new developers into the Spark ecosystem or even agents into the Spark ecosystem, one of the things that we really want to consider is is the
[28:55] traditional developer loop that we typically work with in our traditional workflows going to apply in the same way or is it going to result in a lot of wasted compute and time? And it's probably going to be the latter
[29:07] if you haven't already experienced that already. So one of the things that I think is really interesting about Spark declarative pipelines and Andreas has done a great job of kind of you know explaining this already but we're
[29:19] reducing a lot of the surface area that we actually need the agent to write when it comes to Spark code because it only really ever has to worry about the transformations themselves which is relatively straightforward. This is
[29:32] double if you're having it right like a SQL based pipeline and creating materialized views and things like that because agents really inherently understand SQL quite right. It doesn't really understand all of the Spark configurations that you probably
[29:45] already have baked in and all of the the custom deployment logic that you might have. So if we look at the way that an agent maybe would write using declarative pipelines, one you can have it write and only worry
[29:57] about the actual functions and the transformations including the tables and you can scope that really tightly based on the project structure that we kind of showed already. Like you have this directory with all of your Python and SQL files and you can have agents really
[30:10] only worry about that while you worry about the larger project, maybe the YAML files and things that you want to be a little bit more hands-on with etc, right? The second is that agents can actually validate the pipeline
[30:22] relatively quickly using the dry run feature without wasting all of your compute because you're going to get clearer error messages versus just like this huge amount of Spark logs that you have to then go through. So Andreas will
[30:36] probably show you some of the outputs later but it's a little bit more easier to digest and you're going to be able to use dry run as well as selective refresh to iterate quickly within a sandbox environment. So, this is also one of the
[30:48] things that we talk about when we're iterating here versus just manually trying to do a run and not finding out until maybe hours in, sometimes even days in that something failed because, you know, there was something
[31:00] hallucinated, and then you can deploy it a little bit easier. This is primarily because Spark declarative pipelines actually relies on the Spark Connect architecture. So, Spark Connect, which we'll talk about in a little bit, but just before I
[31:13] got on there, sorry, my slides are also out of order apparently. >>
[31:25] somebody who's been in this space for not a terribly long time, but long enough to know, most of the corpus that these LLMs are trained on Spark 2 and Spark 3, not Spark 4, not best practices. It's actually mostly the tutorial materials,
[31:39] and an agent is going to assume best practices because it thinks it inherently knows an answer versus actually looking at maybe your architecture. It doesn't have a good understanding of your like actual setup or your cluster, all of these things.
[31:52] And so, oftentimes when you have a large imperative pipeline, it's going to infer what you want in terms of retry logic, in terms of parallelization, threading, all of these things, the checkpoints
[32:04] because it knows that it should, but it doesn't know exactly what you need or what you want. And honestly, the way that we've started designing Spark in Spark 4 is just to assume Spark probably knows a little bit better
[32:17] than your agent at this point. So, we're going to let Spark handle a lot of this, and we're going to let the agent handle the transformations because that's really easy, it's well scoped, and it's easier to test, right? So, these are some examples that we see oftentimes
[32:30] with agents hallucinating within a imperative place or imperative code base. So, you can see some of the examples there as well.
[32:43] So, if we talk about Spark Connect architecture and kind of the direction that we see Spark 4 and going and kind of how this relates to SDP, this is particularly if you're in the open source world. So, Spark Connect was introduced in Spark 4. It needs to be
[32:56] manually enabled, but one of the things you can do is actually decouple the driver and the JVM that you actually have to work with and deploy with a local client. So, you can actually have a very thin local client, like very
[33:08] easy, just like a pip install PySpark client there, and then you can manage the actual cluster yourself. And then one of the things that you can do really easily is just point it locally to this
[33:20] remote server instead. This is really nice because again, you don't really want agents running around your drivers and your actual JVM setup, and it's just slow and it's hard to iterate. And if you're trying to sandbox
[33:33] your agents' environment, again, it it's just a very heavy thing to have to deal with, and you may have different setups that you want to point it to. So, one of the things that we can see with the connect architecture where you're
[33:45] decoupling these sort of things is having agents or notebooks, CI runners, or embedded apps right there on Spark applications, iterate quickly, test without any sort of, you know, expensive compute involved, very lightweight, and
[33:57] then you just use Spark Connect, you have a Spark Connect cluster setup, and then you're just sending it there with like a one-line change, removing from localhost to connect really easily. And if you want to mess around with it, you
[34:10] can. It doesn't cost you anything, right? And that's kind of the thing that we're really interested in seeing. So, this is kind of a really interesting workflow when we tie in the idea with actual Spark declarative pipelines here,
[34:22] is how does that workflow change? And Andreas is going to kind of show some of this process later right after this, but let's say that we have a agent that's going to write, you know, some transformations. We can actually scope
[34:35] that agent's surface area to just a single table. So, in this case, silver layer table here with our orders here. And it doesn't really have to worry about anything else because it doesn't
[34:47] have to. Let's say the agent writes it. Let's say that they, you know, everything else is relatively good. The syntax is relatively good. It runs except for maybe one small failure, which in this case is a dependency error. If you see
[34:59] here, there is a column that doesn't exist that it's trying to play in, bring in. But, Spark Pipelines will actually raise that error relatively early in the process during dry run. That way the
[35:11] agent can take that output, iterate it, knows exactly where it's gone wrong, fix it, and then afterwards it can run it again with dry run or do a selective refresh on that single table that it's
[35:23] been working on. And then, boom. Like nice and easy, smooth, and you don't have to worry about any of the deployment around it. So, hopefully that kind of makes sense in terms of how we assume agents are going to work with us.
[35:36] The one caveat I'm going to give here cuz I'm assuming that hopefully, in the best case scenario, you're going to go home and try this, is again, agents need to kind of have a bit of hand-holding right now when it comes to writing SDP code right off the bat. We do have
[35:49] skills available that you can actually provide to your agent to write SDP code right off the bat instead of trying to get it to like go through the source code and try to infer the entire framework. That does work, too, but it's just not as clean sometimes. So, we're
[36:02] happy to help you hand-hold your agents through that process. Please do see me later if you're interested. But, generally this is sort of the workflow that we're seeing in the modern day. And Andreas is going to go into a demo for us. >> Okay.
[36:15] We have not too much time, but let's see. Um I'm going to go to my terminal. Can everybody read the font? Cool. Okay. So, I have a little demo
[36:29] pipeline here. Um It really has not too many files in it. It has the a YAML file that defines it. It doesn't
[36:41] really say much. It says I want to use a database called demo. I want to use this for storage. Now, why do I have to specify storage here if I have a catalog? It's because the checkpoints have to be maintained
[36:53] somewhere, right? So, I have to give it a location where it can maintain that state. Um and then as libraries or files that should be included in the pipeline, I just say everything under
[37:05] transformations. And then in transformations, I have an artifact from a previous run called the Py Cash. Let's ignore that. Um but I have I have three This is actually the same as the example from
[37:18] earlier. It's So, I have um raw sales. This is a Python script. Um mostly what it does is it constructs a
[37:30] schema and then it reads from a CSV file at a certain location um into this table called raw sales. All right. And then we have um
[37:43] this enrichment that ex- that that extracts the date from the timestamp. And then last but not least, we have
[37:55] this This is the materialized view that aggregates by zip codes over the sales of the current year. Um if we want to look at the data, the data looks kind of like
[38:08] this. All right. So, it has like a quantity, a price, a date, and a zip code. In reality, your sales data will have a lot more fields, but for the purpose of this demo, I I it to
[38:21] this. Now, um let's um let's say we want to um uh we have a a typo in one of these. Um so, let's say here I have a typo.
[38:34] And I uh um I misspelled the to_date function. Um so, I have um just to show you, I have Spark SQL running
[38:47] right now. If I do a show tables, there's nothing there. Um so, now let's do a dry run of the pipeline. So, I say spark pipelines
[39:02] dash dash dry run. Now, one thing I shouldn't forget is I'm I'm going to use Spark Connect, right? And my Spark Connect server is running locally on my laptop because I don't have network here. So, um I'm going to say um remote is
[39:16] spark connect local host. I think it's 15002. Um and I do a dry run.
[39:32] Um oh, that was my mistake. I just said dry run, not dash dash dry run. Um Okay. And here we have a problem. We see that something failed, right? There's flows with errors. It's called dated sales, and then
[39:44] there's flows downstream from that that weren't even um really looked at. And if you want to see the actual error, we can see it. Cannot resolve the routine to_date. And it tells me what line in the in the
[39:57] file it is and what position. So, let's fix that file. No. Come on.
[40:20] Now it passes. Run is completed. Of course, we didn't write any data, right? So, we can see that here. Um there's still no tables here. So, now we can run the pipeline. Um So, let's call run here.
[40:34] But let's say dash dash refresh selection. Let's do only the first one, which is called raw sales.
[41:03] Hm, I'm doing something wrong. Okay, let's not do refresh selection.
[41:16] uh then notice the all the all the flows get queued, and then they're starting and completing in in the order that they have to be. And now if we look at um our tables, now we have these tables created. And um we can select
[41:29] from uh let's say sales YTD by zip.
[41:46] And we see this snow oil. This is like an aggregated sales and the zip code. Um yeah, that's all I wanted to show. >> Awesome. Thank you so much. >> Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.