Evolving Apache Spark Structured Streaming in Open Source: A Year in Review and the Road Ahead!

Summary

Spark 4.1 introduces Real-Time Mode for Apache Spark Structured Streaming, enabling operational pipelines to process data with millisecond-level latency — a significant advancement beyond the traditional micro-batch model.
Stateful transformations have been substantially enhanced through TransformWithState and stream-stream join improvements, alongside a checkpoint redesign and state store performance improvements including secondary indexes.
The Spark 4.2 roadmap includes state repartitioning, query evolution, and hybrid sources, with Databricks-specific additions including Photon support for stateful operators and Serverless Declarative Pipelines.

Evolving Apache Spark Structured Streaming in Open Source: A Year in Review and the Road Ahead!

Apache Spark Structured Streaming has quickly become the foundational open-source technology for powering mission-critical streaming ingestion and ETL pipelines worldwide. The Spark 4.1 release introduces powerful updates, most notably a new Real-Time Mode that enables operational pipelines to process data with latencies in milliseconds, all within the Spark ecosystem. Stateful transformations have been significantly bolstered through TransformWithState and stream-stream join enhancements, offering broader coverage and higher performance. Furthermore, robust improvements—such as revamped locking and integrity verification—provide improved stability for stateful queries. Looking ahead to Spark 4.2, the roadmap includes state repartitioning, query evolution, hybrid sources, and other enhancements, further establishing Spark as the gold standard for all streaming workloads. Join us to explore these recent advancements and the exciting future of open-source Structured Streaming!
Talk By: Anish Shrigondekar, Software Engineer, Databricks ; Jerry Peng, Staff Software Engineer, Databricks ;

Chapters

FAQs

What is Real-Time Mode in Apache Spark Structured Streaming?

Real-Time Mode is a new execution model introduced in Spark 4.1 that enables Structured Streaming pipelines to process data with millisecond-level latency, eliminating the per-batch startup and commit overhead of the traditional micro-batch model. It is designed for operational use cases such as fraud detection, real-time recommendations, and trigger-based alerting where sub-second feature freshness is required.

What is TransformWithState in Spark Structured Streaming?

TransformWithState is a Structured Streaming operator that provides a flexible, high-performance API for managing custom stateful logic across streaming micro-batches or real-time events. It replaces and extends earlier approaches to stateful processing with broader coverage and improved performance.

What improvements are coming in Spark 4.2 for Structured Streaming?

The Spark 4.2 roadmap includes state repartitioning for rebalancing stateful queries, query evolution to handle schema and logic changes without losing state, and hybrid sources that combine batch and streaming data in a single pipeline. These features aim to make Spark Structured Streaming more flexible for evolving production workloads.

How does Databricks extend open-source Spark Structured Streaming?

Databricks contributes Photon support for stateful streaming operators, Serverless Spark Declarative Pipelines, and observability improvements as Databricks-specific additions built on top of the open-source Apache Spark foundation. These additions provide enterprise-grade performance and reliability on top of the open-source core.

Full transcript

[00:11] Hello everyone, thanks for joining me here. Uh my name is Jerry. My colleague here is uh Anush. Thanks for joining our talk on evolving Apache Spark structure streaming and open source and beyond. It's going to be kind of a a year-end
[00:23] review. What have we been doing at data bricks um on Apache uh spark and open source also you know on data bricks uh as as well just a show of hands just to make sure like you know gauge the kind
[00:36] of level of familiarity here who who who in here knows what is structure streaming spark streaming okay seems a lot of you do so it seems like we're talking to the you know right crowd of people here
[00:55] so this is kind of the agenda of uh of our talk here. Number one, we're going to just go over have a quick overview of structure streaming just to you know catch everyone up and then we're going to talk about what's uh what's new in Apache uh Spark streaming in the recent
[01:07] um Apache Spark releases. What have we added to basically structure streaming in open source. We'll also talk about kind of more recently what we've added uh in data bricks on for structure streaming. And finally we'll talk about
[01:19] kind of uh you know some what is the road ahead what are the new things that we are doing in uh structured streaming. Let's do a quick quick overview. You know what is stream processing? Most
[01:31] probably familiar with a lot of you since you guys already know what structure streaming is. It's basically stream processing is simply just processing you know you know endless or you know continuous flows or unbounded
[01:43] whatever term you like to use here of data. So data could be coming in from you know uh from from machine logs from IoT devices sensors agents so on and so forth and just keep generating basically
[01:55] these streams of data you want to basically do something with that data you may want to do some sort of pattern detection enrichment routing many use cases right you probably many talks even at you know data and I summit I gave one
[02:08] on basically feature engineering earlier in the week a lot of use cases you can basically be doing with uh with you know that stream of data and processing that stream of data. You know, eventually it can power things like real-time analytics, operational, you know,
[02:21] applications, trigger alerts, you know, so on and so forth. So, this is kind of what we're talking about stream processing and spark. Spark streaming, structure streaming is an engine that enables, you know, every you to do this
[02:33] type of stream processing. So just to you know kind of give a little you know history and context structure streaming is a scalable and fall tolerance stream processing engine
[02:45] built on top of the ex the spark SQL engine. So it does have its roots you know in spark the APIs are the same you can use the same dataf frame API you can use the C same SQL API in spark uh as
[02:58] well in the SQL engine as well as in structure streaming. So why structure streaming? You know, it's a like I mentioned before a unified engine for batch and streaming. It's also fall tolerant things crash. You can
[03:13] basically restart your query and it should be, you know, basically uh restart from where you've left off. It has, you know, low latency and cost effective. It's easy to use because it shares a lot of the same APIs as Apache
[03:26] Spark. Has powerful operators and has advanced, you know, process stateful processing capabilities. high level of how kind of structure streaming you know works. You have like it's kind of similar to the previous
[03:38] diagram but more specifically to kind of structure streaming. So you have on on the left side here you have basically you know your sources it can be from Kafka can be for files can be from databases it can be from you know uh you
[03:53] know lakehouse formats delta iceberg right you can basically read streams of data or you know streams of changes if you will into these kind of ideas of m micro batches. So each microbatch
[04:06] processes a fixed amount of data. You can think of them as you know perhaps like indexes to your data in a file. It says process from line you know zero to five in this microbatch and then from six to 10 in the next one so on and so
[04:20] forth. And this is how basically structure streaming implements continuous processing or stream processing is basically we are having these you know uh these uh units of microbatches which are basically units
[04:32] of data processing and we keep basically running these microbatches over and over again to process any sort of new you know data and we can also write them out to you know Kafka topics delta tables
[04:44] databases you know there are many syncs you know as well but the idea is read from you know sources. See what data is new data is available. Create a microbatch which is just a
[04:57] range of data to be processed. Process the data write the data to the sync and we're going to keep doing that over and over again.
[05:10] So let's talk about um structure streaming and uh open source. Right? This is kind of you know uh the day for talk about open source and Apache. you know what is uh an open source Apache Spark. So let's talk about kind of you
[05:22] know what what the recent developments in this kind of past year on Apache on structure streaming and uh you know Apache Spark open source. So I would say these are some statistics of the contributions on you know structure
[05:34] streaming in the last year. I would say you know there's still a lot of momentum and people willing to uh commit changes and uh add additional features in structure streaming not only from data
[05:46] bricks where you know kind of we are from but also from a various other companies that are also just simply leveraging structure streaming uh in open source as well. Some of the feature highlights uh that
[05:59] went into kind of in the previous year that went to Spark 4.1 and 4.2 too. Number one, one of the kind of exciting topics at least you know for me is uh real-time mode. Um there was a number of
[06:12] talks uh at this data and I summit about real-time mode and structure streaming. Um I gave you know one earlier there were a couple of customer uh talks as well how they were leveraging real-time mode and um structure streaming to power
[06:27] their feature stores you know for feature engineering and how they're able to get extremely low latency with real time mode. So in 4.1 we've uh basically released uh support or open source
[06:39] support for uh stateless queries for real-time mode uh in Scola and also on top of that uh in 4.1 for transform with state which is our kind of uh new and next generation uh
[06:52] stateful arbitrary stateful processor that is I would say leagues ahead of uh the kind of the the the legacy one which is using for flat mapaps group estate. it is like way more ergonomic and way more kind of easy to use. So we released
[07:06] support uh for the pispark API as well as spark connect. Uh for stream joins we made some performance optimizations um basically in consolidating multiple
[07:18] uh state stores uh that were were using for basically each side of the join into basically a single store for performance reasons to basically uh make that a little bit more performant. Uh also state store reliability. We did uh made
[07:32] a number of improvements to really kind of harden uh the state store basically uh reliability and capabilities so that we implemented a bunch of you know invariant checkers to make sure that we're always in an expected kind of uh
[07:45] state. also state data source. Um, we've added additional support to basically be able to read and debug checkpointed state like for example the changes we've made
[07:58] in stream joins in terms of kind of the state layout to improve performance. We've all added support basically for the state store reader. So you're still able to go and inspect the state uh for basically these uh these these operators
[08:11] that we might have uh you know made modifications to. Let's move on to Spark 4.2. Um, we made further improvements again to basically stream stream joins. One of the things we added is you know faster
[08:24] watermark eviction. Um, that basically uh went into 4.2 state store and so additional basically work on state store. We added uh things to help the kind of durability of the
[08:38] state store and kind of the integrity of the data stored in the state store such as the rowle check sums and automatic uh snapshot repair in case like oh you know there's too much lag and how often you
[08:50] basically uh uh are snapshotting kind of your your state. We've added basically additional capabilities there to make sure that you know we are you know frequently doing snapshots. So on replay, you're not necessarily like
[09:02] replaying a bunch of change logs and it takes a long time for you to restart and things like that. Another really cool feature right here is source uh source renaming and query evolution. Now you know with basically this functionality
[09:14] you can, you know, add remove you know sources in your structure streaming query. So that's a super cool and powerful feature. Imagine you know uh it can power a number of use cases. One of
[09:26] them the more the more obvious one is for example some sort of backfill right uh you can first have your query read from a delta table iceberg table S3 whatever right do a back fill run it and
[09:38] then you can change the source now I want to process live data I will basically start reading from Kafka now right I can change my query and uh choose to use a different source and start reading more live data from from
[09:51] Kafka right things like that of course there's also use cases like Oh, you know, I just want to start reading from, you know, use a different source because, you know, my my requirements for my query has uh has changed or, you
[10:04] know, the kind of environment in which I'm running my query, you know, or Spark has changed as well that I want to read from a different kind of a uh uh source. And in 4.2 for RTM, we also added, you
[10:18] know, Py Pispark support as well. So you can run RTM queries, real-time mode queries uh using the Python API as well. So let me just kind of highlight a
[10:30] little bit about real-time mode. You know what it is, you know, uh at a at a very high level. So what is real-time mode? It's something that, you know, I've worked on quite a bit in the past couple of years. My team has worked a bit in a
[10:43] c last couple of years. You know, data bricks has invested a lot, right? As you as you probably heard in this data and I summit this mentioned in multiple different kind of uh environments. So what exactly it is is basically allowing
[10:56] structure streaming the stream processing engine and spark to basically process data with millisecond latencies. And one of the cool things is we're not requiring it's not a different engine.
[11:08] It's not a different API, right? It doesn't require you to say okay like you know change a lot of things. All you really need to do to enable real-time mode is just basically a config switch or one line basically code change. Just
[11:20] specify instead of using whatever trigger you're using now to use that real-time trigger real-time mode trigger and you will be able to see kind of the performance uh or the latency kind of uh the latency decreases and the
[11:32] performance gains. So that is you know at a high level what you know real time mode uh you know does how does it work at a high level right um some of the
[11:44] concepts here is instead of I mentioned before micro batches were basically uh a range of data from say a to b from index 0 to 5 right we're basically processing data in the in the current you know in
[11:58] the uh existing stream structure streaming engine in that manner there's some issues with that when it comes to kind of latency that's why you know structure streaming is very well used for things like ETL but when it really
[12:11] gets to those like under a second kind of latency use cases it struggles you know a bit so we had to kind of revamp how execution works in structure streaming to be able to bring those extremely low latencies. So one of them
[12:25] is instead of just saying every single microbatch is okay this basically fixed data range right we you know at the beginning of microbatch we look at what is the data available we basically now say oh the data available is between 0
[12:37] to 5 for this next microbatch we're always we're going to process that data range 0 to 5 instead of doing that what we're going to say is okay we're going to run basically these micro batches if you will for a fixed amount of time
[12:49] let's say you know five minutes two minutes three minutes whatever you know you want to do there. So what really happens here is we are evolving or changing kind of the uh the the the how
[13:02] we're checkpointing essentially instead of checkpointing based off of amount of data we processed really checkpointing based off amount of time. So we've kind of evolved the uh processing model to be based on to be basically time based
[13:14] checkpointing if you can think about it instead of micro batches and we're checkpointing basically the amount of data we've processed now we're checkpointing basically after say five minutes what is the progress we have done and we're continually processing
[13:26] new data you know during that time and the next thing is concurrent execution if you are familiar with a structured streaming or spark in general how a spark job or an application runs is you
[13:39] So if there's multiple stages within your your your Spark job, right? Let's say map reduce map stage runs first, all the tasks finish and then reduce stage runs, right? What we also changed is instead of doing that in a sequential
[13:53] order, now all the stages and all the tasks of all the stages run concurrently. So now you have a real pipeline kind of execution so that data fl uh flows kind of seamlessly and in a
[14:06] streamlined fashion from upstream to downstream so that when data is immediately proc processed finish processing in upstream it can be sent downstream to be processed immediately as well and that really also minimizes
[14:19] latency. Like I mentioned, this is kind of a diagram to kind of illustrate that before. Microbatches, how we're basically checkpointing is, oh, here is
[14:31] basically, you know, a start and end offset. That's a micro batch. We process that data. We checkpoint on progress and we move forward. Instead of doing that in real time mode, we're just going to move to more of a, you know, timebased checkpointing kind of pattern. we're
[14:44] going to run for a fixed amount of time and then we we are going to basically checkpoint to wherever we basically process to up to that time. So now it's more like a checkpoint interval that we
[14:56] are using in in real time mode. And the other thing I mentioned is basically uh scheduling right instead of running one stage at a time in real time mode we're going to run all stages uh at the same
[15:09] time. And you may wonder like well how do we actually do that? Because if you know in in Spark the shuffle is uh is not really incremental. You have to shuffle all the data first right into
[15:22] the sort merge shuffle that's there and it's basically using block manager to store it and then that's why the next stage comes and reads that shuffle data and that's how processing is done. So one of the other things we did for real-time mode is we basically impro
[15:35] implemented a new shuffle. It's we call it the streaming shuffle but really it's basically incremental shuffle. It allows basically downstream to start processing and reading that shuffle data before
[15:47] upstream is you know done processing it right that's kind of the general idea. So if you're if we if we basically have stages or tasks running concurrently you would need to basically have an
[15:59] incremental shuffle or in this case a streaming shuffle is what we call it to be able to basically communicate between the tasks of different stages. And here are some actually the current
[16:11] numbers are are better. This is somewhat um older older data but it it it is kind of it gets the point across that this is a comparison between real-time mode and the baseline here which is the existing
[16:23] kind of execution mode. So a as you can see for a number of kind of benchmarks or query types that we've uh benchmarked against real-time mode um delivers an order of magnitude um
[16:37] uh latency basically decrease or performance gain however you want to say it um against the existing kind of execution execution mode. So you know you're now not talking about in the
[16:49] order of seconds or multiple seconds we're talking about now in the order of you know milliseconds. That's kind of uh you know where real-time mode allows uh the query processing time latency to be.
[17:05] Cool. So we've talked about real-time mode. Let's talk about some of the other things we've done as well. Um one of the things here is the stream stream join improvements that I kind of touched upon. So we've been kind of continuously uh working on improving stream join
[17:19] performance because it's one of the kind of you know harder problems to solve in stream processing and one of the more kind of intellectual problems to solve there. So people love to kind of work on this. So we so number one we we've added
[17:32] like I mentioned before uh you know virtual virtual column families. So instead of having say four Rox DB instances that we were using before, we can just kind of uh have one right and
[17:44] uh basically uh that also kind of helps performance and also kind of resource uh resource management or resource utilization for us. We also added uh you know in 4.2 to state format V4 which you
[17:57] know we implemented secondary indexes which allows basically quicker watermark eviction so that if you have like a pen-like query and you need to basically you know flush windows it basically allows you to do that more more
[18:09] efficiently. uh kind of what is similar is what is mentioned there the performance inviction we basically have optimized you know that to optimi we optimize the scanning logic to significantly reduce
[18:23] the uh latency reduction so that it's a lot more kind of uh efficient when we're uh evicting state that you know needs to happen when we are uh flushing windows and whatnot state availability I mentioned before it's you know we've uh
[18:37] all the different kind of things that we've changed uh we still basically have uh allow users to read and observe read and debug what the state is in the kind of uh in Rox DB and for users to be able
[18:50] to read that out. So any single anything any changes we have done you know the the reader API the existing reader API is uh is is compatible. We also made some other kind of uh optimizations
[19:02] there for you know improvements and left semi and non-outer joins and things like that. Really at the end of the day it all kind of ties back to kind of these kind of these two things that we have uh we have done which is basically virtual
[19:15] column families and secondary indexes for watermark eviction.
[19:27] like pretty revolutionary in terms of you know arbitrary stateful processing for uh for structure framing. If you're kind of, you know, uh, in have been using structure streaming for a while, historically there was this whole thing
[19:40] called flat map groups with state and it was you know notoriously you know complex to use and there was a lot of gotchas. So we introduced you know transform with state and it's a lot more ergonomic right and I think you know we
[19:53] have a ton of usage now with this new kind of a new API for our users to to express arbitrary stateful uh processing logic. So this is an example here. You can basically you know uh you know
[20:06] declare basically a stateful processor declare your stateful variables use them in this handle input rows and then you can plug it into your structure streaming query and uh and use it and all the state management and fault you
[20:19] know fall tolerance failure recovery is uh is handled for you. So some of the improvements that we've made for you know transform with state uh you know just as a refresher it was
[20:32] introduced uh in spark 4.0 uh zero. Um later we've added additional support for like the par the pispark API also spark connect support and uh due to kind
[20:45] of feedback we've also added better testing. We added this thing called TWWS tester that allows you to run you know actual kind of uh very well-defined unit tests against your stateful processor. So for users kind of developing their
[20:58] transform with state logic it's a lot easier to test it and make sure that it runs correctly. And again this is the you know comes back to the state reader thing I mentioned before is uh we've also uh you know integrated
[21:12] uh the the transform with state with the state reader so that uh even for a live query you can go and observe what exactly is the state held in your kind of transform with state operator.
[21:30] Um just to kind of uh touch up on more of the state management improvements like we've made you know efforts to basically harden um the state management because we know that it's very you know critical and mission critical that it
[21:42] works correctly and it behaves correctly right if not then you know there could be many kind of correctness issues and the uh data that's being processed. So one of the things is we've implemented things like state machine locking to
[21:55] make sure that there is only one writer or owner of a Rox DB instance. So we don't get basically you know kind of a multi-riter situation that kind of uh that each one walks on top of each other
[22:07] and you get uh inconsistent results. We added file level integrity checksums to make sure that you know there's not any sort of corruption that's uh happening there. And I have mentioned
[22:20] before uh in resilience and repair we have the rowle check sums again to just make sure that we have integrity of the data and automatic snapshot repairs to make sure that you have consistent snapshotting. So on restart we don't
[22:33] have to replay a bunch of change logs for you to kind of uh uh get back to kind of the the the the state that you were before the failure happens. And in ter and for storage and
[22:46] architecture there's some improvements better integration with kind of spark itself in in terms of the memory manager we've uh added uh integration to make
[22:58] sure that whatever whatever memory usage especially for things like rosb that uh the the spark memory manager knows about that as well and won't be in a situation
[23:10] where spark will try to kind of don't think that we're anything is using that memory and then gives that out to something else like a shuffle and then it causes things like a like a like a ohm. We also introduced this thing
[23:24] called checkpoint uh format v2 which also gives you more uh more more correctness guarantees for various like corner cases as well.
[23:41] This is uh what I mentioned before but here is like a concrete example. Also something super cool like I mentioned before is source naming and evolution. You're able to change right sources and syncs in your query to do things like backfills or whatever you you can
[23:54] imagine right you know changes of circumstance of what systems are being used to ingest data to write data to you can basically now do that uh in open source spark you can just change the source like the example here one you're
[24:07] basically you know reading from a table then you can read from you know uh uh another uh you know kafka stream as
[24:21] Uh let me talk about the uh road ahead here and kind of 4 do Spark 4.3 what to kind of uh expect. So like I mentioned one of the things that we really work on and focus on is real time mode. Um so
[24:35] we're going to be open sourcing and releasing support for stateful queries in real time mode. I'm actually um just in the process of doing that. Currently, there's already PRs out in open source.
[24:48] We're open sourcing kind of uh stateful query support in real time mode. So, you know, all stateful op stateful operators would be supported um and uh running real time mode and open source starting 4.3.
[25:00] Uh operator naming and uh evolution uh basically we will add uh more support for that. Um and also machine learning extensions which is we've added some
[25:12] additional operators for basically machine learning purposes like K means and vector search capabilities in there as uh as well also for API enhancements. One of the things we're I don't know if you're aware of we also open source like
[25:25] STP uh in in open source as well. So, we're going to be adding some additional APIs to better support kind of this structure streaming and the advanced stream processing use cases for for STP
[25:37] uh SP as well. uh in kind of related open source news that is sort of related to structure streaming um as well is uh what one of
[25:50] the things also my team is working on is allowing uh allowing UC manage tables especially in data bricks to be read by external kind of um you know services
[26:03] right so one of the asks is like okay I have say a delta table or icebrook table that's managed you see which is a catalog right you can catalog instead of you know having to use say data bricks
[26:15] you can really now use anything you can write your own application you can use spark you can use another engine or you can use spark not offered by data bricks you can you can use open source spark to basically read data from a managed table
[26:28] uh and process it so that's another thing I think there was some announcements uh some talks around this as well that's another thing basically uh we've worked on is is basic is improving the kind of delta and iceberg
[26:41] connectors that we have to be to allow users to have this sort of kind of open source and federated access access. All right. Uh I'll hand the presentation
[26:54] off to my colleague here that will talk about uh some of the capabilities we have or things we've worked on at data bricks. >> Awesome. Thanks uh Jerry. So uh so far we spoke about you know what is there in
[27:06] open source spark. Um next you know we'll talk about what is there in structured streaming in data bricks you know that is not there in open source today but kind of you know you might get a sense for what we might uh look to
[27:19] introduce uh in the future. Uh so on DB you know as you many of you might also be running your streaming workloads on data bricks we have customers pretty much in all the verticals that you can imagine uh from
[27:31] digital natives to retail to media uh to healthcare to so on and so forth. So a ton of uh adoption and usage for structured streaming uh within data bricks. And uh to just to give some context uh you know few years back when
[27:45] like I think when we initially started working on this we had maybe one one and a halfish million or maybe even less than that queries u running every week. Now we have north of 8 million like close we reaching almost 10 million
[27:57] streaming queries that run every week on the data bricks platform. uh we have almost 35 to 40% uh plus job growth still continuing to to increase uh year-over-year all the way from stateless queries stateful doing all
[28:10] sorts of uh use cases uh using streaming on on the data bricks uh platform uh and why sort of you know customers choose to run structured streaming on data bricks is essentially we have full
[28:22] compatibility API compatibility with uh whatever we have in open source spark you know we pretty much actively contribute to oss spark I'm a committer. Jerry Jerry has a bunch of contributions. He keeps committing as
[28:34] well. Uh and we keep adding and making it life better for our open source users. But on top of that, we also have a bunch of additional features uh that we only support today on the datab bricks ecosystem. Uh one is of course
[28:46] the connector ecosystem. Uh we have a ton more support for different connectors. Uh I think uh Kinesis, Pulsar, Google, PubSub, um so on and so forth. Uh on the sync side as well, we
[28:59] now have support for lakebased JDBC syncs uh and so on. Um from a performance perspective, uh we want to make sure that you can run your structured streaming queries in the most performant manner on data bricks. Uh so we have additional checkpointing
[29:12] mechanisms. We have autoscaling mechanisms. Uh we have pipeline execution that gives you uh the optimal uh throughput and cluster utilization on top of uh your existing queries uh in data bricks. Uh and uh finally we have
[29:26] security and isolation. So uh a lot of our go a lot of our enterprise customers care about governance security uh and for them we make sure that you know all of your streaming queries are natively integrated with UC. Uh they are
[29:40] integrated with something called league guard which allows for isolated execution of your queries on shared clusters so that you know one user cannot see what the other user is writing or reading. Uh so you have complete sandbox isolation uh for your
[29:53] queries as well. Uh and the combination of all of these things you know uh helps fuel the growth uh for streaming workloads uh on data bricks uh in general. Uh with what we have for
[30:05] real-time mode in open source uh as what Jerry mentioned we have some additional features that are only present on data bricks currently. Uh one is on the source side we have support for kinesis. Currently we have support for kinesis in
[30:18] efo mode. We are very soon going to launch support for polling mode as well. So if your sources or if your queries are reading from kinesis you want to run them in realtime mode please try out uh the offering on on data bricks. Uh same
[30:30] for like you know some additional capabilities for Kafka where you can uh do some bin packing for how you want to schedule uh the Kafka uh partitions on a
[30:42] single task. So we allow for running more than one partition on a single task. So you can basically have um more uh CPU or cluster utilization if if some of these parts of your query are bound.
[30:54] Uh on the sync side we have native legbased support. You can write to legbase uh within data bricks. Uh from a perf optimization point of view additional uh features around async progress tracking async state checkpointing uh optimizations on the
[31:08] Kafka source to make sure that you know we are not spending additional time on collecting and tracking metrics. uh with stream join uh again native support with the v4 format on RTM some of these things will will land in in open source
[31:21] as well but some of them are are currently proprietary to data bricks uh and then observability you have native metrics that we export as part of the query progress you can get them through your query listener uh so you can track
[31:34] end toend uh latency metrics on a P90 P95 P99 basis as well uh and verify you know whether you're getting uh end toend record latencies um in the millisecond ends range. Uh again we have a bunch of
[31:47] benchmarking that we did against Apache Flink. Uh so you know as Jerry mentioned we have had uh uh sort of a case where a lot of the ETL injection workloads high throughput workloads were uh dominated
[32:01] dominated by structured streaming. Uh but the low latency workloads uh were uh you know sometimes people were running uh Apache Flink. Uh so for a lot of stateless queries for feature stream
[32:13] static joins based based on feature enrichment uh aggregations stateful queries uh stateless transformations for encoding we see that we are equal or better than fling uh for many of these workloads and we keep making sure that
[32:26] you know our numbers uh keep going down further. Another thing uh that we wanted to call out is declarative pipelines. uh we are uh pushing heavily uh for a large class
[32:38] of users to run their streaming workloads on declarative pipelines and we are uh happy to announce that real-time mode is also now available in public preview uh on spark declarative pipelines uh so it's just a small config
[32:52] change you just uh set the pipeline trigger to real time and just set the trigger interval uh and we will essentially make sure that your query runs in real-time mode uh on SDP so you can now model uh your uh queries uh as
[33:06] pipelines. You can get all the dependency management. You can get all the upgrades, all the pinning support. Uh everything is taken care of for you with our managed uh declarative pipelines offering.
[33:19] Okay. The next feature uh I wanted to talk about which currently is only available in data bricks is state repartitioning. Uh so as many of you might be aware you know we always uh structured streaming always had this
[33:32] problem where your shuffle partitions uh were static once you created your query and your checkpoint uh you could not change it once uh the query started running. Uh we have now removed this
[33:44] limitation. So you can basically stop your query change your shuffle partitions restart your query. We'll run a control batch before the next batch runs. make sure that all the state is now uh repartitioned using the new number of shuffle partitions. Uh so this
[33:58] will help all of your autoscaling like you know requirements like if you want to scale up you want to add more uh number of workers. Uh but you know previously you could not uh add number of uh state partitions you can now do
[34:10] that. If you want to uh go down you can reduce the number of state partitions without compromising on performance and uh reducing cost. Uh so this is you know uh one of the newest features available with the 18.3 release again in popr. We
[34:25] have a lot of customers already who have tested it out. I have a blog soon uh but you can get a preview for the uh syntax and the feature uh this slide. Um some other performance improvements again
[34:37] specifically only on data bricks we had a session uh I think couple of days back where we discussed about photon for stateful queries. Uh so typically streaming queries were using photon only
[34:49] mainly for u scan performance like you know if if you're reading from delta writing to delta where the data already is in the columner format uh for stateful queries we previously had not done a lot of uh photonization uh but
[35:03] now we are adding a lot of support for photonizing uh stateful operators as well uh and we see a lot of throughput improvements for uh drop duplicates aggregations joins uh all the built-in
[35:15] operators uh will be released uh with photon support very soon. Um Rox DB is optimized for your use cases on data bricks. You've done a lot of additional work to make sure that you run with the
[35:27] most optimized state store for performance for cost uh for throughput latency and so on. uh and we also have additional metrics uh that give you fine grain visibility into what your state store is doing whether it is lagging how
[35:39] we are taking action on that are we doing automatic repair are we doing u uh snapshotting and and so on and so forth uh in general we have a new API along
[35:52] with which is sort of a parallel API to the existing uh structured streaming API in the jobs land uh so you can have almost a onetoone mapping for uh your existing streaming queries in SDP. So if
[36:04] you were writing a streaming query with uh the output mode as append, we have something called append flow. If you are writing with update, we have something called update flow. Uh if you're doing uh merge in for each batch uh and you're
[36:16] trying to do CDC uh into a target delta table, we have native offerings for that with auto CDC. uh and you'll see us launch more and more abstractions that make it easy like super easy for for users to do native ETL native ingestion
[36:29] u well-known transformation logic and so on on top of uh declarative pipelines uh and uh as we discussed all the the cool benefits that you'll get you can do selective uh restarts you can do reset
[36:41] on a specific uh part of your DAG and so on and so forth all right then in data bricks uh what are we looking forward to in terms of the road ahead. Uh one of the features
[36:55] uh that again you might have heard yesterday in a couple of your talks we are launching stream streaming time travel. Uh so essentially what you can do is you can go back in time for where
[37:07] your query was. You don't need to reset your checkpoint. You don't need to do a full refresh uh or you don't need to start with a new checkpoint. You know that okay you know you have figured out that your records are corrupt from some
[37:20] point in time. you want to go back, you want to change your logic and you want to continue from there. Uh all of that will now be done seamlessly. Just uh a simple change uh on your pipeline, just roll back and uh continue processing
[37:34] from there. In declarative pipelines, uh you can just do it on the leftmost side of your graph. We will handle all the end toend cascade uh so that you get end to end exactly one semantics with fanin fan out support just by invoking time
[37:48] travel um on uh your leftmost part of the DAG. Thank you.
[38:01] this is not going to be available in jobs. Another reason for you folks you know to try declarative pipelines. Uh but the only reason for us to not offer it today in jobs is because uh we don't have a full view of your DAG. We don't know you know how you're processing
[38:13] bronze, silver, gold which we do have on the SDP side. Uh so initially at least it's going to be uh fully managed in SDP and uh we will roll out uh very soon uh in in the next few weeks.
[38:26] Uh another uh thing that I wanted to call out is uh seamless upgrades to uh Spark declarative pipelines. uh if you have your existing uh structured streaming jobs you want to migrate to uh
[38:38] declarative pipelines uh we are basically I'll asking like you know you don't need to do any additional changes on your part we are supporting something called import checkpoint on your SDP flows uh so your SDP flow can just point
[38:51] your existing checkpoint location uh and we'll make sure that it becomes pipeline managed and your existing UC delta tables or iceberg tables uh sorry currently UC managed tables uh don't have to change at all. We are not asking
[39:05] you to clone your uh tables or anything like that. Uh they will just uh you just need to set a config or a table property on your table and it'll become a declarative pipeline managed table. We'll clone the checkpoint in the most
[39:17] efficient manner uh possible because you know we control the internals uh and your existing query can just start processing in SDP. So stop your job, just run import checkpoint with SDP,
[39:29] restart your uh job, restart your SDP pipeline, and you're good to go. No additional changes required. Okay. Um the next feature that we're working on is adaptive online
[39:41] repartitioning. Uh so you know sometimes it's hard for users to figure out uh what is the number of shuffle partitions that are needed. uh if folks are on the batch side folks are familiar with AQE aos for autooptimize shuffle for
[39:55] adaptive query execution uh we're basically launching the same thing for stateful queries uh so you can basically set your shuffle partitions to auto you don't need to worry about how uh they need to be set we'll adjust based on you
[40:07] know target freshness or like cost controls whatever you know parameter you want to adjust against while maintaining uh end toend uh latency uh or throughput you know depending on the on the use case that you
[40:19] uh without affecting any of the of the query performance. Uh so this is uh something that is uh coming soon and we are going to top it off by also supporting real-time mode. Uh so for your stateless queries for stateful
[40:32] queries we'll do all the adaptive partitioning figure out what the source partitions need to be what the sync uh writer needs to do what how the shuffle partitions need to be managed uh and so on. Um as already mentioned for open source
[40:46] we are launching something uh where you can add remove sources. Uh so you can start a query uh with let's say delta uh with a certain number of shuffle partitions. You can process it with high throughput. Uh you can remove or change
[40:59] the source to point to Kafka. Uh you can repartition so that you have a lower number of shuffle partitions. Uh and then you can restart your query in realtime mode so that you can get the lowest latency on your live sources. uh
[41:11] all of that you can do you know as sort of an imperative thing we are also going to launch um sorry about that uh we're also going to launch uh something where you can do this
[41:24] seamlessly in spark declarative pipelines uh so we are going to have uh way for you to just note okay this is a flow that depends on this flow and this is what I want to to write to so we will
[41:37] effectively make sure that the back fill happens in the correct order with the necessary number of shuffle partitions and will automatically trigger all the background repartitioning and then do all the rerouting from there.
[41:50] Um then in terms of real-time mode uh autoscaling on on SDP uh again you know one of our our focus areas also with adaptive is something that we're working on. Uh for watermark propagation u a lot
[42:04] of our customers have low volume data or sometimes they don't have a lot of incoming data. In those cases uh what we see is that you know if you have transform with state you have event time timers they don't fire or if you want to
[42:16] emit something in append mode we were not able to close the window and we are not able to make progress. Uh so we have launched uh a new feature where we are now doing uh control messages and watermark propagation so that we're not
[42:28] tied to how or when your data is arriving but we can make sure that your windows are closing in time that your timers are firing in time and so on. So you'll get all the callbacks and everything within the necessary SLAs. Uh
[42:41] along with that we continue to make sure that you know we remain the best engine especially on the data brick side for performance for TCO uh for efficiency you know making sure that your utilization is always uh sort of at at
[42:55] the maximum uh and we continue to improve on performance for various stateless operators for various parts of uh our execution engine. All right, with that, uh, that's all the
[43:07] the slides that we have. We're happy to to take questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.