What’s New in Apache Spark™ 4.1?

Summary

Apache Spark 4.1 introduces four major feature categories: Spark Declarative Pipelines for defining data pipelines declaratively, an official Real-Time Mode in Structured Streaming for sub-second continuous low-latency processing, expanded SQL language features, and significant PySpark and Spark Connect improvements.
SQL Scripting is now generally available with loops, exception handling, and variables; the VARIANT type for semi-structured JSON data with Parquet shredding is also GA; and new approximate sketch-based aggregate functions for top-K, quantiles, and theta sketches are included.
PySpark gains Arrow-native UDFs and UDTFs, Python Data Source filter pushdown, and improved Python worker logging, while Spark Connect adds a native JDBC driver and GA support for Spark ML on the Python client.

What’s New in Apache Spark™ 4.1?

Explore the exciting new features of Apache Spark 4.1 with major new capabilities for both data engineering and analytics workflows. We introduce Spark Declarative Pipelines (SDP) to let users define what their data pipelines should accomplish while Spark orchestrates execution, dependencies, parallelism and retries, an official Real-Time Mode in Structured Streaming for sub-second, continuous low-latency processing, and extensive improvements to the PySpark ecosystem with Arrow-native UDFs/UDTFs, Python Data Source filter pushdown, and better Python worker logging. The release also brings Spark Connect enhancements including GA support for Spark ML on the Python client and increased stability for large workloads, and expands SQL support with GA SQL Scripting, the VARIANT type with shredding optimizations, support for recursive CTEs, and new approximate data sketch functions. Spark 4.1 prioritizes higher-level abstractions, real-time performance, richer Python support, and more.
Talk By: Daniel Tenedorio, Sr. Staff Software Engineer, Databricks ; Wenchen Fan, Senior Staff Software Engineer, Databricks ;

Chapters

FAQs

What is Spark Declarative Pipelines and what problem does it solve in Spark 4.1?

Spark Declarative Pipelines (SDP) lets users define data processing logic in SQL, Python, or Scala without specifying how execution should proceed—the engine handles orchestration, dependencies, parallelism, and retries automatically. This reduces the complexity of building and maintaining production data pipelines compared to writing imperative PySpark notebooks.

What is Real-Time Mode in Structured Streaming and how does it achieve sub-second latency?

Real-Time Mode in Structured Streaming is an official Spark 4.1 feature that enables continuous, low-latency processing at sub-second intervals. Unlike micro-batch streaming which processes data in discrete intervals, Real-Time Mode keeps evaluation continuously up to date, making it suitable for use cases where even small batch windows introduce unacceptable latency.

What SQL language features are new or generally available in Apache Spark 4.1?

Spark 4.1 brings SQL Scripting to general availability with support for loops, exception handling, and variables for procedural SQL logic. The VARIANT type for semi-structured JSON data with Parquet shredding optimizations is also now GA, and new approximate sketch-based aggregate functions for top-K items, quantiles, and theta sketches are added.

How does Apache Spark 4.1 improve the PySpark development experience?

Spark 4.1 introduces Arrow-native UDFs and UDTFs for significantly faster Python user-defined function execution, Python Data Source filter pushdown to reduce unnecessary data reads, and improved Python worker logging for easier debugging. Spark Connect also adds a native JDBC driver and makes Spark ML on the Python client generally available.

Full transcript

[00:08] My name is Daniel if you haven't seen me already this morning. Um, I'm here with my colleague Wan and we're here to talk to you about Apache Spark 4.1 and all the awesome new features that
[00:22] community has been working on. Thanks for coming. I appreciate all of you guys. Appreciate the community. Um, Spark is what it is because all of you. So, thank you.
[00:35] Yeah. So, kick it off with congratulations for Spark being around for 12 years and counting. It's only the beginning. A lot of awesome things that we've built and
[00:50] we've used Spark to do powerful data crunching and jobs all around the world. So I want to kick this off with a thank you. Congratulations. Yeah. So um
[01:04] some numbers to keep uh to kick this off. Um Spark is heavily used, right? There's over there's thousands of contributors depending on how you define it. Um
[01:18] there's tens of thousands of Stack Overflow questions and answers about it. um back when Stack Overflow was still a thing and not replaced by LLMs. Uh 80%
[01:30] plus of the Fortune 500 are using this. So it's pretty much everywhere. 48,000 commits so far and counting. So there's lots and lots of work going into this.
[01:46] Yeah. So Spark was born in the University of California, Berkeley in the AMP lab as a research project. Um, and
[02:00] Cool. So, we'll go over the new features. Um, there's kind of four major categories we can break this down. Um, STP, Spark declarative pipelines. That's one category
[02:12] where you can define your logic declaratively and then the pipeline will run. Um, real-time mode and structured streaming is pretty awesome. Now I can do things with keep and keep keep the
[02:24] evaluation up to date. Um SQL language features, one of my favorites personally, and then Spark Connect and Pispark um productivity. So those are the main categories we're going to go over.
[02:36] Awesome. So we'll start with Spark declarative pipelines and uh go over the cool stuff that we've added recently. Right. So what's the idea here? Um the idea is you can
[02:52] use SQL or Python or Scola to declare the processing you want to do and the engine will take care of keeping it running for you.
[03:05] Right? So instead of imperatively saying do these steps with these details and these micro batches and everything all users generally need to do is define the data sets and the transformations that
[03:18] they want to perform and Spark will turn that into a workflow for you right um you can define these in Python Python's big in the data science
[03:30] community as we all know um you can integrate it with your other Python libraries that you have. Right? So this is a one major way to use declarative pipelines. Here's an
[03:43] example. Right? So you can say import some dependencies and then define some declarative pipeline tables or materialized views and how they're how they're populated and
[03:55] then kick it off. Right? So here's another more complicated example. define a materialized view with a name and then um populate it by means of a data frame
[04:08] starting with the orders table perform some joins and aggregates on that. Cool. Um you can also use SQL. Um so SQL has a
[04:20] special place in my heart uh as well. So you can kind of define define what you want it to do. Um SQL is an ANIE
[04:32] standard. So you can you can transfer your knowledge around any place that SQL the SQL standard is respected and define your logic that
[04:44] should work roughly the same. Right? So here's here's a corresponding example of how to do that with SQL. You create materialize view, create streaming table commands, and then define how they're
[04:57] populated. Cool. So, how do you actually build this stuff? Um you can use Spark Connect to do it which means um the magic with Spark Connect is that you can define
[05:11] your data frames on a client and then you can run the code and when it's time to kick off a job or evaluate a data frame. We dispatch that work to a
[05:24] separate cluster, a Spark cluster, the driver is not on the same process, right? This gives you isolation and lots of other magical things like you can upgrade the Spark cluster separately from the client. You have many
[05:38] different languages to run your Spark clients. Lots of magical things. Cool. So let's think about when you run these pipelines, how do we keep them reliable?
[05:50] And what does that even mean? Right? So reliability is kind of a core critical part of this. Um it's not just about making it easy to
[06:02] define the logic although that is awesome but it's also nice to define how to checkpoint and perform retries in case of transient failures which generally happen over time and the
[06:15] longer this thing runs the higher the chance gets that some flake will happen. And it's an if not when when not if kind of scenario,
[06:29] right? And so the one tool we have in our toolbox is we can separate a full refresh from incrementally continuing, right? And this means that you don't
[06:42] always have to do a full refresh in every scenario. You can if you feel like it's the right step to take, but it's not always necessary. So you can the API lets you make it explicit,
[06:55] right? So here's an example where we can continue from an existing checkpoint or um recomputee the whole graph and you can kind of differentiate explicitly.
[07:09] Awesome. And then when you're prototyping this stuff, um it's often useful and necessary to validate that it's going to work before you kick it off in production. So this validation step kind of gives you a key
[07:23] to do that. You can run some basic checks on it. Make sure this passes. If this doesn't pass, your pipeline definitely won't work, right? So you use this dry run
[07:35] option, the dry run command. Um, and then just kind of like step through all this the logic, make sure it looks good to the validator before you actually start it production.
[08:00] much, right? You don't have to imperatively define how all the retries work and repetitively do that over and over again. Um instead you can just define your logic what you want the data to do and then let Spark kind of concern
[08:13] itself with those things. Awesome. Okay. So we find when we run these pipelines a lot, we find they need to connect to things.
[08:27] And when you have a lot of connections to things and a lot of tables and functions to manage, we find that cataloges kind of become necessary as a way to abstract that out
[08:39] and kind of hold collections, right? So um a big extension we have now spark 4.1 we have the stored procedures API for cataloges. So now we have the
[08:52] generalized extension point where you can add a procedure and say call x in at the point when you want to run the store procedure and this is a thing that
[09:06] you can kick off. It's allowed to have side effects. It's allowed to change state that's persistent outside of the pipeline itself. Right. So this is in contrast to queries where generally functional questions you
[09:19] ask and you get an answer and they don't generally write to disk or send RPCs or otherwise have external state like that right and so cataloges as we find they
[09:32] often have more magical things than just tables and schemas but they can also have governance hooks procedures functions starting to see AI I have AI
[09:45] entities in there like AI models, right? So, procedures kind of fit in here. Um, yeah. So, as I as I mentioned, procedures are a natural place to put
[09:58] things that change the environment, right? When you say call X and it like does a bunch of stuff, you don't necessarily want to automatically retry this. It might be slow. Um, it's
[10:10] necessary. You don't want to duplicate it. You want to run it once and exactly once it's natural place. Cool. So that's Spark declarative pipelines.
[10:23] Um let me switch to second topic. This is one of my personal favorites, the SQL language. So what cool things do we have in the SQL language in Spark 4.1?
[10:38] First one is SQL scripting is now GA which is really really big deal. Um this lets you write programs with multiple statements in them
[10:51] and run it as a procedure, right? And so generally not not many open-source se SQL engines have scripting and SQL
[11:03] procedures, right? It's it's not it's not a something that everyone has. Um, but it makes sense to have this in Spark because you can do longunning jobs and so we're excited to kind of stamp it as
[11:16] G at this point. What are some things you can do in SQL scripts? So you can think of SQL script as the body of a procedure and it can it runs the SQL statements one by
[11:30] one. It can do things like loops and and if statements. Um, another thing it can do is exception handling, right? So you can have this continue handler construct where you can catch exceptions,
[11:43] signal errors, catch catch exceptions and respond to them in ways that make sense to you, right? Um you can another feature we
[11:55] have here is declaring variables. You say declare and then you declare the variable name and maybe give it an initial value. You can now do a string of these with one declare statement. If it's being repetitive, you have like 10
[12:07] variables or something. Say declare, it's like key equals value, key equals value, makes things easier to read. Cool. So, here's an example of what this looks like, right? We have a begin and
[12:21] end block defines the SQL script. Some variables at the beginning. We have this declare continue handler um statement which basically defines error handling behavior in the event of
[12:35] some problem. Some DML DDL and then we have this for loop. So this is kind of like one of the magical things. You can iterate over a relation, decide what to do, and then if you have
[12:49] a condition, you can signal an exception.
[13:03] way to simplify your conditional logic. If you have a bazillion places that you want to check for erroneous conditions and have like a bunch of nested if blocks. Um you can simplify a lot of that with this, right? All you have to
[13:15] do is signal the condition of interest now and you could just jump it'll jump you directly to the continue handler. Awesome. Okay. Second big feature,
[13:29] variant type is now GA. So what is this? Why is it special? This is now enabled by default. And if you haven't seen the variant type yet,
[13:43] it's designed for semiructured data and specifically um useful for consuming JSON, right? So if JSON, if you think about a JSON
[13:55] value, it has a bunch of keys and values in it. The schema is not fixed, right? It's just whatever keys and values are in there. Often times with large JSON
[14:08] um sequences, the schema is the happens to be the same from one record to the next, but it doesn't have to be. So a variant is kind of the SQL answer to that. How do we query this thing in a
[14:21] easier way than just a big string? Um yeah, so this is GA. It'll work for CSV and XML scans now and you can use it in more places, right? So
[14:35] um let's get let's think about some examples. How do you actually use this thing? There's this there's a new syntax now the colon syntax to get fields out. So the idea is you can
[14:48] call the from JSON function over a JSON value to give you a variant and extract fields with this colon syntax. Sorry, parse JSON, right? Says here's an example. Select parse JSON of a blob, a
[15:03] string more precisely that gives you a variant. And then when you have the variant value called payload in this case you can extract fields out and then you can extract them as types
[15:15] say this user ID is an integer this is what I want right and this extraction will if the value is actually in the variant it'll give you
[15:27] the value um and this is kind of like a runtime operation because the schema is not fixed so this is a really nice to have as a feature because often times
[15:39] with JSON these are the semantics you want working with strings is confusing right and if you have a lot of fields you can just access them cool and then that's great for the SQL
[15:55] but what about when you want to write this stuff out to storage um how do you do that uh these structures are often nested right so you have the top level object or record and
[16:07] then you can have substructs inside maybe arrays of them. Um, with Spark 4.1 you can now shred this out to parquet when you're writing it. And if uh if
[16:21] you're not familiar with parquet shredding of strrus, that just means um for every top level field in the strruct, it can write it to a separate parket column and that kind of gives you
[16:34] the benefits of compression and the column storage. Um if you have a giant string and you don't do this, it's just a giant binary like buffer and you just like a big value.
[16:47] um you don't get any compression benefits from if there's a bunch of integers in there parket doesn't know but if you can shred it out then if you have a bunch of integers tople fields then it can compress them nicely big
[17:01] speed up on quering it later and storage storage cost and uh this is now nicely integrated with parquet itself apache parket project formally added the variant type
[17:15] it's now supported by spark Spark. So if your parquet data has variants in it, Spark won't understand them. And if you write parquet with Spark,
[17:28] um you can put variance in there and other other consumers can understand them too. Awesome. Okay. Next next cool SQL feature, approximate sketchbased
[17:40] aggregate functions. Um, I personally worked on this a little bit with the other folks in the community. A lot of different people contributed to this actually over time, which is kind of fun. Um, so I can describe what this is
[17:54] and why it's cool. Um so what it is is we can compute aggregates of common types of aggregates that we care about
[18:07] um with approximation built in in exchange for a speed right so the the aggregations that we've added there's a prox top k to find the most frequent values of a input column kl
[18:22] quantiles to compute quantiles like percentiles. This is really useful for latencies of requests, right? Compute the quantiles, give me the 95th percentile, give me the median.
[18:36] Um, and theta sketches are um approximate count distincts. You want to count the number of distinct elements of a set. Yeah. So, these are things we want to do
[18:48] frequently. Um but spark is for big data and in the data gets big. Trying to compute these things precisely means you have to have
[19:02] memory equivalent to the size of the entire data set. And that's prohibitive in many cases. Right? Right? If you're consuming like 10 terabytes of data and you want
[19:15] to find the top three values of 10 terabytes of data, you're going to spill to disk multiple times and then just explode and it won't even work. But usually um the idea is most cases we
[19:28] don't need that much precision and we're willing to exchange a very small amount of error in exchange for just being able to complete this all the time. Bounded memory. um the error the error that we
[19:41] accept is very small. I believe it's less than 1% guaranteed for all of these right so for example the approx K you can compute the sketch so you select the approx K
[19:56] accumulate function of an input column and later you can write these to storage and later you can call a scalar function approx estimate on the sketch buffers
[20:08] and give you the result. Cool. So what are what are these sketch buffers like what are what does this mean? Why are there two functions? Right? So the idea is this produces a binary buffer, right? You can
[20:22] put it in a storage for this. In this case, it's a table aggregate over the whole table. Gives you one result. You can also do this with a grouping keys and it'll give you one buffer for every group. write this
[20:34] to storage and then later you can take the buffer and get the actual result of interest. Why do we want to do that? Because over time you can combine these
[20:47] buffers together and not repeat work, right? So you can have like tons and tons and tons of values every day. you're consuming a terabyte of logs or whatever and you want to find the top prox top five values over these logs
[21:01] over last week, last month, whatever. You don't have to recomputee the top K over the last like year of data every single day, right? You don't you can just do it for the most previous day and
[21:14] then merge it together. Cool. So that's the idea with sketches, right? Um by the same token the quantile sketch as I mentioned you can compute percentiles of things this is what it
[21:26] looks like same idea select the sketch aggregate function which gives you a buffer and then write the buffers to storage and then later you can
[21:39] consume the buffer with a scalar function and get the result you want which is the 99th percentile in this case and then finally data sketch ES are for approximate distinct counting of things.
[21:53] They're very fast. If you've heard of HLL count distinct, it's the same idea. Um, but the additional you get the additional benefits of being able to intersect and union these, right? So,
[22:06] it's kind of more flexible. Compute the intersection of two sketches and union them together and then finally perform the estimation at the end.
[22:22] Cool. So, those those are the things we've added in Spark 4.1. Sneak peek. Probably shouldn't have put this slide in here, but for 4.2, there's another family called Tupal sketches. And this is very powerful. So,
[22:34] a sneak peek here. Basically, what you can do is you can compute approx.
[22:49] And you can do an aggregate over all of the values of the second column within each group. Right? So this is a little bit more complicated, but in this example, we want the approx distinct count of
[23:01] customer ids and for each one, how much did they purchase? You can do all this in a sketch and then later you take the sketch, you can get the estimate,
[23:13] right? which you gives you the approx distinct count and then you can compute the aggregate, right? And that'll tell you everything you want to know, right? So
[23:26] you can and you can combine and merge these later, right? So if you want to if you want to compute the number of customers in a set and sum something over each customer, you can now do that
[23:39] incrementally and very very fast and you don't have to recmp compute it each time. Cool. And then DSV2, we already talked
[23:51] about this. Um it's now mature in Spark 4.1. there's a lot more features table operations connectors push downs right so this makes spark more powerful
[24:08] um they generate metrics so you can kind of observe what the operations are doing um yeah as we saw a couple hours ago merge into a schema evolution now so if the target table isn't exactly the same
[24:21] but has coercible types to the relation you're trying to put in it'll work Now, here's an example of what that looks like, right? Merge with schema evolution. It's just a nice convenience factor when
[24:36] you have floats and you're trying to merge into a table doubles or something like that. You don't have to stress about it or add a bunch of cast functions. Just use schema evolution, make your life better.
[24:50] You can have table constraints, right? So this is something many modern databases have right so here's an example of that create table with some columns and then apply this constraint
[25:02] in this case it's a primary key right you can have other constraints like not null constraints on columns
[25:15] right another awesome DSV2 feature join push down join from two two data sources you can do the join and spark scan from the data sources or you can push it down. If the target data source supports
[25:27] this, you tell it, I want you to scan both tables and join them for me. Spark won't doesn't have to do it anymore. This is often faster um the if the system knows exactly what to do.
[25:41] Cool. And there's also lots of other goodies, right? Lots of richer catalog APIs, list table summaries. Um, you kind of inspect what you're doing, help the
[25:53] connectors express details about themselves if you want. Here's an example of that.
[26:05] Very good. Okay, that's it for SQL language features and uh hand it off to Wenshan to talk about real time structured streaming.
[26:18] Okay. Hello. Hello. Hey. Hi everyone. Uh thanks for Daniel for covering so many uh exciting features. I'll take over from here and cover the next two categories. One is the real time mode in streaming and the other is the spark net and pspark improvements. Let's get
[26:30] start. So oh so I think many of you already heard about the real time mode in different places before right but spark 4.1 is the first open release that ships this feature. So I'd like to cover more details in this session. So uh
[26:42] before we uh dive into it, I'd like to have uh some background introduction about the streaming input spark. So in a spark uh start streaming is the spark streaming API built on the same batch
[26:55] engine and the dev API and uh but also includes the for tolerant system from spark batch engine and also includes all the SQL from spark which is partisted but also a bunch of set operators for
[27:08] streaming and u as you know that how a batch engine has for streaming is that we spark size streaming input into many microbatch. So each web batch is running as a as a batch query. That's how Spark
[27:21] supports uh the streaming query which uh reuse the batch engine but also has a problem about uh the latency. So usually it can only get to at least a second of latency not not possible for second. So
[27:35] in 4.1 we released the mode which delivers a millisecond latency on the same API also on the same batch engine. So uh and also to be honest here so the first list is kind of scoped. So we ship
[27:48] the scala API and also the stating. So we will add the Python API and set for operator in next or next two releases. So someone may ask why we need so much
[28:00] low latency. I'm good with single latency with my ETL workloads or some streaming just streaming workloads. So the the uh low latency can open new use case for spark. So for example if you
[28:12] want to do a fraud detection uh personalization and alerting they need a millisecond latency. So while before you have to deploy a different engine to serving this uh use case while now spark
[28:24] can natively support this kind of latency level and you can use a unified engine for all the use case.
[28:36] So uh and now let's go to some details about the how we make it possible. So why the letter mode can hit uh line second latency? So letter mode change the design choice of the microbatch that can drastically improve the latency. Uh
[28:50] so one is that we don't do mac anymore. We use a longer time based batch to run the uh the streaming query in the return mode. And also we change the scheduleuler. So we don't need to wait
[29:03] for scheduling but we always schedule a task at once. And also we uh use a new in-memory activity based chuffer to replace the special ones. So I will cover all of them one by one later.
[29:16] So uh for the uh batch uh story so for the microbatch you have many many small batch queries running on your cluster. So this brings overhead right because every batch query you have to uh
[29:28] schedule it you have to create result and with mode we uh use a long lived uh larger batch to run and within that batch of course we process the data
[29:40] continuously so you can deliver the output uh as soon as the data arrives so which can help you to hit the second latency and for scheduling as a bash engine span
[29:52] natively wants to launch tasks as soon possible. So when you have a big cluster so you have to wait right. So your slots are occurred by other queries and your batch query can wait for the slots to launch the rest of the tasks. Uh imagine
[30:05] that in your inbash model. So maybe one mbash is h for for a long time because it's he's waiting for its floats while in the u in the retail mode because we have to deliver the inputs continuously.
[30:19] So we have to launch all tasks for that long lived large batch at the same time. So once the batch is launched your task can keep learning continuously. There's no I mean uh race in the scheduling. So
[30:32] you can always deliver the lower latency. The third is the shuffle mode because we have to uh continuously uh consume the input from the streaming source and the uh output to the streaming sync. So
[30:45] there is no way to have a uh blocking boundary in the middle like the shuffer based based shuffer. So we have to use a streaming uh based sorry based stream in memory shuffer that so that we can
[30:58] continuously uh flow the data from the sync source to the sync. So uh this is uh not in 4.1 yet sorry I'm sorry about that. So we will add the support in the next or next version to support stateful operators.
[31:12] So comparing all these three design changes. So mode is it can be very fast uh mining latency but still for tolerance. So within a batch we follow the traditional streaming engine that we we continuously processing the data uh
[31:26] like the old continuous processing mode if you remember that and the difference is the batch boundary. So each bounded uh batch commits of sets to the right headlock is same as the back batch model
[31:38] that we uh always have the checkpoint. So you can you don't have to start over from the various beginning if you if a query fails. So we just reduce the the batch. So once not at least once this is
[31:51] uh the guarantee the symmetric guarantee and this batch duration is point not the latency. So as I mentioned within the batch we continuously deliver the the data from source to sync. So but point
[32:04] only at the the batch ends. So when you uh so when you're processing the lens is very low but if you fail you have you have to restart from the batch. So and a bit more background about the continuous processing mode if you still
[32:17] remember that I think it's in spark 2 something uh we deliver we shipped this continuous processing mode but it's stored at the map owning uh phase because if we want to support set for operators you have to reinvent a new
[32:30] engine in spark that supports the streaming semantic that you do a different forent system and you uh everything so that's too much to contain uh to maintain that's why we switch uh
[32:42] the design to this LA time mode we still inher architecture to do for talent but within the batch we are doing continuous processing
[32:55] so uh a more details about what we shipped in 4.1 so we finished the basic execution framework and we added the scala API which is a new track API and we uh support all the CS operators uh which means no shuffer and we integrate
[33:07] Kafka so if you run Kafka as a source you can as I think sync you can run the pipeline end to end is millcond dependency. And uh to use real time mode, you just change one line in your code. Just change the trigger to the
[33:19] real time trigger with the second of the the the time uh duration of your batch. And in the next next two releases, we will add Python API and the state for operators.
[33:35] Okay. Next category is the pi spark and the sparknet pispark uh about the productity and improvements. So sparknet is becoming a production grade getaway for all your applications anywhere in any language. That's kind of
[33:48] the motivation of sparknet. So we already add many different clients uh in different language for sparknet like python, scala, go rust, I mean third party or first party and in spark 4.1 we also ship a native JDBC driver in u in
[34:03] sparknet. So you can use your bi tools like tabular powerbi to connect to the sparket server. So where before you have to deploy the spark h server which is very lexy it is copied from the hive and
[34:15] and and is I mean have many kind of real high behavior in it while the with new native GDPC driver we can count everything so now we can even do arrow based results so you can get your result
[34:27] with arrow arrow format which is more efficient and so you can now you can use a truly one server for everything uh in in your all your applications so to do that it's just a very simple
[34:39] change. So if you already use GBC, just change the GBC URL to use the GBC SC uh format and then you can call all the favorite of your favorite JBC APIs to run SQL on Sparkness server
[34:57] and another thing is that the spaspar ML the machine learning library is fully support on sparknet and it's G. So now um it's uh the four API the estimators pipelines and tuning they run against remote search sessions and uh uh which
[35:10] will launch tasks and jobs on the spark remote cluster to finish the job. So it also has many hard work to make it really operation ready. So for example uh we have a model cache that overloads
[35:22] to disk to bound the driver memory and also some security fixes to make sure that we uh we don't uh allow people to run some u dangerous code on the spark driver side and also we jo
[35:35] models loading. So everything is uh making the spark paspark mirror library much more reliable on sparknet
[35:47] and also we have done many improvements to the spark itself to make sure it can support the production scale workloads. So for example um uh we now compress the big plans with STD. So you have small memory footprint when you deal with very
[36:00] big queries and also the large will be streamed back to the clients with with alo chunks. So which will reduce the risk of OM and also for large local data set we we uploaded with a very efficient
[36:14] way that is also uh alo chunked and also um if your client has some problem get down. So now we support an important link cushioning and C cloning to support the Londoning clients.
[36:32] So now let's move to the pispark side which already we have covered some of it in the this morning's session. So I will go on quickly. So uh the thing that pispark is becoming um is trying to be
[36:44] hello end to end. So in 4.1 we added this ludf support. So now you can write your UDF with consuming the L array and input L array. So which can remove the pendas conversion in the execution path
[36:57] to make it faster and also same thing happens for UDTF. So just a different annotation but same for the UDTF and also we support the fil.
[37:09] So now you can u reduce the IO between the spark and your your data uh system by applying the filters locally on your data
[37:21] and also we improved the inflow of the pi spark u uh runner. So um we use the Unix domain sockets instead of TP TCP lock loop back for the communication
[37:33] between the JVM and the Python process and we also use a faster ADO UDF serializer and ADO work measurement that cut overhead. So your Python UDF can run faster. I mean whatever Python UDF the
[37:46] arrow the pandas the classical one can run faster on now because of this improvements. And for debugging I think we already recall many of this mon session. So I just we just go through it. So now you
[37:59] can use the standard log APIs in your UDF and you can see logs with UDTF function that you can print it on your uh on your driver. So here example. So you just use the log
[38:11] API and then you just enable that config to turn it on by default because this is still in the beta phase and then you just uh invoke this UDTF and then you can show it and you can see all your logs on the Java screen.
[38:25] So to do a quick recap so now sparknet is a production grade getway to Spark. So you can launch CQ while the new JDBC uh driver uh you can run bigger and uh and the bigger world in a more way and
[38:40] we also create J pass by MR. So now I think we can use sparkn net for almost all the workloads you have have now and running it more efficiently and more reliably and passmark is now alo end to
[38:54] end. So we have UDF that can consume and output AO data and also the result and also the input I mention is a local data set. So they are all Alo ent. So everything is kind of Alo in pass now.
[39:06] So the data flows with the current format everywhere to make it a faster book path. And you can also download the stuff open now and try it out like you can use your BI tool to uh to point out the new GBC Java and you can uh rewrite
[39:20] your UDF with the AO format uh which will require the Python 3.5 and the parallel 15 plus and this is end of this session and I
[39:32] hope encourage you to join committee as well you can report issues in the Apach spark which we just opened the GitHub issues uh feature in the Apache lipo and you can you can also open podcast to the code and a bug fix. Uh thanks

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.