Skip to main content

What’s New in Apache Spark™ 4.0?

Summary

  • Spark 4.0 overview: This release resolves 5,000+ tickets with 400 contributors, introducing major enhancements across SQL, Python, streaming and infrastructure.
  • SQL modernization: Spark 4.0 makes ANSI SQL default, adds scripting, SQL pipe syntax, SQL UDFs and session variables for more flexible data processing.
  • Streaming and infrastructure: Discover new state management features, Spark Connect improvements for 100% API parity, VARIANT data type support and better error messages for modern AI pipelines.

What’s New in Apache Spark™ 4.0?

Watch: What’s New in Apache Spark™ 4.0?
Talk by Daniel Tenedorio, Sr. Staff Software Engineer, Databricks ; Wenchen Fan, Senior Staff Software Engineer, Databricks
Join this video for a concise tour of Apache Spark™ 4.0’s most notable enhancements: SQL features: ANSI by default, scripting, SQL pipe syntax, SQL UDF, session variable, view schema evolution, etc. Data type: VARIANT type, string collation Python features: Python data source, plotting API, etc. Streaming improvements: State store data source, state store checkpoint v2, arbitrary state v2, etc. Spark Connect improvements: More API coverage, thin client, unified Scala interface, etc. Infrastructure: Better error message, structured logging, new Java/Scala version support, etc. Whether you’re a seasoned Spark user or new to the ecosystem, this talk will prepare you to leverage Spark 4.0’s latest innovations for modern data and AI pipelines.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are the major features in Spark 4.0?

Key features include ANSI SQL by default, SQL scripting and pipe syntax, Python plotting API, VARIANT data type, improved streaming state management, and Spark Connect API parity for better application compatibility.

What is the VARIANT data type?

VARIANT is a new Spark 4.0 data type that flexibly handles unstructured JSON and semi-structured data, improving support for modern data formats common in data and AI workloads.

What's new in Apache Spark 4.x?

Apache Spark 4.0 introduces significant performance improvements including GPU support for accelerated computation, improved Spark Connect for client-server architecture, and enhanced streaming capabilities. This video highlights how Spark 4.0 enables enterprises to leverage modern hardware like GPUs for faster data processing and machine learning workloads. The release also emphasizes better developer experience through improved APIs and compatibility with popular frameworks.

What is the Spark Connect client-server architecture?

The Spark Connect client-server architecture in Spark 4.0 enables lightweight clients to submit work to remote Spark clusters without embedding the entire Spark runtime. This video explains how the architecture simplifies deployment, reduces client dependencies, and enables polyglot programming across languages while maintaining Spark's distributed processing capabilities. Spark Connect is now production-ready in version 4.0 and represents the future direction for Spark integration.

How do I package an app that uses a Spark Connect client?

Packaging an app that uses a Spark Connect client requires only the lightweight Spark Connect client library rather than the full Spark distribution, significantly reducing bundle size and dependencies. This video demonstrates that developers can build portable applications that connect to remote Spark clusters, enabling easier deployment and version management. The approach works across Python, Java, Scala, and R, providing flexibility for polyglot development teams.

Full transcript

0:00 Introduction to Spark 4.0

Uh hi everyone, welcome to our session. I'm Win and We are from the Spark team in De Bricks. So I'm also listen 4. 0 and today we are very excited to be here to uh tell you about the new updates in the SPA 4. 0 release. Okay, I will uh start and will jump in later. Okay, I think most of you have seen this uh slide in the keynotes. So, in SPA 4. 0, it's a huge release. So, it resolves over 5,000 tickets uh with the help of around 400 contributors.
So, uh there are improvements in many areas like the making SQL better and adding more power to the Python APIs and new streaming APIs and so on. In this session uh we will divide uh we are dive into the four key areas of innovation in Apache Spark 4. 0. So we are first uh explore the updates to the infra level of spark especially the sparkn net component and following this uh we will discuss the new features for ETL data science and data warehouse users.
So Spark is a general and unified engine and we hope these new features are help are useful to you and can make Spark a better tool for your all your use cases about data processing and um I will be taking you through the first two parts and will jump in and cover the rest. So first up uh let's talk about Sparket. So Sparknet is the most important architecture shift for Spark in the recent years. Let's quickly do a recup of sparkn net.

1:38 Spark connect and infra

So it's all about making spark more fatible and stable. So essentially it decor the heavy spark driver from your application code and putting it into a separate its own process. So this makes your application uh smaller and also uh the your application will communicate with the spark driver using a server client protocol. So one of the really important things we focus on in this release is to ensure the API community between the JavaScript APIs in spark connect and um um we are we have created short interfaces for the major APIs like data flings spar sessions colon and whatever stuff uh in uh for the uh classes in spa and class and connect.
So um by doing this when you compare application against spark now it compares against these short interfaces. So this will ensure that we have 100% API parity between the connect and the classic. Uh unfortunately Spark has a tons of features right so uh and while we have made a huge progress uh not every single API is integ um but we have put in a lot of efforts to uh uh reaching the feature parity um so we have uh implement many uh basic functions and features including spar mr in in the sparkn net and uh in spark 4.
0 No, I think we have we can claim that we have like fluish like 99% of the feature in inspate and uh um we are still uh pushing for it and we hope we can uh uh hit 100% priority very soon and you can also see our tracking work on these two tickets. So, so we do that uh getting started with a big active change is very uh painful. It's a very big uh change that you have to uh redeploy your SP applications with a different architecture.
So in spark. 0 we make it easier by adding this uh new flag spa. So this will let you uh turn on sparknet without deploying a server beforehand. So when you uh run your application with this uh config uh set to connect so spark will launch a dedicated spark connect server for your application and they will connect with a secure local connection. So it won't get you all the benefit of spark net because you now you have a dedicated server for your client.
There is no shining uh clients uh for the server. So but it's a good start. So it can help you to prepare your application to be compatible with sparknet which will be more robust more support of languages and uh and uh more stable. So uh to make the onboarding process even more simpler so 4. 0 also brings two more pispark packages in the pi um so first there is a pispark uh connect package. So this package is just the full pass spark with spark net turn on by default which means the API mode I just mentioned will be set to connect in this package and it also includes all the necessary dependencies that required by the spark like the gc like the arrow dependency they are all in the this package so you can use spet out of the box very simple and if you already have set up a spark driver server ahead like by your admin or someone else in your company so you can also use spa pispar dash client package.
It's a thin client. It's only 1. 5 megabytes. So, it will be a much lightweight solution for your application. Uh last but not least, the multi- language support. So, we actually mentioned last year that this new server CL architecture make it much easier to implement new language support for Spark and we are happy to tell you that we have delivered on this. So, in Spark 4. 0, Oh, so uh first the building support we have support Python, Java and Scala Autobox in Spark and also the Spark community maintains two official sub projects for the Go and Swift clients and also in third party there are clients for Rust donate and R and also there's a good news that the author of the RO last client is contributing this to a spark.
So it also becomes a first party very soon. Uh so the next feature I want to talk about is the start logging. So uh so if you have addressed any uh logs in spark before you know that it's very painful because it's just text lock in lo 4j right so they unstructured and uh it's very hard to find some meaningful data from the text logs starting with spar 4. 0 Oh, so logs can be in JSON format. So this feature makes it easier to consume the logs and fet the information you want and uh uh all you need to do is to turn on this flag uh spar. log ststructures lock enabled and then you just uh run spark applications and can lock files which are just in JSON format.
So with JSON format spark can read it natively, right? So now you can use Spark to read your JSON logs and run queries in either define or SQL or whatever language you want uh to see the insights from the logs and debug your application. You can also check out the four response of SPON to find out other any other changes that we made to the infra level. And let me also have something here. So first that in the 4. We dropped the Java 8 support which is kind of support in the by the official and we also support uh Java 17 by default and we also dropped the Scala 2.
12 support and make the 2. 13 the default and also we provided the experiment support for Java 21. So you can now la and uh and uh deploy your uh uh sparking with java 21 and also we added a new native spa kubernetes operator. So which makes it easier to run your spark caching on kubernetes and also we dropped the python 3. 8 support which is also end of life and there are many other other features. Uh now let's move to the next section and uh let's talk about the new features for ETR users.

8:21 ETL and data sources

The first thing I want I'm happy to announce is new vant data type. So for ETL it's common to that you have to deal with semi structure data like uh uh like JSONs. So uh which they don't have a static schema. So maybe every low has different schema and you don't know how to turn it into a table. Right? So a very common solution people have with spark is to turn the data into JSON uh format and save it as a string current in some table and then when you want to query it just read a string and use the spark functions like pass from JSON and whatever get field to pass the JSON on the fly and get the fields you want.
So this is very inefficient because as you can imagine you have to pass a bunch of JSON string only to get a tiny bit of data you want. So in SP 4. 0 Oh, we saw this problem with the new varant datab. So, it's as visible as JSON that you can have dynamic schema for every record and it's also open standard in package, but it's much faster than JSON. So, let's see an example. Um, so let's see that the variant type uh was formed from some JSON string that you pass using the pass JSON uh function in Spark CQ, but it's a standard over.
So, you can also use other APIs from the third party to build your own valant values. So after you have the valid value in spark you can now use the valant get uh function to get what the field you want out from the the valant data. So this this function uh request you give a JSON path syntax which is also open standard in the uh JSON world. So to specify which field you want to read and then you specify the type you want the field is because we don't know what the type is is a dynamic.
So in this in this example we want to get the field user agent and it's a string type. So then this function will return a string value to to spark cq and now you can use all the function you want to deal with the value uh to continue your application. So our back shows that there is eight times speed up for a simple close using the variant compared to using JSON a string to pass the string and get the field. Okay. Uh next uh let's talk about the connectors.
So when it comes to ETL the very first step is to loading the data right so and that's where the spark shines because spark has a rich set of containers out of the world right there are so many third party libraries and plugins to provide you a solution to access the uh external data. So in spark 4. 0 we added the built-in support uh of XML. So now you can use spark to read XML files out of box. So there's no other plug-in needed just use XML and put the path and the data is here.
Uh in addition so this uh XML DOS uh was computed from a third party project and but it's not just a single port. So we write almost every line of code in this uh third party project to make sure that it has the feature parity with the building files in spark like the JSON and CSV. So they have support partition colors and uh some options uh they have feature quality and with the same performance. We also added the FL XML XML functions to for you to just deal with some strings in XML format.
Another exciting feature that is the Python data API. So we designed this to be easy to implement. Um, in fact, you can just use AI to generate the Python datas for you, which is much simpler than you're writing the Scala code or Java code to implement a traditional data v1 or V2 uh impression. And thanks to Pi, it's also very simple to uh publish and share your datas with others. Just uh publish your data to pipe and tell us the little name and people can just download it with P install and use it in their applications.
Um so essenti is just like a singular version of data v2 in spark and uh which means that we hope to have more people to contribute to this data existent and they can use their their favorite python language to implement and also make the loading step more uh much easier in spark by these third party data and will also cover more details of this in the later so stay tuned. So uh for powerful and competed resources so data v2 is still the API to use and we also have many improvements in this v2 in this 4.
0 release. So uh that's Samsung. So we we support the clusting uh mechanism in the uh start desk v2 which is uh already data lake and we also want to have this in as well and we also add the API for procedure. So which can help you to invoke some uh like procedure code to do some uh work and we also improve the uh some performance feature like storage partition joining and many others. Okay. Now the last topic I want to dive into is the streaming.
So in the real world the data keeps growing right? So the engine must keep up. So we need to process the data incrementally and in other words we need the ability to do streaming queries. So when we talk about streaming queries one of the most important thing is the streaming state. So because when the things go wrong in your query you have to restart and we don't want to repro all the data right which can be like years data from the old time so it's too expensive so we need to have some state to remember where were we when we processing the streaming data incrementally so um however if there's something wrong with your streaming state like some corruption or some like bugs in spark so it's very uh hard to debug And it will also like make it also will stop your application from restart from some point.

13:36 Streaming state management

So uh in 4. 0 we provided two uh data to allow you to look into this streaming state and debug for your uh application. So now you can use state dashstone uh sorry metadata data source to look at the high level of your streaming state. So which like like the operator ID, number of partitions and so on. And there's another one called uh uh state stone. So which provides more low-level information like the partition ID and the low JSON string of your state.
Another powerful feature is the new transform with state API in the uh ST streaming. So this unblocks advanced state management and event processing and timers and schema evolution for user defined streaming processing logic. So for more details you can also read this de blog post that we published a while ago. Okay. Now I hand over to my colleague for the class. the Spark 4. 0 talk. So, thanks for coming. Cool. So, next section we're going to cover data science features and then after that we'll cover data warehouse features.

16:00 Data science features

It's kind of like the way we decided to split the new features apart. There's a lot of stuff in Spark 4. There's a lot of new stuff. So, I'll try to go through this and give each new feature the justice it deserves. There's just a lot of stuff. So, try to get through it all. Okay. So, yeah, start off PiSpark native plotting. Um, this is this is kind of fun. Um, so now you can take any pispark data frame and just callplot on it and it'll use plot the plotly library to generate charts for you.
So this is just really integrated and built in. So this is nice for iterating if you're trying to use pispark data frames and you want to see what the data is, you can just iterate and uh crank out plots with it. There's a lot of stuff you can do with that. For example, pie charts. You just see get a quick visualization of the disposition of the values in your data frame, right? So, it uses plotly um and you call different methods on the data frame and it'll do what you tell it to do, right?
So, um here's kind of like a list of the different plot types that it supports. There's a lot. Um, my personal favorites are the pie chart and probably histograms, but that's just me, one person. Um, there's a lot more of them. So, um, if you've used plotly, these will be familiar to you. Otherwise, uh, it's really easy to kind of get into it, right? So, here's an example of a scatter plot. You can call df. plot. catter and then give a bunch of parameters to it.
Um, so on the upper left of this slide, there's kind of like the df. show rendering of the data and then to kind of compare it with what a scatter plot might look like. So this is one example of things you can do. Um, cool. And then this is a pie chart. Um, this is probably my favorite rendering because it's just super easy for people to understand if you show them a pie chart breakdown of something. you have all the categories in a list and then you have the pie chart.
Um, so it's just like a lot easier for some people to consume than a table. Um, and it's really easy to generate one as you can see in this example. Cool. All right, next big feature, Python UDTFS. Um, I'm personally excited about this one. I personally helped contribute on this one. Um, this is pretty powerful. Um yeah, so the idea is that you can define a UDF in Python that you can call in the from clause of your SQL query or um invoke as a data frame to get a relation, right?
So it generates a table as an output instead of a single value. Um yeah, so this is kind of an example of a brief example of how to define one, right? like you define a class say what the return types of the schema columns are and then make this eval method that gives you the data and it basically yields rows right so this is kind of like zoomed in like picture of what a basic implementation looks like so it's easy to get started so this is an example of what it might look like to call one right so you can select from the UDTF generating a Here we pass two scalar arguments.
Use SQL named arguments for this just because it's kind of a nice feature that is now available in Spark and Spark SQL. And it's kind of similar to the Python like passing named arguments feature. And then here's how you might um invoke the UDTF of the data frame API. So this is kind of like the intro. So okay so these are these are kind of basic use cases and it's like okay this is kind of cool but why do I care um so you can there's a lot of advanced features um if you look dig into the API right so for example um you can actually define instead of the schema of the UDTF being a static table with static names and types of columns you can programmatically define it right so you define this analyze method um that takes the arguments values of the function call, right?
So you can inspect the actual literals or values or whatever and then use them to make decisions and then generate a schema and metadata for that particular call. Right? So for this example, you could take a file name, look at the string, open it up, see what's inside, and then depending on what's inside, you give it a table that comes out. So this very reusable So for example, you might call it like this. Give it a file name, open it up.
Oh, there's two values inside. So that's one thing you can do, right? So um what else can you do? Um so the these user defined table functions can take scalar arguments, but they can also take tables as input or one table as input, right? So and to do that you can pass the table as an argument with this table keyword in SQL table parenthesis t or name of the table um or you can pass a table subquery and that will give the input rows of the table to the Python TVF as an iterator. you can iterate through them.
And when you do that in the function call, you can you can specify a partitioning of the rows and an ordering within each partition. Which means that for every unique combination of values of these partitioning expressions, the function will get called exactly once. Right? So if you see this example partition by ID divided by 10 for every unique value of ID divided by 10 UDF UDTF will get called once and get all of the rows of that partition and be sorted by ID.
So this is useful to kind of split up the work right and then at the end um you can define this terminate method to do something at the end right after you've consumed all these rows of the input you can do finalization steps at the end which can also yield additional rows right so for this example you init the class when it gets uh when it gets created and this eval method gets called once for every row and you can inspect the row.
It's a pi spark row um object which already exists and then you can look at the values um update internal state possibly um yield rows or not um and then in terminate method you could do any finalization steps. Cool. So yeah, they can take scalar arguments or tables as input define the output schema dynamically. Um for each of these methods, you can also make them accept variable um arguments, right? So you can say star args or star kW args like Python will let you do and you just define it like this and then in um function will accept variable arguments and you can just inspect them, right?
So if you say it accepts star kw args whatever you call the function with it'll work and then the methods can look at what the arguments are right the names of them and then the values right so you can make a UDTF like this just pass in some values and then do what you need to do cool and then finally within this custom analysis is polymorphic analysis where you define the schema of the function call for the provided arguments.
You can also put other metadata in there, right? So you can subclass this thing and in this expensive initialization that you do, right? opening a file or connecting to some API, making REST API calls somewhere, doing whatever you need to do, you can put all that state into a subclass of this analyze result, and that'll get passed around to all of your um eval methods or your sub your instantiations of the of the Python class.
So, you don't have to do that again, right? You can just like save the work. Um, so if you want to do expensive stuff once, you can just do it then and then pass the results. This is kind of what it looks like, right? Your analyze method can take arguments, look at them, say, okay, I certain things to do, figure out the schema, and then return a subclass with a schema in it for that call and then some other data metadata in there that all that will get passed into the eval method.
Cool. That's Python user defined table functions. Give it a shot. Okay. Next thing, Python data sources. As Wshen mentioned, this is new. Um, why is it awesome? Because people like Python, not just me, other people. Um, it's really easy to use um to get started with. Even AI bots can use it, right? So, this is kind of like the base class. And you can um use these things to define read access to data. You can also write to them.
So you can do DML operations into the into the data source. Right? So the idea is you define one with this class, register it in the current Spark session, which is kind of very similar to registering like a PISpark UDF, right? and then go have at it with a spark read and you can just say do format equals my data source and then load the data out or you can do df. right into it as well and all of the data that you write will get propagated into the API so you can do what you need to do.
This is pretty easy to use. You can ask chat GBT or your favorite AI to make me a parks the spark pispark data source doing X especially especially works well with APIs. You say here's a rest API make me a data source based on this and it generally will work. Um so that's pretty cool. Okay, next next feature Pispark UDF profiling. So this is kind of in response to people using Pispark UDFs over, you know, several years.
Um, right. So, and a big feedback from the community was what's going on with my Python? I'm doing expensive stuff and now it's AI time, like AI era, and they're doing even more expensive stuff. What's my Python doing? Um, I want to figure it out. So, this is kind of one answer to that, right? So you can kind of track the behavior in both performance and memory usage, right? So to enable it, you set these configs and then you can say spark. profile um collect performance or memory and then dump it to somewhere.
So this is kind of like an example of what it looks like, right? you define your pispark udf do stuff and then set the um profiler config and when you show it um you get an output that might look like this right so here are all your different um elements in the performance profile and you can kind of look at what happened and how long it took so that's kind of one useful feature you can do to debug where the time is Right.
And by the same token, you can do the same thing with memory use. So you just use the Pispark UDF profiler with the memory profiler and then um print it out and then you'll kind of see where the memory went and broken down by components, right? So um Python can eat up memory even just importing libraries to find like where did my several hundred megabytes go? It's gone. Python ate it up. So now you can ask where did it go.
Cool. Okay. It's kind of next thing for debugging debugability, right? So you can have a um error like error conditions, error contexts, right? So if your program fails or something happens, um a big feedback from folks using Spark was that the error messages got a little confusing. um there's kind of like not a lot of consistency among different types of exceptions and some of them were pretty vague and stuff. So, you know, we kind of went through them all um and like categorized them up.
So, yeah. So, now they have these nice categories that are all documented. Um you can look look at this list kind of see um you can Google around if you get one of these and it's like the key the condition serves as a nice key people can uh kind of we can look at the docs or like discuss it right so now um as a result when you run into a problem you can get kind of like a nice code and then the description I think uh the code can be a link you can click on that it'll take you somewhere useful.
Right? So this is kind of like a nice this is like a nice illustration of what division by zero looks like now versus what it used to look like. Right? used to get this like pi spark internal esoterica before and the actual important stuff was like in there somewhere deep and now it's a kind of categorized by importance right so the code concise description of the error and then we have the SQL state which is like canonical way to describe the category of error right and then last Python thing there's a pandas 2 is now part of spark right so there's lots of new features Um, lots of new stuff.
Pandas is awesome. Um, it's a really nice API. So, you can use it. You can kind of if you have small data, you can just use pandas directly or, you know, you can kind of go distributed if you, you know, need more power and regular pandas will kind of not work anymore. Cool. So, that's the data science part. Now, we can go into the data warehouse part. Um, and there's there's a lot of SQL features in here. Um, I'll have to kind of speed through them a little bit faster than I might have otherwise wanted to because they're all so awesome, but we only have limited time.

30:55 Data warehouse and SQL

Yeah. So, SQL scripting is now in Apache Spark, right? This is a big deal, right? So this means you can do procedural return complete logic in SQL in a program without dropping into Python or Scola. Right. Right. So you can do if conditionals, you can loop, you can do loops, you can do exception handling and throw exceptions. Um you can even call execute immediate generate a string of SQL and then call execute immediate and run it kind of like Python's eval.
So there's a lot of powerful stuff you can do, right? So it starts to look like this. And this is SQL PSM. So this is in the standard, right? So this is not just something we made up arbitrarily. This is like very standards compliant, right? So you can get you can get pretty deep with this stuff, right? So you can declare variables. Um you can do loops. Um then you can assemble up strings um like this alter query example and then execute it.
Call execute immediate to run them. Right. So you can do all this stuff if you prefer SQL or your organization has analysts that are really deep into SQL and that's what they want to do and they do not want to use Python. Now there's an option for that. Right. Cool. So SQL scripting is a big deal. Okay. Next thing SQL UDFs. Um they've been in data bricks for a while. Um so now they've been properly kind of ported and integrated into Apache Spark.
So this means you can take your logic in SQL and package it up into UDFs that take arguments and return values. Um these can be scalers or tables and they support named arguments for function calls, right? So um there's um kind of a lot of consistency with this feature and other kind of SQL engines. A lot of SQL engines have this. It works a pretty similar way, right? So, here's an example of what you can do. You can create the function, apply defaults on the arguments, you can apply comments on them.
Um, you can have this return, you can have an optional returns clause that says the return type or you can skip it and then put the content of the function and when you call this thing, um, Catalyst will just inline the body of the function at the call site. So it's a very very fast. It doesn't like invoke any kind of serialization or um virtual machines or anything. It'll just run the SQL you tell it to, right? And you can do this with table um tables as well.
So it can return a table. This is the SQL TVF here that you can take a basically make a view and parameterize it. It's like take a view make it take arguments. Right. Right. So you can kind of use it like this. Cool. And uh when you call these things, you can use named arguments, right? So this kind of makes things easy kind of easy to use um when you have complicated arguments you're passing. Um right. So that's kind of nice.
And they also accept default values. So you can add new arguments to your existing functions and have make them defaults. So they don't have to be specified. And you can fast named arguments when calling them. So you can add new optional arguments that you can just put named arguments at the end. So they're very extensible. Cool. Okay. All right. So I'll go through pipe syntax briefly because we did cover it on Monday.
Um but I personally did this one. Uh it's based on a paper from Google last year at VLDB. Right? So the idea is this is regular SQL. The idea is when the logic in SQL gets more complicated than the regular clauses of a query can support, you have to use subqueries to continue. Right? So this is TPCH query 13. Um and with pipe SQL uh SQL pipe syntax you can basically pipe the clauses of SQL in any order. So this is like data frames, right?
It's very similar, right? So you can say from customer from table then do whatever you want in any order you can say pipe where predicate you can say pipe join pipe aggregate um and then each one's kind of like independent orthogonal operation you can do right so um if this is kind of like what regular SQL looks like what the clauses ordering is this is regular SQL this is pipe syntax and you can just put them in the order you it will do what you tell it to do in the order you tell it to do it kind of like dataf frames and the optimizer can still do its job right this is still declarative right so you're not suppressing anything that catalyst wants to do or can do today with regard to predicate push down call and pruning anything else right so for some examples um if you want to write sub queries this is another more complicated TPCH query Um, with pipe syntax, right, you have to add a nesting level every time you do a table subquery.
Um, with pipe syntax, you don't have to do that. You can just keep adding stuff, right? Add pipe operators to it and it's just a sequence and your brain can abstract the complexity because you don't have to understand every single one. You can say, okay, what are what am I looking at now? Oh, there's this pipe where sales price is greater than 100. that just takes whatever table came in, relation came in and filters it.
So yeah, a nice property of this is you can you can take a pipe syntax query and every prefix of operators is itself a valid query. That's a nice property, right? So to debug this, kind of see what's going on, you can highlight it in whatever like editor like in the data brick SQL editor, highlight it, run selected, then highlight some more, run that, and then just keep going and you can see it transform, right? You can starting, you can start with just from t and that'll give you the whole table.
Then just keep adding more steps and then see what happens to it. So it has this nice property, right? And there's this backwards compatibility property where you can just jump into it. Um, you can add pipe syntax to the middle of an existing SQL query or just add new pipes to the end of an existing SQL query because any valid query can be the input. Cool. Okay. Session variables. Um, right. You can now define these and assign values to them.
Um, you can use them any number of times. Um, it's limited to the session. So when the session ends, the information is gone, right? So this is kind of what it looks like. Declare and set variables. Um you can set multiple of them at a time, use them as needed, and when the session is done, they're just automatically kind of cleaned up. All right. So this is kind of another example. Cool. Okay. Any mode is now on by default.
Um, this has taken a several Spark versions to get here, but it's now the default. So, this will keep your data consistent and correct, right? So, for example, if you do a division by zero without any mode, you get null and they may happen in the middle of your three-hour job and you just don't know and that's bad because the data is incorrect. Whereas with Spark um 4 with ANI mode enabled, it will throw an error as per the SQL standard.
And in Spark 4. 0, the error message error messages are cleaned up. So it's actually easy to read and it'll kind of give you the source location and what happened. Cool. There's a now string collations in SQL SQL. So you can assign collations to columns or operations like group by and order by. Right? So this is the default collation. If you sort something, it'll kind of give you this, which might be confusing. If you give it unit code, it'll sort differently.
Um, you can tell it to be case insensitive, which is nice if, for example, if you're migrating from Terara data or other engines that are case insensitive. You can assign a case insensitive collation and kind of keep it consistent, right? Here's another example of that. Um, so you can kind of have accent accented characters and letters appear kind of next to the regular letters. And finally, execute immediate. As I mentioned, you can assemble these strings any way you want.
Call execute immediate and it'll run the string you set as SQL, right? So here's the kind of unified example of that. Cool. Yeah. So that's the Spark 4. 0 talk. Um, yeah. Thanks, uh, thanks for listening to us. Yeah, I think uh we have to give up the room but um for the next next session, but if you'd like to ask us questions um Winon and I can kind of speak with you guys in the hallway if you prefer. So yeah, thanks again.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.