The Upcoming Apache Spark™ 4.2: The Next Chapter in Unified Analytics

Summary

Apache Spark 4.2 introduces five benefit areas: metric views for a first-class semantic layer, Spark Connect for multi-language reach, AI-native SQL with vector functions and geospatial types, Auto CDC in Spark Declarative Pipelines, and operational improvements including a new performance UI and dark mode.
Metric views solve the dashboard drift problem by defining business measures such as revenue per user once in a semantic layer within Spark itself, ensuring consistent results across all tools, notebooks, and dashboards without per-tool redefinition.
The release reflects contributions from over 260 community members across more than 1,600 resolved issues, with a language-agnostic UDF protocol and Arrow-backed performance improvements making Spark more interactive and AI-native.

The Upcoming Apache Spark™ 4.2: The Next Chapter in Unified Analytics

AI workloads demand real-time feature pipelines, multimodal data support, and seamless AI-assisted development. Apache Spark™ 4.2 brings these capabilities into the engine, unifying batch and streaming while optimizing them at the planner level. Metric views provide a first-class semantic layer to define measures once and ensure consistent results. Catalog-managed flows make incremental and streaming queries lifecycle-aware catalog objects. A language-agnostic UDF protocol enables portable user logic across Python and emerging Spark Connect languages. Enhanced LLM discoverability and local performance improvements, including Arrow-backed caching and shuffle-free execution, make development faster and more interactive. Apache Spark™ 4.2 evolves into a governed, incremental, and AI-native platform built for the demands of the AI age.
Talk By: DB Tsai, Senior Engineering Manager, Databricks ; Xiao Li, Engineering Director, Databricks ;

Chapters

FAQs

What are metric views in Apache Spark 4.2?

Metric views are a first-class semantic layer in Spark 4.2 that let teams define business measures — such as ratios like revenue per user — once and query them consistently across any tool, notebook, or dashboard. They solve the dashboard drift problem where the same metric is defined differently in BI tools, dbt, and notebooks, causing reports to disagree.

What AI-native SQL capabilities are new in Apache Spark 4.2?

Spark 4.2 adds vector distance functions, nearest-by and top-K similarity queries, approximate distinct counts with Theta sketches, and native geometry and geography SQL types for geospatial workloads. These bring analytics and AI capabilities directly into the engine without requiring separate tools or external libraries.

How does Auto CDC work in Apache Spark 4.2 Declarative Pipelines?

Auto CDC in Spark Declarative Pipelines handles out-of-order CDC events for SCD Type 1 patterns declaratively, so developers state their intent rather than writing explicit merge logic. Spark 4.2 also extends the unified DSV2 API to make CDC a first-class engine feature across Data Source V2 connectors.

What is Project Feder mentioned in the Spark 4.2 roadmap?

Project Feder is an initiative aimed at making small-data workloads faster in Apache Spark, targeting the performance overhead when Spark's distributed execution model is applied to datasets that would fit on a single machine. It is listed as a roadmap item for development beyond the Spark 4.2 release.

Full transcript

[00:09] So good afternoon. So I'm Shiao. Um I work on the Spark in DEX. Um the DB uh same team. Um both of us work for the Spark over have worked for Spark over 10 years. Um today we'll show what is new
[00:23] in Spark 4.2. I take the first half and the DB will take the second. So this the um the focus of the spark is a native an
[00:37] uh for data and AI spark is moving more of the stack into the engine. Let's start. So one quick note spark 4.2
[00:49] uh is built by the whole spark community uh more than 260 contributors from community. So everything we show is open source spark.2
[01:03] in one line more semantics AI Python CDC and operations powers all in the same engine. So the number over 1,6 Jiraa 6
[01:18] 100 Jiraas and from over 260 contributors. So one go ties it together make analytics and AI more native to spark.
[01:32] So we build a talk around the five benefit. One define choose once matches view. Two, reach spar from everywhere. Connect
[01:45] pispar arrow python data source. Three native analytics in SQL vectors sketches geospatcial.
[01:57] Four move changing data simply auto CDC changes DSV2 streaming. Five operator and evolve predictably the
[02:11] UI the better UI reliability faster release and looking ahead we started the project fther to made the small data faster and
[02:25] the other ongoing speed will be presented too. So I covered the first uh three and DB will cast rest. So this is a whole release on one
[02:38] slides. Six groups about 24 features. We are not cover them all. Uh just see how much is here. Keep it as a map.
[02:50] Um let's start benefit one define choose once. Let's start with a problem.
[03:02] So takeway today's takeaway is a move your top dashboard mattress into a mattress view. So the problem in pen wars revenue per user is a radio
[03:17] ratio cannot be summed directly. You cannot average them across the groups and get the right answer. So when every team writes a formula by hand so the
[03:29] dashboard drift apart same magics different numbers. Let me show how it works. Here it is in one table two countries revenue and
[03:43] active users adds up fine but revenue per user is a ratio. So if you average the two rows you get a 15. That is
[03:55] wrong. So the right answer is the total revenue over the total users. So that is 13.33. So where should this matches life live?
[04:08] So it is right every time. So that is a semantics layer problem. Every company already has a semantic layer but it is scattered.
[04:20] Revenue active users and the churn are defined again and again in BI tools in DBT in notebooks. The cost is real.
[04:33] The dashboard disagree. No one owns a definition and it is about correctness. So retails and the account distinct break when you roll up. Client tours sit
[04:45] outside the engine. So they cannot enforce it. AI needs this too. An agent must answer the same way every time. A govern mattress view gives the BI and B AI one
[04:59] source of truth. So 4.2 P mattress view that is a speed uh spar 5411
[05:11] 9 uh you define the mattress once then you query it by any breakdown. So there are two parts dimensions are the way to slice measures of new column type that
[05:23] aggregate without a fixed group by. So this stop drift retail and distinct counts stay correct for every grouping and the measures can build on other
[05:36] measures. It lives in the engine and the catalog. So the optimizer keeps it correct and the permission are the same everywhere. One source of truth for SQL, BI and AI.
[05:53] So three ideas nothing text. A dimension is how people slice a matrix. A measure is what business cares about. Here it is revenue per user a ratio. The query
[06:08] asked by any breakdown. The point is not syntax. The matrix is now an object spar understand so it is a crack every time for everyone
[06:20] now so please try it so in 4.2 to this is the latest uh bit features. Now how to spark how to connect spark.
[06:33] So we can reach spark from everywhere. First connect spark connect. So take away call from your own service language.
[06:45] So the idea is simple. Your client builds a plan it open a gRPC stream. The server resolve the plan,
[06:57] optimize it, run it and then send the arrow batch back. The client is needs, it doesn't need a full spar runtime and no JVM next to your app. So
[07:11] the client and the cluster can change on their own. You can upgrade your server easily without affecting your client. Before connect, using spark from another
[07:23] runtime means wrapping it or embedding it. With connect, spark is a service API. Your app does not need to be a spark app.
[07:40] The point is a use case, not a language list. So the Rust service submit a walk a Python noble connects to a remote server a remote cluster a small app use a spark
[07:54] without the packaging it python and the scala shipped with the spark the ro swift ras scala 3 rs typescript
[08:06] net speak the same protocol so the next step is to make your udf portable So the DB will come back to that in the looking ahead section.
[08:27] In 4.2 we keep closing the gap with a class driver. Old RDD helpers comes to connect zip with index to JSON empty and the read.json CSV XML contain a dira. So
[08:40] there's a new data status API and the error are cleaner for example when you use a column from the run and the cl names and the line number
[08:54] so debugging becomes easier password in spar 4.2 loop is moving to arrow first passing
[09:07] Python UDF get faster by default. Hot pass can use a native arrow UDF and the Python teams can just get a first class data source plus better uh profiling and
[09:20] debugging. So today's takeway is a simple upgrade the upgrade for the free speed up then optimize only the Python code that matters.
[09:38] So important part is the default in 4.1 you already turn it on but in 4.2 you can you don't need to do anything. So arrow python udf are the default. The old UDF, the old your old Python UDF
[09:51] just run faster. Arrow IPC between the GVM and the Python is on and arrow input skip one conversion step. The upgrade story
[10:04] this bit with no code change. The same UDF you always write. No new decorator, no rewrite. But the engine moves the ideas as arrow same code
[10:18] faster. 4.2 continue the arrow first direction. The tarator arrow UDF since 4.1 a pyro array in a pyro array out. It
[10:33] removes a pander's round chip one less conversion. So in our benchmark it is about the 10 f 10% faster and 40% less memory than a
[10:46] regular pending is a type fidelity. So penders mango types and the nested types are outward. So arrow keeps the
[11:00] nested and the structure type and the use a python standard corion and stays sync friends even on the nested data. So at a different level you also get
[11:13] mapping arrow applying arrow and the two arrow. So the arrow UDF decorators arrays in arrays out no two penders no wrong chip
[11:28] column from start to end build a spark readers and writer fully in Python batch or streaming no JVM no
[11:42] scala register once use spar read format t a growing ecosystem some of connectors you can use to draft a longter.
[12:03] So this is ecosystem today connectors the committee already built in pure Python before you before you learn the JVM API now you just write Python
[12:22] this is a full custom customized sauce a reader class register it read it like any format that's the whole
[12:35] New in 4.2 too. You can provide this patented source like a UDF for times memory. One config turned on. Then you can see where your time memory go in your read and write code.
[12:49] Your connector is no longer a black box. One interops we uh worth calling out is a pi capture protocol. Spark now speak
[13:03] the arrow C data interface. So you hand data to porous DB or other aeronative tools with no copy. So when both sides supported less glue
[13:17] less overhead between the spark and the rest of your Python stack. A sparked frame goes straight to the porous or dark. So the arrow C data
[13:31] interface. So when both sides support it, they share the same arrow memory. No copy, no serialization,
[13:45] no debing. Your Python UDF runs in a Python worker on an exector far from your notebook when it hands before you saw nothing.
[13:57] In 4.2 Two, you can see the H. The worker dump is statues and idle workers are dumped and cued. The one prof provider covers a Python,
[14:11] penders and arrowf CPU and memory by lines. It works on spark connect too. And you can use a Python login interface
[14:23] inside a UDF.
[14:38] So start worker with a stat shown for you and the UDF log you can read back as a normal table. This is just example
[14:52] an AI native analytics in SQL. Today this workloads are scattered a vector system here verbals sequel there 4.2 bring them into the engine vectors functions nearest by qualify
[15:07] sketches geospatcial embeddings are now normal table data. The first need is simple. Compare two
[15:21] vectors where the data lives. Spar 4.2 adds a vector function for that coign similar similarity inner products air2
[15:33] distance normalize and aridations all in SQL. So this is scoring u primit primitive not an index yet.
[15:51] The simplest vector search one query vector against a table score sort take the top 10. It is just SQL. But the real workloads have many query rows at top products for every user by
[16:05] hand. That means a cross join a window and a filter. So nearest by the latest feature replace that. Now we generalize from one query vector
[16:17] to many query rows. For each left row, find the K B right rows. The slow is usually sim similarity, but it can be any person. The old way cross
[16:32] one row number filter has a intent near the buy makes it explicit in 4.2 Two is that is a brute force top K just with a clearer SQL
[16:48] a part keeps the same shape ready for the future index backed plans the same search now batch for each user
[17:00] the top 10 products by similarity the point is the shape not the function no cross run no windows no filter search is a common case but the nearest
[17:12] by fits any top K row uh top K joints. So this demo exact uh process keeps the query stable but for smarter execution later
[17:30] before and now four times each one remove a pattern we have written for years. Qualify top 10 uh top end without subquery filter conditional aggregate now runs over a window too
[17:44] some filter where over that was rejected before 4.2 two pivot with alias readable columns and the cursor row by
[17:57] row logics in pure SQL script. That's um so the takeaway is a reach for the qualify first and use a filter over window where you use a subquery
[18:12] set pass is a simple search path. Before every name needed needs the four TLO and the schema. Now you set the path once the unqualified names resolved across
[18:26] the schemas. It is saved in the views. So resolution stays reproducible but it is op in. You need to turn on this config.
[18:38] It is not input basically objects stay where they are. You just improve the analyzer.
[18:50] A few more comps like a time data type time with local time zone. By the way time data types is not on by default. It's a preview. So it's highly possible 4.3 we can create a time data type
[19:04] and more building functions sketches are small probabilistic summaries one pass small
[19:16] memory bounded errors spar already post them as a secret aggregate before now in the 4.2 two we introduce a tuper sketches. So this a cuber sketches distinct user
[19:30] revenue summary in one pass then we can merge daily sketches in into a m months instant um with with no rescan so location data is everywhere delivery
[19:44] IoT map uh risk 4.2 to add a geometry and geography as a native SQL type by default no actual package geometry is a
[19:56] flag geography is a globe so there's a full SR ID support you read and write in part and the type as srd are kept it works in SQL different pispark and it
[20:10] follows open part and as geo special it cost engine. So, Spark 4.2 support like a multiple ST
[20:22] functions and SR ID helps um yeah build a geometry, write it into part and read it back the type and SR ID tips.
[20:34] Um so far 4.2 2 brings more logics into engine matches, python, SQL but analytics is not only query the data changes underneath
[20:46] um underneath you so DB will show the 4.2 um house bar 4.2 to changes itself safer CDC streaming DSV2 to um yeah DB
[20:59] >> thank you SH for the first portion is my mic on
[21:25] CDC uh and SDP. So I think it's a hard problem actually. How do we keep a table in sync with a stream of the changes?
[21:40] So okay let let's say a common use case. Let's say we have lehouse table and we want to have it in sync with some of the oper operational change. So let's say we have a customer. We have a version one with customer 101 say is in SF and then
[21:55] you have another data coming in version three say then you have version two that is arrival list and it say Los Angeles right so if you want to process by arrival order um LA can you know
[22:10] override Seattle but it's not what we want actually even though LA is the uh order so uh and let's that uh we have a version for we want to delete the data
[22:22] right so a CDC event has a four part so when is a key uh that is by the uh ID over here customer ID operation uh you have different operation and sequence
[22:35] and also the data payload so now the key is the low identity and the sequence is a business order so if you want to process this kind of data you can handwritten for each badge you know, a
[22:49] lot of merge stuff and it's going to turn a really simple problem into a huge challenging, you know, um disputed uh um
[23:01] uh computational problem. You need to handle the order in the tube uh you know retry. So a lots of time you can cause a lot of mistake. So uh in
[23:14] um SDP we have a new feature inside uh spar 4.2 we call auto shdpc uh change of data capture. Uh this can can help to address this issue. So uh
[23:28] SDP here spot the corrective pipeline is a pipeline manager is run as a pipeline. It's handle a dependency data dependency streaming checkpoint for you automatically and it's a special flow uh
[23:42] you know auto CDC is one of the special flow inside uh SDP and you give a key customer ID a sequential column version
[23:54] um and they handle it for you. So uh there are two type of the um auto GDC. One is uh SGD type one and one is SGD
[24:07] type two. Uh HTTP SGD type one keep uh only keep the latest result. So as you see from the first example uh over here uh the last one we will
[24:22] delete the data on the customer uh 101. So we only keep the raker of uh 102. So sedd type one actually
[24:36] one second only keep the uh final result and we are working on open sourcing the type two uh
[24:49] to open source community. The PI is ready actually but uh we haven't be able to merge it into Spa 4.2 in time. So this will be part of SPA 4.3. So we will be able to keep the entire history of
[25:03] the uh CDC. Okay. And on the structural streaming side we also add a bunch of new feature actually. So there will be a full talk by Anish and Jerry um on this Thursday
[25:18] and if you are interested in please attend the talk and here will be the brief overview on the streaming side. So spa uh 4.2 Two, we have some optimization on stream to stream join
[25:30] optimization with fast watermark eviction uh stay integrity check with low label check sum and also uh the new pi bar support for realtime streaming
[25:42] mode. So for the stream to stream join uh in 4 uh spa 4.1 we add a virtual column
[25:54] family that move the joint state into one rock DB store instead of four that give a good performance game but still watermark eviction can still scan too much data in joinstead and in spa 4.2
[26:08] into the state uh data source can read a join state directly and from the state format V4 and this add a
[26:20] secondary index waterm eviction and this also help to you know optimize the left semi and non outer join u performance scan. So over here
[26:33] uh we see that the evision scan go from about 2 seconds for a four scan to about like a 30 millisecond uh bounded per million low. So it's a huge performance scan in SPA 4.2 two
[26:48] and also for the STO integrity >>
[27:01] manual recovery pass still too common and it's really costly so 4.2 two we got five grand a lowle check sum using CIC32 sampoint and we also have the new
[27:14] feature like a automatic snapshot repair low pass checkpoint v2 and also the loss DB memory usage will be unified within the unified memory manager within spark
[27:27] uh so this help to manage the loss DB memory consumption as well and next is uh data source v2 2. So um
[27:39] data source v2 is a way how we can connect spot to data and this will be a short summary of what suhan and g going to talk about uh later
[27:52] um this week. So uh it's a standard API DSB2 for data source. Uh previously you know it was mostly used in iceber but now data is
[28:07] transition to use DSB2 as well as part of the unification between data and iceber and connector. This provide a way that we have a consistent way to handle
[28:19] a meta data you know small uh and and also have a contract between spark and data. So I'm going to talk about more and with that Spark can do heavy lifting
[28:31] between uh uh analysis pling optimization and finally execution. So loads don't have to be implemented inside the data source and this give us a more consistent syntax and behavior
[28:45] between different data source. So um we have a lot of momentum recently in uh B1 B2 uh in DSB2 uh between a spar
[29:00] uh 4.1 and 4.2 into as part of the unification. We try to bring data and iceberg closer. Um we have a implement we have some improvement on DML hardening. Uh how we handle schema
[29:14] evolution inside spark. Uh right summary is a DML uh you know metrics. Uh how to handle uh transaction uh change log and alo also partition
[29:26] predicate. I'm going to share about more. So uh for first class CDC so um there will be a show version of Gandang and uh
[29:38] Johan uh CDC implementation talk uh uh on Thursday. Uh so we were talking about CDC it's on the uh consumer side that uh
[29:50] we can use uh CDC inside SDP but here is how we actually implement CDC inside data uh sorry inside the data source. So previously
[30:02] uh before this uh as part of DSB2 framework data iceber and hoodie they also support a change fee but each has its own API and also different uh
[30:15] implementation and different default behavior and now in spark 4.2 two we make it as a first class citizen so that we have a single change syntax one algorithm inside spark to support uh
[30:29] CDC. So um the first um user facing changes is uh one surface right. So uh today
[30:41] each connector has its own CDC uh implementation has different API and syntax. Now we unified it into a sho key as a changes uh inside spa 4.2 too and
[30:56] this can uh you know help to ask the difference between uh different version different time stamp or a stream changes between different table between a fun table. So um
[31:12] yeah so on the connector side the implementation is pretty uh tiny and intentionally um and and it's only declared the output column provide a scan builder for low
[31:24] change uh low changes low and the name of the low identity and and low version columns and then the list can be handled by a spark.
[31:39] Okay, second. Okay, the next is a schema evolution. So it's another uh another area that SPA DSV2 has been go up in the uh past like one year. So a schema
[31:53] evolution is how spark can compare the source and target uh they can including like a nested column you know is compare where need to be changes right whether you want to add a new column or widen
[32:05] the type you know uh uh upcasting fun like you know low uh integer to long uh feed in the machine target field with null or default
[32:18] and the basic concept is that the connector can decide whether it's valid for this table format or not but spark can have a suggestion on this one have some implementation
[32:32] so data and iceber they might not have this widening rule but DSB to give each connector a place where they can reject it or set it so we can have a more unified behavior if the data source feel
[32:46] like it makes sense for specific data source So yeah uh so a new column uh let's say appear in the source with schema evolutions but can compute a schema
[32:59] changes and ask connected whether this apply is value or not. So this is the whole concept of schema evolution here and the next area we has been improving is uh transaction. So a DML is uh uh is
[33:15] basically doing re transform and writeback. So without isolation a writer can override a changes in neo. So in spark 4.2 we add a transaction API that
[33:29] uh spark can have a transaction for DML. So the connector can stage change uh track reading uh validate a commit time uh so that avoid um you know uh
[33:42] transitional issue. So now we only support single DML uh statement and multi-statement and co catalog transaction will be our feature work.
[33:54] Um and another area is on the metric side. So uh DML on the DML we add a merge operation matrix in spa 4.1 and in
[34:06] spa 4.2 we are adding update and delete matrix and spark can create a matrix in the background and then committ it um uh in through the DSV2 commit API.
[34:22] Okay. And finally on the DSV2 area we work on the uh partition predicate. So it's a really big gap for a DSB2 right now. So in DSV1 because we expose the
[34:35] cat uh catalyst expression directly to data source. So they can do a lot of optimization. For example uh they can uh push down like you know udf equal to
[34:47] certain stuff and they can be pushed down. However, in a DSV2 uh framework is not allowed because we only expose a smaller set of the expression that can be allowed to use in
[35:00] DSV2 framework. So, UDM and non-standard expression cannot be used in the predicate. So we add a new interface partition predicate that can bring
[35:12] specific catalyst predicate into DSV2 framework to provide better uh performance less data will be read and faster query. Okay. Next uh we also have some
[35:26] performance UI uh uh improvement. So the web UI we have new look a dark mode first faster interface and interactive SQL printing um
[35:40] and also uh some park optimization on the vector vector park uh pre-agregation uh and also some cogent improvement and we also improve like a IO so we use
[35:55] arrow to transfer data between Sabian and Python as Shiao has mentioned this cause uh less memory overhead without any code changes. Um and also we support spa uh in SPA 4.2
[36:11] two we start to support JBM25 and in Kuberneti area we also support many new feature like you know impress executor PBC resizing uh having smaller image and some of the network policy
[36:25] isolation and we also improve uh history server as well um that is u uh quite a bit of the implement
[36:41] okay um okay and then looking for the feature. So in spa 4.3 we plan to work on many area. Uh the first one is a project failure. Uh so spark
[36:54] is um one second. Yeah. So spark is really good in large scale reading pabyte of data doing a huge job. However, uh in the late hub experience,
[37:07] it's may may not be the necessary the best environment because it's it was designed as a disputed computer engine. So a lot of overhead for example like a shuffle the printing is for disputed. So
[37:21] we create a new project called project feder that Daniel will talk about that uh later today and it's targeted for small job um for query less than 100
[37:33] megabyte that can still take second but we want to reduce it to like you know uh millisecond and how we can do it so uh we are able
[37:45] to do it by having a faster planning and scheduling for local mall uh single path analyzer uh for optimization on the single file query that can uh make it as
[37:57] a single task so that we don't have shuffle uh directory. Uh we also have the arrow native caching. So the cache will be cached in the native uh you know memory with the arrow format and that
[38:11] can be used by UDF as well. So UDF can now take error as a input without any data conversion and also we have a shuffle free uh local execution as well uh that can eliminate writing data to
[38:25] local this even though it's a single lup uh mode. So I think the goal here is not like you know it's different like a db or uh polus. So the value here is that
[38:38] user can start small uh doing local development with uh first efficiency and they can scale up to large scale in production. So I think that's the key difference between other uh single no
[38:51] engine and another area we are working on is a language agnostic uh UDF protocol. So right now we are support different language for the spark connect but for
[39:03] the EDF UDF part we are not able to support different language mostly Scala and Python. So we are building a new framework where you can execute uh the UDF using different language and even it
[39:15] can be run on the remote server. So you can decouple the spark cluster and UDF runner. So Hayang and Tang will talk about more detail uh this uh Thursday.
[39:29] on this. Okay. And this SP IP has been passed. So we have been lending new uh code into opensource spark right now. So feel free
[39:41] to take the preview and we are also supporting the nancond precision timestamp is a SPIP has been passed uh and we we have been uh actively working on this on the open
[39:54] source community. Okay. And there are other improvement on the spa 4.3 streaming and pipeline um SDP improvement for example like a sedd
[40:06] type two will be supported in spa 4.3 okay and uh I think it's really important uh that we can release the spa uh faster we have been doing it like a
[40:19] twice um uh a year release schedule and we have been doing a lot of innovation in open source bar it take some time to lend it uh to the people who are
[40:31] actually can use it in production. So um so in the community we decide that we want to do a two-layer model. So we have a quarterly uh minor release that we can
[40:45] release new feature and we are able to uh lo um you know feature much more quicker but without any dependency changes and every year we will launch a
[40:57] new major version where we can do breaking changes so we can accelerate the development velocity in opensource spa community. Yeah. So all of them are the new stuff in the uh Spa 4.3, Spa 4.2
[41:12] and then the future work for Spa 4.3. So yeah, thank you for the talk and any question is welcome.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.