Apache Spark™ Connect: The Path to Modern, AI-Ready Spark

Summary

Spark Connect replaces the monolithic Spark driver architecture with a gRPC-based client-server model that eliminates class-path conflicts, reduces driver memory from 900 MB to 80 MB, and has already enabled 6.2 billion jobs to upgrade independently without code changes.
As AI assistants generate more Spark code, Spark Connect's DataFrame-first API design and machine-readable API discovery make it the interface best suited for AI-generated code compatibility, positioning it as the strategic default for the next generation of Apache Spark applications.
Multi-language client support (Python, Go, Rust, Swift), built-in reconnection, idempotency, and exponential backoff reliability features, and full Pandas API and ML library coverage are among the production trust improvements that have shipped since the previous Data + AI Summit.

Apache Spark™ Connect: The Path to Modern, AI-Ready Spark

Spark Connect is more than a new client interface for Apache Spark™ — it is the foundation for a modern, AI-ready, and production-friendly Spark experience. As AI assistants generate more Spark code, the defaults they learn will shape how the next generation of data applications is built — that makes Spark Connect a strategic shift, not just a technical one. In this video, we will share our vision for making Spark Connect the default way developers build with Spark. We will discuss how DataFrame-first guidance, better AI-generated code compatibility, and machine-readable API discovery can make Connect the easiest path for developers. We will also cover the work needed to build production trust, including closing migration gaps, improving observability, and increasing ecosystem compatibility. Attendees will leave with a clear view of why Spark Connect matters now and what it will take to make it the standard interface for modern Spark applications.
Talk By: Haoyan Geng, Sr Staff Software Engineer, Databricks ; Justin Breese, Staff Product Manager, Databricks ;

Chapters

FAQs

What is Spark Connect and why does it matter?

Spark Connect is a client-server interface for Apache Spark that replaces the monolithic driver architecture with a gRPC protocol, eliminating class-path conflicts and enabling the Spark client to upgrade independently of the cluster runtime. This video explains that it has already enabled 6.2 billion jobs to upgrade without code changes, making it one of the most impactful architectural shifts in Spark's history.

How does Spark Connect reduce driver memory usage?

By separating the Spark client from the driver, Spark Connect shrinks the driver memory footprint from 900 MB to 80 MB, since the client no longer bundles the full Spark runtime. Driver isolation also means that a crash in user code cannot bring down the Spark driver, improving reliability for production workloads.

What programming languages does Spark Connect support?

Spark Connect now has client libraries for Python, Go, Rust, and Swift, allowing developers to build Spark applications in languages beyond the traditional JVM and Python ecosystem. This multi-language support makes it easier to integrate Spark into diverse engineering stacks and for AI-generated code to interact with Spark using language-idiomatic APIs.

Why is Spark Connect important for AI-generated Spark code?

As AI coding assistants generate more Spark code, the API surface and defaults they learn will shape how a large share of future data applications are written. Spark Connect's DataFrame-first design and machine-readable API discovery make it more compatible with AI-generated code than the legacy driver model, which is why the presenters in this video describe its adoption as a strategic shift, not just a technical one.

Full transcript

[00:07] Spark Connect. So hopefully you're here for Spark Connect. If not, you are now. Too bad. U we're going to lock the doors. So I'm Justin, a PM at Data Bricks. I'll introduce myself in one
[00:19] slide. And with me is my dear friend Howan. So we will introduce ourselves the the small boy band you see here. Um, I'm
[00:32] greatest hits things. Spark, we all know it. We all all love it. Uh, Photon, a little bit of Delta stuff. I spend a lot of time on release infra as well as the entire dependency management stack.
[00:44] Anything from containers to cluster libraries, firstparty artifact registries, all sorts of fun things. Uh, I live in the LA area in a little town up in the
[00:56] mountains called Tanga. Uh, and I have a wife and then two small tyrants that boss me around and I listen to them. Uh, they're cute and fun. And, uh, happy early Father's Day to anyone and
[01:09] everyone. All right. Um, you can always find me, uh, jaydats.com. That alias was available. I realized that about a month ago and I took it. It's mine. Uh, so you can email me that way. You can troll me
[01:21] on Reddit as well, Justin Bricks. Uh, so I frequently show up there and talk. So, uh, that's me, Hian. I don't think you did an intro slide, but >> yeah, I don't have a slide.
[01:33] >> Who are you? >> I'm I'm the engineering tech lead for Spark Connect. No, I joined Data Bricks roughly a year ago. Before that, I spent several years at Google, you know, doing mostly, you know, doing fun stuff,
[01:47] building OLAP databases at very large scale. So you know basically I came from a distributed system background and that makes me kind of a natural fit with spark connect all those stuff. Yeah. So if you have any questions feel free to
[02:00] you know so talk to me or shoot Justin email or a message and he can relay anything of course. >> Yeah. Anything good goes to me anything bad goes to him. That's that's how we want to delineate it.
[02:12] >> All right let's jump into it a little bit. So um just make sure mic is good. All right. If if you run spark need connect now why uh lots of things so I can't
[02:25] tell you how many times over the years uh in my spark tenure people complain about the class path uh so at the end of the day I think there's only like a few few things in the class path that that happens only only a few right not and
[02:39] when I say few I mean thousand few thousand uh Jackson everyone's favorite thing yes it's there um independent upgrades this is a huge huge one. So instead of having this monolith that's
[02:52] super brittle, super fragile, you know, if you blink at it, it breaks. If we blink at it, it breaks, it becomes a much nicer, cleaner contract. U 10x slimmer images. So as we talked about
[03:05] class path and hundreds, if not thousands of jars, like we don't do that anymore with with this model. And then um not just Scola anymore, not just Python anymore like so you can uh interop with it in many different
[03:18] languages. For example, Go, Rust, Swift, and whatever whatever else becomes cool next year. Cobalt, I'm calling it right now. It's gonna be Cobalt. Um and also the the ability to reconnect midquery.
[03:31] So how many times did internet get weird and all of a sudden you lose access to your query and it's like, "All right, I know it's still running, but I can't talk to it." Well, again, fixed and then um future first. So new
[03:45] features they're shipping connect first now um at data bricks everything is spark connect based so our entire world is spark connect so six reasons and we'll kind of doubleclick into these a bit so as I alluded to class path um so
[04:00] with with this model it's it's much cleaner so like we're not we're not surprising on on the class path anymore it's like literally what what you put in there works um the u lockep upgrades
[04:14] So meaning um traditionally there's been this uh this very very tight coupling. So you had all dependencies, you had client code, you had server code, everything all mashed
[04:27] together made it made it it made it super super hard. Um and for example apps were were stuck on older versions of Scola. I mean going way back Scola 210 211 212. Um then also uh at the end
[04:42] of the day having you know you don't want a huge huge image. So used to be what was it like I've seen gigabytes but now down to down to much smaller. So at
[04:55] the end of the day smaller thinner better faster stronger. That's going to be what we're going to hear about. Now, what this enables us to do, and I'll kind of talk about in a couple
[05:07] slides, is we um by ripping all the dependencies and putting them into the client and getting them out of the server, that allows us to do independent upgrades, which is super super cool. Um
[05:20] case in point, I don't think I have a slide on it, for serverless notebooks and jobs, we've been using Spark Connect from day one. So for about two years in data bricks speak um this was DBR we started in DBR1 14.1 I think technically
[05:34] and I think it was uh 18.2 I think as of last week um or the past couple of weeks um in spark parlance that was spark 3 into spark 4. So a lot
[05:49] of things I mean we're talking about like at the end of the day I think we've auto upgraded 6.2 2 billion jobs and it's been like 99.99999% successful and the whole reason is due to this decoupling. Um
[06:03] three years ago I remember talking to customers saying this is the model that we're going to do it's not going to break your workloads and they're all like Justin you are on crack and I said yes but it's true. Um and sure enough
[06:16] like we've now been you know the thesis has been proven out at this point two years plus. So the idea is this decoupling enables independent upgrades of the
[06:28] server while maintaining compatibility. So you can think about it like things get better, faster, stronger with zero effort. And this would not have been possible without Spark Connect. So that's my mini like whoa auto upgrade
[06:42] tangent. Uh but it's true. It it really is good. Of course, add any flavor, color, seasonings if you'd like, but uh All right. So, there's that. Keep moving
[06:54] along a little bit. Now, how does it work? We talked about, you know, there's this decoupling between the two. We have a client server. So, client um so we can say pretty much any language. I think I
[07:06] showed you the the five that we have and others of course that are always in the works. The whole job of the client is to create an unresolved logical plan. It's all it cares about. It doesn't need to resolve anything. Just create me an
[07:18] unresolved logical plan. From there we send it over the wire via gRPC and it then goes into the the Spark engine. So the catalyst we know we know we love we loathe uh hate all all the things it
[07:32] goes there. Um from there Spark does its Sparky things distributes things off to executors streams the results back via arrow and the client has its answer. Woo! Yay! Awesome. Um, so super clean,
[07:49] super easy. GRPC out, arrow back in. And of course, since if you guys have questions along the way, you can feel free to pepper them in.
[08:05] So, um, six things that this model changes permanently. So, as we as I talked about in my little rant, uh, life cycle independence. So, upgrade Spark without your app. Great. Um, so get the latest and greatest features of a new
[08:17] Spark version without having to touch your application. So I like what that means I for our serverless GC users we looked into the data year um via the auto upgrade from I think between like
[08:30] DBR so if we say 14 through 18 there's been about a 25% performance improvement across like broad benchmarks. So these workloads got better faster stronger overnight with zero effort and so you can do this too. So, auto upgrade Spark
[08:44] from three to four, get all the better, faster, stronger, you know, whether they're Perf improvements or security improvements, it just happens. Um, multiple languages, we kind of talked about that. Going from 900
[08:57] megabyte down to 80 megabyte, super cool, super awesome. Make your app smaller. And networking resilience, we kind of alluded to that uh before too, where meaning you'd have uh connections
[09:09] that break midquery. Don't worry, you can just reconnect. It's still working. Still has a heartbeat.
[09:23] All right. Driver isolation divided. So, talked about already 900 to 80 meg. That's cool. Now, the other big thing is the the memory for the driver goes down a lot. 8 gig down to one gig. And uh
[09:35] build time because we kind of do write and build code. So, this does matter too. Uh I swear. Uh so eight nearly you know eight and a half minutes down to like five minutes. So at the end of the day thinner and faster on the dev loop
[09:55] and also like this is past the early adopter stage. It's like way past it. So again on serverless notebooks and jobs at data bricks it's been in prod uh for uh over twoish years and it runs 100% on
[10:07] this and again we've auto upgraded scary auto upgrades from DBR14 to 18 it just works no one knows or cares I care a lot but users don't care but I guess that
[10:19] means it works so that's nice um also for uh so in shared clusters so for non-serverless we have shared clusters in classic that's been GA for a very long time. Uh it's going to be default
[10:32] very very soon. And um let's see the lakef the artist formerly known as DT or yeah lakeflow DT I think I was supposed to put SDP darn. Anyway, we call SDP because we like acronyms and changing
[10:44] acronyms. We want to keep you on your toes too. So DT SDP same thing but also Spark Connect Native. And if we and if you attended some of the keynotes earlier this week, Ali and Reynolds were
[10:56] talking about Raiden or Reynolds dream engine. Uh which is just a great great name, but anyway, we'll we'll let it stick. Reynold's embarrassed about it. That's why we do let it stick. But the whole idea is that's built to be Spark
[11:10] Connect compatible too. So basically, this allows any workload to just point it to a different engine. So in this case, Raiden or whatever the team dreams up next year, Reynold's
[11:23] other dream engine maybe, who knows. Um, so the idea is it's been default at data bricks. It's, you know, for a lot of our products, that's exclusively all we're running at this point. And uh, also if
[11:37] we look out the community, uh, EMR, for example, has done a few blogs um, on Spark Connect as well and having support for that. So we're just seeing more and more adoption throughout the community. um shipping fast, lots of PRs merge into
[11:51] Spark probably due to cloud code u probably uh at the end of the day I think everyone's just you know shipping lots of PRs quickly no one's having GitHub problems I swear right GitHub's not not down for anyone including us
[12:03] internally um and at and at real scale 12 plus million connect workloads are running per day uh it's kind of insane so yeah it just works
[12:23] So what shipped since data and I summit a year ago 2025 year's a long time in this world. Uh so five official clients as I talked about but I might as well reiterate again Python Scala Go Rust
[12:36] soon cobalt. No I'm joking. Um P50 query latency went down significantly. So 4 480 millies down to 210 for small interactive queries also huge. Um ML and
[12:49] pandas this was a huge huge lift. So getting pi spark ML and pandas now fully supported and covered via spark connect. And then with serverless and some of our other products it's now default.
[13:02] Fun stuff language clients. So I rambled off the the languages but this is like kind of a better uh eye
[13:14] chart or you know things that people like to take pictures on and that's okay. So it's kind of an understanding of where all the various languages are, what is the status of them uh in terms of their development and since when have
[13:27] they been available. So um I don't have to read the whole thing. It's it's all good. Uh but the idea is um you know basically choose choose your own language um whatever
[13:40] your LM whatever cloud wants to write for you uh whatever codeex wants to write for you that's fine uh odds are there's a connect client for it and it's going to work just fine u but personally I see a lot of Python I see a
[13:53] lot of Go um I would say Python more for kind of traditional of course like Pispark applications go is picking up more and more as well as Python more in the app space. So those are my little
[14:06] insights and I would say like obviously Swift is a little earlier um in its in its journey here still experimental and it's recent as a spark 4 whereas whereas Rust is in in beta. The rest though have been GA for a long long time.
[14:27] Python we know it uh we've been around it for a bit here on the Spark side. So the idea is super easy just pip install uh spark- connect and you can use it um so full dataf frame API par so every
[14:42] every operation and pispark works through uh connect for for dataf frame um there's always the caveat people bring up what about my RDDs like no we we don't support the RDD API in spark
[14:56] connect regardless of of which language Hayan can can talk to you about that in depth but uh dataf frame API and at the end of the day like when we look the aggregate usage it's something like
[15:08] 99.95% is all dataf frame API um besides every stack overflow answer uh every stack overflow answer for some reason has like to rdd or like create a UDF that now has
[15:22] a higher order function but whatever we're working through that all right and then And also beautiful thing is no JVM on the client. And that's like one of the main reasons why we can go down from say 900 megabytes
[15:36] down to 80 megabytes because JVM. And I'm sure no one's crying like no I want my JVM back. Like that's that's not happening. But super simple. Um the the main difference with with Spark Connect
[15:48] versus uh non-spark connect for lack of a better term is just in the connection string. So um for example it's you have your your Spark session session builder define your remote. So if your remote is
[16:02] local or if your remote is a true remote so you just point to it get or create and then from then from there you're just using your regular dataf frame API. So oftent times what we see with customers is all they have to do is just
[16:15] change out their initial spark session create command and now your application works with spark connect. So, uh, not a ton of like invasive heart surgery by by any means.
[16:32] Structure streaming as well. So, write stream and then everyone's favorite for each batch. Uh, we have those as well, stateful operators. Uh, and a lot of these are very very new as of Spark 4. Um, query reattach also, uh, a new and
[16:47] also loved feature. and then listeners via gRPC. So, um this was I mean having proper streaming support was a was a big lift. Uh it's multi-year effort. Uh but
[16:59] we did get there and the idea is being able to do structure streaming from a very very thin client in Python for example is is now possible.
[17:18] Scola the JVM decoupled. Uh, so one small jar instead of one huge jar. I mean, that's like the the big thing. And we talked about independent upgrades. That's fine. We don't have to run through that. These are just more for like posterity so people can
[17:31] remember things. Um, yeah. Now, and then yeah, class class path conflict. We can go through that a little bit again, but the idea is we
[17:43] used to see like so much class path infighting. Um I've heard of this for years from customers like why is it like this data bricks customers predominantly to fight through the class path issue they would have an init script which so
[17:56] basically the cluster starts up then they have an init script to do one final override and it was just weird um at the end of the day not kind of natural or again with this model you don't have to do that anymore the class path is much
[18:08] slimmer and uh there are no surprises when the cluster starts
[18:21] So before 247 jars, now 12. So yeah, and everyone's favorite Jackson. Uh you don't have to worry about that anymore. And Netti, you've seen that so many times. Um so yeah, again, as we talked about, we get
[18:34] slimmer, we weigh less, it's all good. And then the three new official clients, Go, Rust, and Swift. And you can see kind of where they are in their their
[18:47] various maturities. So Go meaning has the uh dataf frame API catalog, streaming reader, and um Rust, we have dataf frame, udf registry, streaming is
[18:59] in preview, and then swift. Uh you know, we're still in experimental phase, so a little bit of less support, but kind of core dataf frame things. Um yeah
[19:15] and then one protocol but five idioms or five ways to do the same thing with different languages. Uh so you can see it here. I mean so basically what we're talking about is um dataf frame API and how that language works. So don't have
[19:27] to go go through it uh one by one as fun as that would be me reading code but I think you get it. So basically do the same thing in dataf frame and it just works. Those are your clients. And
[19:39] again, the server can read that unresolved logical plan, figure it out, and send you send you back exactly what you need. How? You don't have to hear from me anymore. So, uh, now you have to hear
[19:52] from him though. >> Yeah, I guess. So, all the fun part now have finished. So, I am taking over for the not too fun part, which is actually a good thing. >> Perf is fun. Perf is fun. >> Yeah. If you think about you know things
[20:05] like performance or more specifically reliability right this used to be maybe some hard problems that's makes people wake up at night or you know make you really think okay you're tackling the
[20:18] hardest problem of all but now after several years of evolving and development you know smart connect is reaching the point where you know basically this thing becomes no longer interesting which is a good thing both I
[20:31] guess for developers like me and for you guys as users. Yeah. So since last year you know we have as Justin also kind of mentioned half the query latency for you
[20:44] know small interactive queries at P50 and also at the same time for you know those queries that do return large results for the throughput of streaming back the result we also you know kind of
[20:56] well made a 2x more than 2x on that and so this is because of a combination of you know improvements within spark connect itself and also improvements from the engine. So maybe just take a
[21:09] few small examples, right? So I guess the biggest solving point for spark connect is really, you know, for this whole result delivering path, it's just taking sending arrow batches all the way. That's it. No unnecessary, you
[21:22] know, row to columnar, comma to row conversion whatsoever. And that saves you a lot. And also we have done you know quite a few targeted optimization that people want you know from the
[21:35] feedback we get for example you know say sometimes the plan when serializ into portal buffer can be pretty large right so why don't we compress on it and also you know so on the server side you at times need to keep this local relation
[21:49] and previously you know if this goes up to this threshold of 2 GB then we just say hey stop it we don't support anything more than that now We lifted that that limit altogether and also made
[22:01] a whole bunch of optimization to improve the performance when this is large. And also you know before in some old Spark connect release whenever you send an individual query you do need to pay the
[22:13] cost of establishing a whole new connection and that's not free that takes time. So now with connection pooling you know this is no longer the case and you know that all that part of latency overhead they're gone.
[22:27] And you know I want to maybe double click on this one point where you know basically I said in the result delivering path you know we have this arrow format end to end right this is really a key enabler for performance
[22:41] into the future and the reason why I'm saying this is because just think about it so if you have a modernized engine like you know if you enable photon for example right the whole key processing
[22:53] execution stage is already vectorized and you are processing those batches in columnar fashion and then on the client side if you're using things like pandas polers or duct db which are all pretty
[23:05] popular right they support taking in arrow batches and operate on those natively so in this sense basically it's not just the resulting path it's the whole processing execution plus result
[23:18] delivering plus the client side handling everything really from end to end no unnecessary format conversion and zero copy whenever it's possible. So this really you know enables a whole lot of
[23:30] performance optimization. You know the the essentially the ceiling becomes much higher. Yeah. And of course you know these are just some of the example we mentioned going forward if let's say you know
[23:42] there are new performance optimization opportunities we find or you know any of you guys tell us then we will not miss that. We will catch and further improve. Now on to reliability. So one killer
[23:57] feature I would say for Spark Connect is you know you can rely on it to reconnect to ongoing queries and not worry about those things going away if the connection the network is trendingly flaky. Right? So for each request sent
[24:11] from the Spark connect client to the server it's identified by a unique operation ID. And what that gives you is if you know for whatever reason the network is doing weird things and you know the same request gets end up
[24:24] sending over to the server side multiple times we will not duplicate the execution. It's just going to be once and that's it. You don't waste resource and also if that request have side effect you know it won't have unintended
[24:36] results causing correctness problems. And if you know let's say you are issuing queries from your laptop and somehow your Wi-Fi goes down you know momentarily so that that ongoing query
[24:49] will not get lost and the client whenever it's back up it can automatically try to reattach to that ongoing query already and same can be said on the server side as well. The server knows okay there is this client
[25:01] sending me all the request and as long as that client is not going away you know for a very long time or completely dead then the server knows not to clean up those ongoing queries kill them immediately so that part is also
[25:14] preserved and if you know let's say in in some cases the query is not always succeeding right so if we hit a trending error then the client do automatically retry and there's exponential backoff baked in
[25:28] natively So you are not bombarding the server and when it do comes to the case where you know some query fails permanently we service that fast and also we are making a big push to improve
[25:40] error code error class SQL state and all those to make sure you know the errors they are understandable and actionable
[25:52] now you know if something unexpected do happen and of course you know as an engineer I know I can't say 100% it it will never happen right So by design spark connect is also very friendly for you to debug it because you know so the
[26:05] reason we choose to define the the client server protocol in in the portal buffer spec it gives us this natural advantage. So basically let's say if you have a whole bunch of you know a big
[26:17] chunk of code doing very complex chain together data frame operations right they're not just maybe presented as as a chunk of string in you know in whatever ways you you're written instead every
[26:29] single operator every single expression they are going to be converted into typed class defined in a centralized place that's you know basically the standard spark connect portal definition and so you can look at it and directly
[26:42] you know visualize then debug and see hey you know is there anything wrong or not for SQL you know we don't need that on the client side so basically it's just going to be a SQL command with the SQL string inside it and you can tell it
[26:54] that yeah and for all those plans you know let's say if well typically you can just directly get it on your client side but if you know for whatever reason you want to do in-depth debugging or you
[27:06] don't have enough information and all you get is the spite string of the serialized portal buffer there's also standard commands so you can des serialize and you know basically reconstruct this whole portal structure where you know you or your AI can go in
[27:19] and just inspect what's going on and one other nice thing about portal buffer is you know so every field in any portal buffer message is numbered right and
[27:31] this gives us the spark connect developer a very easy to follow best practice to make sure you know it's very unlikely for us to make a mistake and what is what I mean is basically if
[27:43] we're adding new field new features the simple rule is like okay don't touch all those existing field use you know assign field numbers that are never used before right and what this gives you is
[27:56] essentially what Justin described you know you have completely decoupling the client and the service so you're you know if let's say your client is running old and the server you decide to upgrade right so whatever your client sends over
[28:10] this server should still support and there will not be breakage. And the other way around too, if let's say you happen to use a new client against an old server, well, as long as the old the new client only send the subset of, you
[28:23] know, semantics that this old server still supports, then that's all also okay. Nothing will go wrong. So this is basically what we put into in terms of performance, you know,
[28:36] reliability in the core spark connect service. What about all the you know dependency libraries people love right so last year in Das I wasn't here Justin was here but if let's say he got the
[28:50] question about hey what about pandas API or machine learning there used to be you know the only answer we can give is okay we acknowledge that's a gap but no longer today so for pandas right now
[29:03] spark connect do have full coverage so all the documented pandas api they are natively on spark connect you can use it freely and also you don't even need to change your code on how you write them. So your existing you know pispark pandas
[29:17] code will just directly work and you know if let's say you are operating your data in pandas data frame and some of other part of your code you know is doing this with native spark data frame
[29:29] it's very easy to convert in between these two and because they are both columnar format so it's not really expensive so let's say you know if you have something like the code on the right so
[29:41] you're doing some data frame operation in pandas and at the end this df thing is really a lazy panda data frame right so if the next step you know by by now let's say if the code is as is nothing
[29:55] happens basically no execution but if next step you're doing df.2 two pandas then the the smart connect client will just fire up the execution request to the server and then in the end what you get is essentially the materialized
[30:08] panda data frames and all of these they are in arrow they are in columnar and there's no row column format conversion as we described before
[30:22] also machine learning right so the support for machine learning library is also much better for those you know who love using pytorch Spark connect now do support torch distributor where you can use to launch your pi you know pytorch job onto a
[30:35] spark cluster and then so you know you care about what you want to do and the spark connect connect cluster take care of all the rest of you know scheduling task and you know do the coordination and all those things and also for the
[30:49] standard machine learning use cases the pispark.mml library is also fully supported by now and there are also other libraries is one example is this XG boost where you know people I think
[31:02] yeah quite a few people they're using it to build those tree structured boosting pipelines to do their machine learning training tasks and classification task right so now um last time I checked the
[31:15] you know we we work together with their community and the code to support spark connect is fully merged so with the next xg boost release that spark connect support will also become native there
[31:27] So all of these are you know basically good features things we have in spark connect Justin earlier mentioned you know this is kind of past the early adopter stage and it is true so for serverless at data bricks right every
[31:40] single workspace is powered by spark connect and then you know for shared cluster I think since DBR14 you know if you're you're using it both in scala and
[31:52] python and sql you know so spark connect is also native and what's may be slightly less known is even for single user or dedicated cluster which is designed for you know non-spark connect
[32:05] or spark classic right there's also a way that you can configure in the cluster cluster settings to allow external connection to attach to the cluster and if you do that then in theory you can use dbconnect to you know
[32:19] use spark connect protocol to you know attach to this cluster talk to it in the language of spark connect but of course you If any of you are still running you know dedicated clusters, we do recommend
[32:31] strongly that you consider moving to either shift cluster or serverless because that will save you a lot of maintenance trouble as well.
[32:44] Yeah. So you know basically all what this is saying if let's say you have a lot of your spark programs and you're doing things like you know data frame API SQL or you're doing machine learning on Python or structure streaming then
[32:57] yeah spark connect is ready and it's production grade please consider moving over to spark connect and you know it's natively supported and also one more thing to call out you know if let's say in your team right people like to write
[33:10] their application in different language anguages then for now I think spark connect is probably the only way out spark classic is not easy to support all the various languages and speaking of
[33:22] that just to mention there are five pretty much standard popular language you know client language implementation of the spark connect client right but that's not all in fact if for whatever reason right you decide to write your
[33:35] application layer completely in c C++ orn net there are also open source implementation ation of the same spark connect client library and also we are recently adding you know Scala 3 and
[33:48] Ruby client among others. So the portfolio of languages is only going to increase. But of course on the other hand spark connect is not supporting everything in classic. So notoriously you know if
[34:02] you're using spark context API sure it's not there. If you're using RDDs you know so far you're out of luck right? Or let's say if someone's doing very low-level direct access to the driver
[34:14] JVM that's also you know by design not allowed because of this abstraction layer we we intentionally imposed but for some of these actually we are actually working on you know to inspect
[34:26] okay for example within spark context within RDD right which subset of those APIs are in fact not touching the low-level details and not breaking the abstraction boundary and try to figure out and work with community to see if
[34:40] there's a way to provide some sort of a limited support to those and in the cases we really don't right then we are also making a big push to try to tell all the cloud codecs all the um agents
[34:54] out there on hey you know well you need to learn what's the alternative way you should write the code and how to reason about correctness so that you know basically if let's say any of you want to migrate your old code to spark
[35:06] connect compatible you will have a much easier time. Now if you do think okay my code my Spark program seems ready and I want to try out Spark connect all you need to do
[35:19] is pretty simple in fact. So you just you know go ahead and download install the Spark connect client and then in your code you do need to make some small change to swap out the Spark session
[35:31] creation to say okay create Spark connect session and connect to this cluster. But that's pretty much the code change and after that you can try to go ahead run all your test. If indeed you
[35:43] know you're not using any spark connect incompatible features then this test should should basically pass. But if you do hit one of those then the error should tell you hey you are using these
[35:55] and that which are not supported yet in Spark connect. And that's where you know maybe we or your AI need to go figure out how to modify the code right assume we pass that stage then now we can
[36:07] basically just go ahead and get rid of the clunky big spark driver altogether from your application layer. So no more coupling with that big monolithic thing.
[36:22] Now you know threat connect today already is pretty ready but you know we do have a lot of things on our mind to do next. So first of all is you know client languages as I mentioned you know there are now maybe almost 10 different
[36:35] language implementations of the spark connect client but we are working you know on our own and with the community together to make sure those languages you know they they get more mature and more ready for you to use. And for
[36:48] structure streaming you know we are continuing making improvements on various places including you know the re reattach behavior and also spark declarative pipeline would allow you to
[37:00] natively define your DT pipelines on on top of spark connect and in terms of you know the spark connect core capability itself right so we really want to make
[37:12] spark connect the standard the particle for the agentic AI error. So we're making a big push to make sure you know AI understand how to write Spark connect code and how to debug the code so that
[37:26] basically you know your user experience when on Spark connect with your AI agent can be much more easier and then also you know there are other nuanced things which you know individually may not seem to be big but they can be really you
[37:39] know painful if you hit it and we are actively addressing those. For example, you know, we're talking about, okay, there are all these spark connect clients implemented in different languages, right? And so you can directly integrate your application with
[37:51] that. But what if let's say, hey, I write my UDF in Scala 3, but then you know the Scala serverside runtime doesn't support that. What's the way out? So we are investing in, you know, can we express the UDF spec in a
[38:05] language agnostic way so that that problem can completely go away, right? Also, you know, there are other features we're looking into. For example, you know, in your Spark Connect sessions, you should be able to select and switch
[38:17] catalog and all those among many other features. But eventually, the end goal is to say, you know, maybe sometime in the future, not too distantly hopefully, we want all the new Spark Connect, all
[38:29] all the Spark program to speak the language of Spark Connect. And so you don't you you don't need to worry about all the Spark classic nuances anymore. So that's that. Um I think I will close
[38:42] with maybe you know one quick demo or you know maybe just some early screenshot to show you okay how we are making spark connect more AI native. So
[38:55] one question I constantly get I don't know Justin if you had similar experience is that you know if people thinking moving to spark connect and especially if they're moving to spark connect using some kind of a you know
[39:07] managed engine for example the data bricks serverless right what they are worried about is hey now I no longer have access to the you know spark driver to the low-level details the logs and those how am I supposed to debug if my
[39:20] query doesn't go as expected right And what's worse is I don't even have that good old Spark UI to look at. Right? So here we are actually in the early prototyping stage to build something
[39:33] that's you know native to spark connect a UI on the client side which basically you know maybe reminds you of how the old Spark UI look like right so when when you go in you know it will it will
[39:46] tell you okay this is the all the history of your query the queries you run and let's say if you click into one of these which is what this screen screenshot about you know you will see the query string all the metrics you
[39:58] here and the status the error message and so on. But this one is not the most interesting you know screenshot. If you scroll down on this page on the right side you will see okay this query failed
[40:10] but now this new UI actually tells you it fails because there's a data skill problem. there's this you know one executor task maybe taking around 38% of all the drawing data and you know it's
[40:24] like spinning like crazy right so that's the that's exactly the problem and the reason why you know we can show this on this page is because you know in the background under the hood we ask the the
[40:36] AI to automatically analyze all the logs all the events and come up to this conclusion so now you know here you directly know okay well what exactly is the problem? And what's even better than that is the
[40:49] bottom part, the the green box. We not only show you, okay, where where this thing goes wrong, right? But also we tell you specifically what exactly the thing you can do to you know make this
[41:01] error go away. And so in some scenarios where you know you can actually actively do something to fix and mitigate the problem, we will show you directly the actionable items. But there are cases
[41:14] also where you as the end user don't have control. In those cases you know if let's say the spark engine under an underlying under the hood can automatically do something then we will also feed that directly into the engine
[41:26] to make the adjustment so that you know next time if you retry the same query it has a higher chance to go through. So this is just one of the examples on you know how we as a whole right making a big push to make spark connect really
[41:40] the standard for the AI time. So with that it's pretty much it. Thank you. >> Yeah. Thanks everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.