Faster Queries in Local Laptop Mode for Apache Spark™

Summary

Project Feather is an open Apache Spark initiative focused on eliminating fixed overhead costs that make local-mode queries on datasets under 100 MB disproportionately slow compared to large distributed jobs.
The project addresses four categories of overhead: query plan creation and scheduling, task scheduling for single-partition data, an Arrow-based caching implementation that cuts cache load times significantly, and shuffle-free execution using Java virtual threads.
Together these optimizations position Apache Spark as a practical unified engine for both laptop-scale exploration and petabyte-scale production without requiring separate tools for each scale.

Faster Queries in Local Laptop Mode for Apache Spark™

Apache Spark™ is optimized for distributed large-scale processing, but this architecture introduces significant overhead for small, local datasets — often making sub-100 MB queries take several seconds. In this talk, we present a suite of optimizations that dramatically improve Spark’s local-mode performance across three fronts: smarter query compilation and task scheduling that eliminates unnecessary shuffles, an Arrow-based df.cache implementation that cuts cache load and query times in half and a novel shuffle-free execution mode using Java virtual threads for lightweight in-process data transfer. We will walk through the design, benchmarks and engineering trade-offs of each approach, and show how together they position Apache Spark as a practical gateway engine from laptop-scale exploration to petabyte-scale production.
Talk By: Daniel Tenedorio, Sr. Staff Software Engineer, Databricks ;

Chapters

FAQs

What is Project Feather in Apache Spark?

Project Feather is an open initiative within the Apache Spark project—tracked through a Spark Improvement Proposal (SPIP)—focused on making Spark run significantly faster on local machines and laptops. It targets small datasets under 100 MB where fixed overhead costs from distributed scheduling and query planning dominate total execution time.

Why is Spark slow for small local queries?

Spark's architecture was designed for distributed large-scale processing, so it incurs fixed costs for query planning, stage scheduling, and task serialization regardless of data size. For large jobs these costs are negligible, but for sub-100 MB queries they can account for several seconds of overhead, making Spark feel slow compared to single-process tools.

What are the main optimizations introduced by Project Feather?

Project Feather introduces four categories of optimizations: a single-pass analyzer that replaces multi-pass query evaluation to reduce planning overhead; smarter task scheduling for small single-partition datasets; an Arrow-based df.cache implementation that cuts cache load and query times; and a shuffle-free execution mode using Java virtual threads for lightweight in-process data transfer.

How does Project Feather benefit data scientists using Spark locally?

By reducing fixed overhead costs, Project Feather makes Spark a practical tool for interactive exploration on a laptop, not just large cluster workloads. This allows data teams to use the same Spark API and code locally during development that they will use in production, eliminating the need to switch between different frameworks at different scales.

Full transcript

[00:08] Yeah. So, this talk's called Project Feather, which is a name we made for a project to make Apache Spark run faster on laptops and local machines. Um,
[00:20] it's important for reasons we'll get into, but there's a lot of focus on like big jobs with Spark and how to do them very well. That's something that Excel's at doing. Um, but this is a specific
[00:33] focus on running queries on one machine because it's an important part of what Spark can do. Cool. So, yeah, as we know, the original
[00:45] design of Spark was to run the same API and the same infrastructure on one machine and scale all the way up to big clusters and anywhere in between. That was the original vision. Um and and it
[00:58] still can do that. Um but over time it's kind of focus on big jobs kind of like led to some design decisions that um can make it a little slow on laptops
[01:10] sometimes. And so we'll talk about what we can do. Cool. So um this all this stuff is covered in this SPIP called project feather. It's an open this doc is open
[01:23] and describes what we're going to do. Um, so thanks for everyone who like participated in the discussion on this and voted on it. I think it passed the 24 plus ones. So if people seem to like
[01:35] this idea, hopefully you guys like this idea too. Cool. So um, yeah, and the doc is still open and you can still comment on it if you want. So please feel free to
[01:48] collaborate in the discussion. Awesome. Okay, so let's talk about why this is important, why we care about it first. Yeah, so as I mentioned, Spark API is
[02:01] the idea is to have the same API for both big and small jobs, right? That's kind of the design. Um, so the problem is that when we do
[02:14] big jobs, we need to do a lot of scheduling and preparation of stages, right? So we need to plan queries, prepare the job and we do those things. Um but as we've put a lot of energy into
[02:28] the implementation to build those things over time um little overheads have crept in uh that don't really proportionally show up and no one cares for big jobs but for small
[02:41] jobs the they take up a bigger percentage of the total time. So these are like small fixed costs start to show up. And so there's some cases in the SPIP that kind of show up that you can
[02:54] call out where around 100 megabytes or less of scan data is where this stuff starts to show up, right? You're quering parket files or CSV files or whatever you want.
[03:06] Cool. So um what success looks like is to cut some of these small overheads um and make laptop queries on Spark queries faster, right? And then this
[03:18] will have approaching zero effect for large jobs, right? It won't really change for large jobs. It'll take like small amounts of overhead out, but that doesn't really matter anyway for large jobs, but for local mode query
[03:30] execution, it will matter a lot. And so that's the goal. Cool. So yeah, we want to go faster from Pispark API to results, right? And we
[03:43] want the behavior to be the same. No change in answers, no change in behavior. Uh we don't want a lot of risk here. Spark is used by a lot of people for very important things and we don't
[03:55] want to jeopardize any of that. So we want to be careful. Yep. So how are we going to do it? Um the answers that we'll go over here, we're going to try to remove stages that
[04:08] are not necessary, reduce overhead from metadata crunching and analysis, and basically try to start less distributed machinery in general. If the query is simple and or small and
[04:23] it can fit on one machine, we'll just try to do that. Awesome. So yeah, as I mentioned, so there's three kind of sources of latency that we'll kind of zoom into here. First
[04:35] one is creating the query plan and scheduling tasks. So that's one. Second category is DF. Cache, caching things. Um, which we feel deserves its own kind
[04:48] of section because people use it a lot. And if you can cache a big complicated result, then you can prototype on the cache result pretty quickly, right? And
[05:00] then the last category is shuffles. Shuffling between stages because that's part of a query execution, multiple stages, right? So we think of these as bottlenecks. These are not bottlenecks
[05:13] on big jobs, right? They're just not. They're like a tiny percentage of the time. And it's basically need to focus on getting fault tolerance to work
[05:26] properly. Adaptive execution concerns like that. Getting the pipeline to finish reliably is the goal for big jobs. Um and but these become bottlenecks for
[05:38] small queries. Awesome. Okay. So those are the categories. Here's the first one. Creating the query plan and scheduling tasks. So let's think where are we spending
[05:50] time right now? So we can think of this I like to think of this in two cases right one where the data is small but the SQL is big
[06:03] right so in all these cases as I mentioned we're going to zoom into when the data being scanned is 100 megabytes or less right so maybe from one table um
[06:16] it's common to have just one parquet file with 70 megs in it just one file that actually happens a lot for example stuff like that and from that one file you can do simple things or you can ask
[06:28] very complicated questions right so the first case let's let's think about OLAP style cases where the
[06:42] analyzer can kind of eat up time and kind of take up a lot of latency um so looking into this problem the analyzer
[06:55] the way it works is use catalyst framework for running rules to a fixed point just the same as the optimizer does. Um and so this is very powerful and it works well for query optimization
[07:07] where one rule might change the plan and then rely on a loop of the batch to let column pruning or whatever other rule filter push down run again. Right? it
[07:21] might need to run multiple times, which is great for optimization, but for analysis, it's not a great framework um because of several reasons. It's it can be slow if we loop through the rules multiple times. Um it can be a little
[07:35] non-deterministic. um changing a rule ordering can that you don't think about it but changing any ordering of the rules can alter the behavior of analysis in ways
[07:48] that are really hard to understand at the time. Right? So to to resolve this um we're making a single pass analyzer framework where basically we're going to do single
[08:03] bottomup traversal of the tree pre-order traversal of the tree starting at the leaves resolve the leaf scans resolve the relations and the leaves and then moving up to their parents and so on all the way to the root right in just one
[08:16] pass. um we're going to reuse the same data structures that we have now. So we have the unresolved nodes that we use today um generated as
[08:28] a result of the parsing step right or initial unresolved data frames those those will still remain but we'll just do one traversal over it um in o on the
[08:40] size of the tree right so um the idea is we'll keep all the state that we need during this analysis step in either the nodes themselves or other objects.
[08:54] Specifically, there's one called the name scope that we'll use for resolving um attribute references. And this is kind of a clean way to avoid
[09:06] a lot of intermediate assignments of fields and metadata to nodes just for purposes of tracking state. Right? So, we don't need to do that as much, but in this object, right? And so the idea is
[09:20] every time we visit a node, we have exactly one step that we take depending on the unresolved node ID, what type of node it is, right? And we do exactly one thing
[09:33] and then once we're done resolving it, that's that's it. Um, so that's kind of the idea. Cool. So this is the single pass analyzer. Um it's being checked it's
[09:46] being checked into Spark as it's developed. You can check it out. It's under the resolver directory of the analyzer. Um and this will help helps a lot with small queries. We see cases where you know can
[09:58] cut analysis time substantially. Awesome. Okay. So then that this will help a lot when the data is small and the logic is big.
[10:10] What about if the data is small and the logic is also small? You're asking simple questions, right? And um the
[10:25] interesting thing in that case is if you're asking simple questions about it, we might schedule like two stages or three stages today and schedule tasks on those stages. The thing to keep in mind is this task scheduler
[10:39] that we use today was built for huge jobs, right? So it will schedule tasks on many machines. It can scale out to many executives. It's powerful, but sometimes you just
[10:51] want the small query to run, right? So um to to address this we want to reduce the serialization deserialization overhead
[11:03] of small queries like that when you can fit the whole thing often on just one task or just one stage right so sometimes you can fit the whole thing on one task
[11:16] and that's it you don't need any shuffles right for example like the case I said where there's 70 megabytes in a one paret file And then you aggregate from that.
[11:28] That's it. You can actually do that on just one task. You don't need to shuffle. You don't need two-phase aggregation, right? Um and that way we only pay the overhead of scheduling the task once,
[11:43] right? So the the way we do that mechanically, there's this uh insure requirements rule that the optimizer will go up through the tree and say, do you need to shuffle? Do you need a shuffle? What are your partitioning requirements? We can update
[11:57] the that one single file paret scan to return the single partition, right? So I have one partition. I don't I don't have an unknown distribution of data across
[12:09] files because I only have exactly one file, right? Um and then that's enough. Then the optimizer will skip the shuffle, right? So um here's the example I was saying prior to this work um we we
[12:24] schedule this query single file table perform a sum aggregate we would scan the file do a partial aggregate shuffle
[12:36] and then on the next stage read the shuffle files back and perform a final aggregate is generally what you need to do if the scanning part is across many tasks Right? For example, like a count star,
[12:49] you would do a count. You do a count on the partial aggregate and then a sum of the counts on the second step. But you don't need to do that if it's just one file on one task. So we just do the
[13:01] aggregate on the one task. We skip the whole stage. Um right. So for more slightly more complex example, this idea also works for small in-memory tables. Um right if it's like
[13:17] 10 megabytes of in-memory data we can also put it on one task. So you can do this with a union all where prior to this work you would have two stages for each of the scans and then shuffle and
[13:29] then union them in the next stage. But now we can just scan the parquet file and then we already have the inmemory data so it's there and then we just say we're done. Union is complete. One stage
[13:41] one task. Cool. And what else are we spending our time doing here? Let's think about it.
[13:55] Right. Another source of time source of latency that we're spending time on is AQE, adaptive query execution. Super critical for big jobs, right? Where you're running stages in the pipeline and you find out halfway through that
[14:07] there's an empty stage and you can empty relation propagate your way to to success. or other optimizations like that. Um, but the steps that AQE performs can have
[14:21] overhead that shows up for small queries. For example, comparing plans using the hash can accidentally trigger subquery executions
[14:33] and that's slow, right? So we can fix that by um using identity based hashing inequality to avoid that right we don't want to run the subquery multiple times
[14:45] aq is like doing this comparison multiple times as it goes through um
[14:57] cool so yeah by the same token when we're adaptively deciding what to do stage boundaries we have hash joins um we can decide between a shuffle hash join Right? Which is where you scan both
[15:09] sides in their own stages and then shuffle both sides by the join keys and perform the hash join in the following stage. Right? It scales very well. Right? You can have large large inputs.
[15:23] Um or alternatively, if you have a very small hash join on one side, right, you can perform a broadcast hash joint where basically like if you if you
[15:36] have a very small build side, you can broadcast it to all the probe side and skip a shuffle, which is great, but that can move a lot of data, right?
[15:48] So um we don't want to do that sometimes and when for example when the probe side is much smaller than the build side broadcasting the build side doesn't actually help because the broadcasting
[16:01] overhead is greater than we would have spent just shuffling right so it works when the build side is very small and the probe side is very large but if the probe side is super small it doesn't actually help so we can
[16:15] skip it right And in addition to that, some of the probe side plans are just completely independent. So we can start both of them at the same time. Right? With the broadcast case, you have to wait until you're done with the build
[16:27] side, then you start the probe side, but you can start both of them at the same time with the shuffle join, right? So you can kind of get some parallelism there,
[16:39] right? And finally, the statistics can be wrong. It's optimization, right? um it looks the the optimizer makes guesses based on the initial plan but
[16:51] that can be different at runtime. So shuffle joins in these cases are much more robust if we're make wrong estimations. It doesn't fail catastrophically as much, right? So they can it can kind of
[17:04] prevent really really bad cases from hitting your longtail P95 P99 latencies get really bad if you make the wrong decision here. Awesome. And then the last thing for
[17:16] stages and stage scheduling, broadcasting stage description is expensive, right? So it's something both the driver has to do to prepare it and then the executive has to do
[17:28] work on its end to decode it, right? Um so for short queries, this can add up, right? And we don't like that. So the answer to this is we can actually just
[17:40] inline the stage description directly into the task and ship it out. Then you don't have to have everyone do that, right? So we don't have the broadcast set up and you don't have all that overhead. You just like inline it in. If
[17:54] you have only like two tasks or something like that, then it's pretty easy to do. Cool. So those are overheads on kind of scheduling
[18:06] this first category, right? Y'all still with me? Awesome. Okay, now let's talk about df.cash and how we can make it better, right? Because this
[18:18] is the original part of Spark. Um, it's kind of unique and very powerful. Um especially on laptops and prototyping it's it's actually particularly great if
[18:31] you can do some expensive thing first and then cache it and the result is small cache it you can often prototype quickly on the cache result but how can we make this better like
[18:43] what what happens where are we spending time now when we're doing this stuff
[18:57] format and this cost CPU and and latency, right? And so if you think about it, this is actually kind of a central local mode feature part of the experience and
[19:11] it's very important. It's not just like a feature. It's a kind of like very critical um right. So here we go. Yeah. So basically looking
[19:24] into it, it uses kind of a row oriented protocol um where get input rows and then serialize them to the format. Um and this is kind of inefficient sometimes.
[19:37] And so when you're repeatedly caching and querying the cache result many times um the inefficiency of the row oriented protocol can add up, right? So the idea is if we can speed
[19:50] this up quering the cache result gets fast and you can do that many times.
[20:03] Yeah. So the answer that we've arrived at here is um the Apache Arrow project which has actually moved a long way in the last five years or so. Um Arrow is great for
[20:15] moving data around in memory. It's kind of this this library that makes vectorization vectorized processing columnar processing of data really easy to do right so use the arrow
[20:27] library like I want to ship data across this RPC boundary this IPC boundary wherever um you can also put it in storage although it's not the primary
[20:39] focus of arrow but you could also like arrow encode a bunch of data and then store it in that format it has limited compression right? Um there's better
[20:51] in there compression formats for storage right out there. But depends what your trade-offs are, how much you're willing to spend on the serialization
[21:03] and how much your storage costs are. Um but Arrow excels at intermediate transfer of data from one place to another in flight, right? So the idea is
[21:15] if we use arrow for this instead it's been optimized significantly optimized by many people. It's columner right and it has compression and the compression
[21:27] is been balanced for a good trade-off for the cost of the compression versus the benefits of consuming it. Right? it's not going to overly exert CPU and latency for
[21:41] compression that you're only going to decode it once. And
[21:54] to memory or disk, right? Local disk or memory. With this compression, this can make the difference between being able to fit in memory or not, right? So if you have a lot of data that you either
[22:08] couldn't cache before or could only go to local disk, if you use this compression now it'll fit in memory and that's like order of magnitude kind of performance win, right? So it can make a difference between being able
[22:20] to fit in memory or not and you get the vectorzation, right? You get the columner access on top of that. So it's it's a pretty solid win. Um, right? But
[22:32] the there are costs to encoding and decoding and whenever we kind of look at this we have to make sure that whatever we're paying whatever we're paying for the encoding and decoding kind of yield
[22:45] results in the end right so we don't want to overspend on serialization and have a net loss um so that's just a detail to keep in mind as we work on
[22:57] this um in initial prototyping it never has not happened yet, right? Like pretty much if as as long as you query the df.cash once, then it's a win.
[23:20] Cool. Yeah. So, this is implementation is underway. Um, here's the Jira umbrella Jira for the SPIP and you can go look at it. Um, and the subtasks will appear here. Um, we also
[23:33] welcome contribution from the community as well. If you want to help with this, um, this is where to look. And then here's like initial work. If you're curious of poking around what this might look like, um, this is initial work
[23:47] going out where, um, the PR kind of describes what we're trying to do, right? So, here's a like an example. There's this arrow cache to batch class that represents a bunch of data that's
[24:00] been serialized into arrow and cached, right? And then this method that convert to the arrow batch will take the input data and then go through the arrow
[24:12] API, the arrow protocol and code it up. So you can actually poke around at the steps that will take if you're curious about this. Um yeah, so this this should
[24:24] be should be good and it should not increase the complexity of Spark a lot because all the like details of the compression and stuff are handled by arrow.
[24:36] Cool. Awesome. Okay, that's the second category. Last category, shuffles, right? Shuffles are expensive sometimes, but they're a core part of how Spark
[24:49] works, right? Um, but in some cases we can think about skipping them for small local queries. And we'll think about exactly how we can
[25:02] do that here and how we can make that work and fit into Spark without really losing Spark's focus on big data that it already has. We don't want to lose that. Super important, right? Um,
[25:16] so let's think about what are shuffles and why are they good for big jobs and what can we do about small jobs? Okay, so why are they good for big jobs?
[25:28] Why does Spark have shuffles in the first place, right? Um, one way to think about it is we schedule stages with blocking
[25:41] shuffles in between to give a way for fault tolerance to take place. Right? If you have long pipelines with meantime between failures of
[25:53] transient failures, the chance of some transient flake or failure happening just goes up really high as the job gets big and runs long and has a lot of tasks, right? Um and so the idea is we
[26:05] shuffle to disk between stages because when such a flake happens or transient failure happens on a task, we can just retry it, right?
[26:17] Either scanning from its original source files if it was a leaf stage, right? resscanning the parquet files again for example or for a non-leaf stage um the
[26:29] data that task is consuming is saved and we can just we can just scan it again that's the biggest kind of like reason why um we have shuffles today
[26:42] and they also give um adaptive execution opportunities at the shuffle boundaries as well so you can look at all the metadata at the shuffle boundary ask how big it is histograms, hotkeys,
[26:54] anything you want and make adaptive decisions on what to do, which is actually very important and powerful. Um, easy like an easy example for that. If there's nothing, there's no data, that's
[27:07] an empty relation, right? And you can make optimizations like not continuing after that. Um, so that's kind of like an easier way to understand it. This AQE feature is very powerful and blocking
[27:20] shuffles give us that. However, for small queries on your laptop, shuffling all of these rows to disk and then reading them back again can take up
[27:34] some time. And in some cases, it can take up more than half of the time of running the query.
[27:46] Right? So even though you're in one JVM, you're still paying for serializing the data after you're doing the useful work of the operators like aggregating you, filtering, whatever. You're like, okay, I'm at a stage boundary. I'm going to pay for serializing data, writing to
[27:59] disk, reading from disk, deserializing. Um, and that's kind of takes up time, right? And so also this task runtime skew
[28:14] is a pretty big part of this as well, right? Um the all these tasks in the stage can have different run times even though we endeavor for them to have roughly the same amount of work, right?
[28:26] We might say scan 100 files. I'm going to make 100 tasks. Each task scans one file. It should be roughly the same amount of time for each one to finish.
[28:38] If you have a big enough cluster or enough threads on your local machine, start them all at the same time. They should finish all the same time. Awesome. We have parallelism. But sometimes, actually, often times,
[28:51] one of those tasks is a straggler and it runs for really long and you have to wait for that guy to finish before the stage is considered done. Everyone's waiting. It can really drag drag out the
[29:03] whole query time. So that's called skew, right? And here we're asking whether we can do a faster way when the data is small and we're running on our local machine.
[29:16] So what we're proposing is multi-threaded execution um which is a new way of executing tasks and stages specifically and only for local mode executions right because
[29:28] there are trade-offs involved here that I'll mention um but for small data queries on a local machine it can it can be better right so the idea is in where we today
[29:42] we have disk based shuffle boundaries and we serialize to disk. Instead, we'll have threads that communicate via channels. Right? So, instead of
[29:54] task running writing to disk and the next task reading from disk and the next task running, the execution model is a thread runs the same operators, writes its output data to a channel, next
[30:08] thread runs, reads from the channel. So, we skip the disk access, right? But the operator semantics are the same, right? So we use these FIFO channels and
[30:21] they replace the shuffle files. Um whenever a thread has data that it's ready to generate for the next um next stage or
[30:33] the next task then it can write to the channel and um whenever thread wants to read from the channel it can just consume from it.
[30:45] So yeah, so this sounds great. Um, but disks are big and you can write a lot of things, but what happens if
[30:59] you have a lot of intermediate data and memory is not big and you fill up your RAM and crash? That's bad. Let's not do that. Um, instead let's make a limit on the amount of
[31:12] memory these channels can have um consumed inside them to kind of control our resource usage. And the way we implement that limit is via back
[31:24] pressure, right? So we'll slow down or stop the producers when we're getting full. Right? So as as you fill up a channel, the producer thread fills up a channel. Memory usage increases. When it gets
[31:38] full, we'll stop the producer. It can block the piece of code that says, "Okay, write this little block to the thread channel." We'll just wait until there's space. And
[31:52] um it can wake up when the downstream thread that wants to read from the channel has consumed something and produ and generated some space for it and it'll issue a wakeup call, right? And so
[32:05] the idea is if this is rare, everyone can just work all the threads can just generate data in the channels producers and then the consumers can just start immediately and consume
[32:17] and no one's blocking, right? everyone can just run. So here some kind of some of the classes that we're um modeling this with, right? There's a channel which is like the FIFO Q for data exchange. There's a channel
[32:31] gate which manages multiple channels. Senders and receivers are classes that can write to the channels async, right? So you say here's a block of data sender. Send immediately returns so I
[32:43] can do more work or it can block if it's full, right? And then there's the future versions of that kind of for convenience, right? So, um, what this buys us, it
[32:58] buys us two things. Um, the ability to skip the overhead of writing to disk and reading from disk is a big one.
[33:11] Um the another perhaps equally as big benefit that we can get is pipelining, right? And with this model, we don't have to
[33:23] finish all of the threads scanning the files in the leaf stage before we can start the next stage. We can just immediately begin as soon as
[33:35] as soon as those leaf threads have produced a little bit of data for their channel. The next thread can start consuming it and go, right? This is actually a big deal because
[33:47] um we can be spending a lot of time today for non-leaf stages waiting to begin, right? And they can just begin fairly soon now, right? For example, you can have a hash join that
[34:00] um on the non-leaf stage and says, "Okay, I want to join two scans. As soon as the build side thread has produced data from the scan,
[34:13] the hash join can start building its hash table, it can just start, right? It can start consuming the data and then doing its job. And this can oftentimes overlap completely hide this because the
[34:26] scan might be the bottleneck. Right? So you can basically hide all of this time building the hash table. As soon as the scan is done, the hash table's ready to go. Right? So that's an example of what we can do.
[34:39] Um right. So in contrast with shuffles that block um you have to wait for all of the tasks in the leaf stage to finish before you can start any work on the
[34:52] next stage. You have to wait for all of the stragglers to finish. Right? So even if 99 of the 100 tasks are done in like 3 seconds, if one of the tasks is taking 5
[35:06] minutes, you have to wait five minutes before anything can continue.
[35:18] than skipping the overhead of writing to disk and back. Um, so we can just kind of like everything can run as fast as it's able and as long as we have enough threads. Um, then
[35:31] we can we can just start the whole thing. Cool. So just a caveat for this, we need to be careful um and only use this when we know it's going to be appropriate. Um, only for small queries,
[35:46] only for local mode queries. Um, if we make if we use this when it's not appropriate, it can be disastrously bad, right? Like if you try to run a 40
[35:59] minute pedibyte scale pipeline with threading, then your transient failures can flake out and kill the whole kill the whole job 10 minutes into the job.
[36:12] That's like a basically an outage, right? When you have failures like that with this multi-threaded mode, you have to restart the whole thing. You restart the whole query, which is fine.
[36:24] Honestly, in most cases is actually fine. Restart the whole thing. Percentage of the time the failures happen low enough that you get away with it. But super important that we make this decision kind of carefully.
[36:39] So, we need to um be very conservative and deliberate about when theuler decides to use this. Cool. Yeah. And like I mentioned, um
[36:52] when the jobs get long, um we need to use shuffling and fall tolerance again. Cool. So, thank you very much. Um yeah so in some spark is
[37:04] better when we have a good means to run big jobs like spark has done so far but also we can make it fast on laptops and prototyping
[37:16] and uh make spark better for everyone. Cool. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.