What is lazy evaluation in Spark?

Summary

• Lazy evaluation means Spark transformations do not compute their results right away. Spark remembers them and runs them when an action needs a result.
• Transformations such as map, filter and join are lazy. Actions such as collect, count, take and saveAsTextFile spawn a job.
• Seeing the chain of operations first lets Spark avoid work, such as returning only a reduce result to the driver instead of the larger mapped dataset.
• A transformed RDD may be recomputed on every action unless you persist or cache it, and accumulator updates inside transformations are not guaranteed to run.
• Spark Connect keeps the same lazy execution model and also defers analysis. SQL queries are lazy, while SQL commands such as insert run eagerly.

What is lazy evaluation in Spark?

Lazy evaluation means Spark delays work until a result is needed. The RDD programming guide says: "All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program."

lines = sc.textFile("data.txt")
lineLengths = lines.map(lambda s: len(s))
totalLength = lineLengths.reduce(lambda a, b: a + b)

In the guide's walkthrough, lines "is merely a pointer to the file" and lineLengths "is not immediately computed, due to laziness". Only reduce, an action, makes Spark break the computation into tasks. The same model applies to DataFrames: transformations "are encoded in a logical plan". The docs add: "The actual computation is triggered only when an action (e.g., show(), collect()) is triggered."

Which operations are transformations, and which are actions?

Kind	When it runs	Examples from the docs
RDD transformation	Not until an action needs the result	map, filter, flatMap, reduceByKey, join
RDD action	Immediately, by running a computation on the dataset	reduce, collect, count, first, take, saveAsTextFile, foreach
DataFrame transformation	Not until an action	df.filter(...), df.select(...), df.limit(...), spark.sql("select …")
DataFrame action	Immediately	df.collect(), df.show(), count, write
SQL command	Immediately	spark.sql("insert …"), spark.sql("create …")

The Spark glossary defines a job: "A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect)".

Why is Spark lazy?

• Less work. The RDD guide's example: Spark can "realize that a dataset created through map will be used in a reduce and return only the result of the reduce to the driver, rather than the larger mapped dataset".
• Optimization. The Databricks docs say that "Rather than evaluating each transformation in the exact order specified, Spark waits until an action triggers computation on all transformations." Spark SQL also gets "more information about the structure of both the data and the computation being performed" and uses it for optimizations. One example is Parquet filter push-down, on by default through spark.sql.parquet.filterPushdown.

What does it look like with DataFrames?

df = spark.read.parquet("people.parquet")
counts = df.filter(df.age > 3).groupBy("name").count()

counts.explain()
counts.show()

The first two lines are transformations. The action, counts.show(), triggers the query. The docs describe explain() this way: "Prints the (logical and physical) plans to the console for debugging purposes." With explain(extended=True) you get the parsed, analyzed and optimized logical plans plus the physical plan.

What are the common gotchas?

Symptom	Cause	Fix
An accumulator is still 0 after a map	"accumulator updates are not guaranteed to be executed when made within a lazy transformation like map()"	Update accumulators inside actions such as foreach
The same transformation runs again on every action	"each transformed RDD may be recomputed each time you run an action on it"	persist() or cache() data you reuse
collect() strains the driver	It returns "all the elements of the dataset as an array at the driver program"	Filter first, or use take(n)
A wrong column name raises no error on Spark Connect	Spark Connect defers analysis	Trigger analysis with df.columns, df.schema or df.collect()
count() or show() fails on a streaming DataFrame	They are "actions that will immediately run queries and return results"	Use ds.groupBy().count() or the console sink, then start()

Both Spark Classic and Spark Connect "follow the same lazy execution model for query execution". Only Spark Connect defers analysis: spark.sql("select 1 as a, 2 as b").filter("c > 1") fails immediately in Classic, but in Spark Connect the error appears at df.columns or df.show().

How does Databricks handle lazy evaluation?

The Databricks docs state: "Apache Spark uses lazy evaluation to process transformations and actions defined with DataFrames." They list spark.read and spark.table among the transformations, and display, show, take(n), first, head, saveAsTable and count among the actions. They add that "In production data pipelines, writing data is typically the only action that should be present. All other actions interrupt query optimization and can lead to bottlenecks." To check an intermediate step, "call an action".
To inspect jobs, select your compute on the Compute page and open the Spark UI tab, where the job details page shows a DAG visualization. RDD APIs are not supported on serverless or standard compute, so use DataFrames there.

FAQs

Why isn't my accumulator updating?

Accumulators "do not change the lazy evaluation model of Spark". A value updated inside a transformation changes only "once that RDD is computed as part of an action". For updates performed inside actions only, Spark applies each task's update once. In transformations, an update "may be applied more than once if tasks or job stages are re-executed".

Can I force evaluation?

Yes, call an action such as count() or take(n). collect() is "usually useful after a filter or other operation that returns a sufficiently small subset of the data".

How does lazy evaluation affect streaming?

The streaming guide says a streaming DataFrame "is not currently receiving any data as we are just setting up the transformation, and have not yet started it". You then "start the streaming computation using start()".