What is Tungsten in Spark?

Summary

• Tungsten is the Apache Spark project that reworked the engine behind Spark SQL and DataFrames for CPU and memory efficiency, through explicit memory management on binary data, cache-aware data structures and runtime code generation.
• Its first pieces shipped in Spark 1.4 and it became the default execution path in Spark 1.5. Since Spark 1.6 there is no off switch: Tungsten mode and code generation are always enabled.
• Spark 2.0 added whole-stage code generation, which collapses a fragment of a query plan into a single generated function where it can, and vectorized, batch-at-a-time processing where it can't.
• You don't call Tungsten directly. SQL and DataFrame code runs on the same engine whatever the language, and EXPLAIN CODEGEN or df.explain(mode="codegen") prints the generated code.
• Photon, on Databricks, is a separate native C++ vectorized engine that replaces JVM-based execution for supported operations and falls back to the Spark runtime for the rest.

What is Tungsten in Spark?

Tungsten is the name of the project that rebuilt Apache Spark's execution engine to use CPU and memory more efficiently. The 2015 Databricks engineering post that announced it set three initiatives: manage memory explicitly and run operations on binary data instead of Java objects, design algorithms and data structures around the CPU memory hierarchy, and generate code at runtime. Tungsten isn't a library you import or a setting you turn on. It's how Spark SQL and the DataFrame API execute queries, and it has been always on since Spark 1.6.

What does Tungsten change inside Spark?

Concept	What it is	Why it matters
Explicit memory management	Spark manages memory itself and runs most operations directly against binary data instead of Java objects	Less JVM object overhead and less garbage-collection work
Cache-aware computation	Algorithms and data structures built to use the L1, L2 and L3 CPU caches, such as the cache-friendly hash map layout used for aggregation	Less CPU time spent waiting for data from main memory
Code generation	Spark emits optimized bytecode at runtime for the query instead of walking a generic tree of operators	Avoids a virtual function call per row and keeps intermediate data in CPU registers
Whole-stage code generation	Since Spark 2.0, each fragment of the plan is collapsed into one generated function where possible	The fragment runs as one function instead of a chain of operator calls
Vectorization	Where code can't be fused, operators process batches of rows in a columnar format	Spreads the cost of each call across a whole batch

How did Tungsten evolve across Spark releases?

Release	What the release notes say
Spark 1.4.0	"Initial performance improvements in project Tungsten" (SPARK-7081)
Spark 1.5.0	Optimized execution using manually managed memory (Tungsten) is enabled by default, along with code generation for expression evaluation
Spark 1.6.0	The spark.sql.tungsten.enabled flag is removed; "Tungsten mode and code generation are always enabled" (SPARK-11644)
Spark 2.0.0	Whole-stage code generation for common SQL and DataFrame operators, plus vectorized Parquet scans

Before whole-stage code generation, Spark 1.x evaluated queries with the iterator model, commonly called the Volcano model: each operator exposes a next() call that hands one row at a time to its parent. The Spark 2.0 engineering write-up calls the new design the second-generation Tungsten engine.

How can you see Tungsten at work?

df = spark.sql(
    "SELECT k, sum(v) FROM VALUES (1, 2), (1, 3) t(k, v) GROUP BY k"
)
df.explain(mode="formatted")  # operators marked * carry a codegen id
df.explain(mode="codegen")    # physical plan plus the generated code

In formatted output, operators that run inside generated code are marked with an asterisk and list a codegen id. A shuffle Exchange sits between two generated fragments and has neither. In SQL, EXPLAIN CODEGEN <statement> prints the generated code for a statement. The Spark UI's stage DAG labels the same fragments WholeStageCodegen and annotates each one with its code generation id.

Where does generated code stop?

Generated code covers the JVM side of a query. When a query calls a Python UDF, the Python code runs outside the generated code, and Spark moves data between the JVM and Python processes. Apache Arrow can make that transfer efficient, though the PySpark docs note its use isn't automatic. The Spark 2.0 write-up names calls out to Python or R, and operations such as CSV parsing or Parquet decoding, as cases where code can't be fused into one function. That's where vectorization takes over.

How Databricks approaches query execution

Photon is the Databricks-native vectorized query engine. For supported operations it replaces the JVM-based Spark SQL execution engine with a native C++ runtime. Catalyst, Spark's query optimizer, still plans the query, and Photon executes it in columnar batches. When Photon reaches an unsupported operation, it falls back to the Spark runtime, and existing Spark API code runs without changes. Photon is enabled on serverless compute, SQL warehouses and serverless Lakeflow Pipelines. On classic compute it's on by default and controlled by the Use Photon Acceleration checkbox; if you create compute through the Clusters or Jobs API, set runtime_engine to PHOTON.

FAQs

Do I need to enable Tungsten?

No. Since Spark 1.6, Tungsten mode and code generation are always enabled.

Can I turn Tungsten off?

Not with a supported setting. Spark 1.6 removed the spark.sql.tungsten.enabled flag, and the current configuration reference lists no Tungsten switch.

Does Tungsten apply to PySpark?

Yes, for DataFrame and SQL operations. Spark SQL uses the same execution engine regardless of the API or language you write the query in. Python UDF code is the exception, because it runs outside the generated code.

Is Tungsten the same as Catalyst?

No. Catalyst is Spark's query optimizer, which plans the query. Tungsten is the execution side: memory management, cache-aware data structures and code generation.

How does Photon relate to Tungsten?

They're separate engines. Tungsten is part of open-source Spark's JVM execution engine, while Photon is Databricks' native C++ engine for supported operations.