What is Apache Spark?

Summary

• Apache Spark is an open-source, multi-language engine for data engineering, data science and machine learning on a single machine or a cluster. It's developed at the Apache Software Foundation under the Apache License 2.0.
• A Spark application is a driver program plus executors. The driver turns your code into jobs, stages and tasks, and a cluster manager (Spark standalone, YARN or Kubernetes) supplies the executors that run them.
• Most Spark code uses DataFrames or SQL. Transformations are lazy, so Spark sees the whole computation and optimizes it before an action runs it.
• One engine covers batch ETL, SQL analytics, stream processing with Structured Streaming, machine learning with MLlib, and remote clients through Spark Connect. As of October 2026 the newest feature release is Spark 4.2.0.

What is Apache Spark?

Apache Spark is an open-source engine for processing data at scale. In the project's words, it's "a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters." You write DataFrame, SQL or streaming code in Python, SQL, Scala or Java, and Spark plans that work and runs it in parallel across a set of executor processes. It started at UC Berkeley's AMPLab in 2009, was open sourced in early 2010 and moved to the Apache Software Foundation in 2013, where it's still developed in the open.

How Spark runs your code

Every Spark application has a driver program, the process that runs your main() function and creates the SparkContext. The driver connects to a cluster manager, which allocates resources; Spark ships with a standalone manager and also runs on Hadoop YARN and Kubernetes. Spark then acquires executors on worker nodes. Executors run tasks and keep data in memory or on disk between them, and each application gets its own executors, which isolates applications from one another.
Work is organized in three levels. An action, such as writing or collecting results, spawns a job. Each job splits into stages that depend on each other, and each stage is a set of tasks, one per data partition, sent to executors. Transformations such as filters, joins and aggregations are lazy: Spark records them and computes nothing until an action needs a result. That design lets Spark optimize the whole plan before running any of it.

# pip install pyspark
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders").getOrCreate()

orders = spark.read.parquet("data/orders")
daily = (orders.where(F.col("status") == "shipped")    # transformations build a plan
               .groupBy("order_date")
               .agg(F.sum("amount").alias("revenue")))
daily.write.mode("overwrite").parquet("out/daily")     # the action runs the job

Core concepts

Concept	What it is	Why it matters
Driver	The process that runs your program and schedules tasks	It must stay reachable by executors for the life of the application
Executor	A process on a worker node that runs tasks and caches data	Parallelism and memory come from executors
Cluster manager	Standalone, YARN or Kubernetes	Spark doesn't depend on any one of them
Job, stage, task	An action's computation, its dependent steps, and the per-partition units of work	This is the vocabulary of the Spark UI and of performance tuning
DataFrame	A distributed dataset organized into named columns	The main API in Python, Scala, Java and R; Spark SQL optimizes it
Lazy evaluation	Transformations build a plan; actions execute it	Spark can optimize the entire query before running it

What's in the project

• Spark SQL and DataFrames, the module for structured data processing, which accepts SQL and DataFrame code alike.
• pandas API on Spark, for pandas-style code on distributed data.
• Structured Streaming, which runs streaming queries on the same engine as batch, by default as a series of small batch jobs with exactly-once guarantees.
• MLlib for machine learning and GraphX for graph processing.
• Spark Connect, a client-server architecture added in Spark 3.4 that sends unresolved logical plans from a thin client to a remote Spark cluster.
• Spark Declarative Pipelines (SDP), added in Spark 4.1.0: you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries.

Spark 4.2.0 runs on Java 17, 21 or 25, Scala 2.13 and Python 3.10 or later. The R API, SparkR, has been deprecated since Spark 4.0.0.

How Databricks approaches Apache Spark

Databricks was founded by the original creators of Apache Spark and continues to contribute to it. Apache Spark is the technology behind Databricks compute clusters and SQL warehouses, and the Databricks Runtime adds optimizations and proprietary features on top, including Photon, a Databricks-native vectorized engine that's compatible with Spark APIs. The platform creates and manages the Spark session for you. Databricks Connect, built on open-source Spark Connect, lets IDEs and custom applications run code on Databricks compute, and Lakeflow Pipelines extend and interoperate with Spark Declarative Pipelines. See Apache Spark on Databricks.

FAQs

Is Apache Spark free to use?

Yes. Spark is open source under the Apache License 2.0, and you can install PySpark with pip install pyspark or run the official Docker image.

Do I need a cluster to run Spark?

No. Spark runs locally on one machine with Java installed, using a local or local[N] master URL, and the same program can later run on a cluster.

Is Spark only for batch processing?

No. Structured Streaming expresses streaming computations the same way as batch queries, and the engine runs them incrementally as new data arrives.

How is Spark different from a table format such as Delta Lake?

Spark is the compute engine. Delta Lake and Apache Iceberg are table formats that define how data and metadata are stored, and Spark reads and writes them through their Spark integrations.