what are the capabilities of Spark?

Summary

• Apache Spark is a multi-language engine for data engineering, data science and machine learning on single-node machines or clusters, with APIs in Python, SQL, Scala, Java and R.
• Its core components are Spark SQL and DataFrames, Structured Streaming, MLlib and the pandas API on Spark, all running on one optimized engine.
• Spark 4.1.0 added Spark Declarative Pipelines and Structured Streaming Real-Time Mode. Spark 4.2.0 added native geospatial types and a SQL CHANGES clause for reading row-level changes.
• On Databricks, Apache Spark powers compute clusters and SQL warehouses. Lakeflow Pipelines build on Spark Declarative Pipelines, and the Databricks documentation lists AUTO CDC, expectations, continuous mode and a queryable event log as additions.

what are the capabilities of Spark?

Apache Spark is a multi-language engine for executing data engineering, data science and machine learning on single-node machines or clusters. One engine covers batch and streaming processing, distributed ANSI SQL, DataFrame programming, machine learning and, since Spark 4.1, declarative pipelines. You can run it locally on one machine, or on Spark Standalone, Hadoop YARN or Kubernetes.

What is Apache Spark?

Spark's core abstraction is the resilient distributed dataset (RDD), a fault-tolerant collection of elements that can be operated on in parallel. Spark SQL's interfaces tell Spark more about the structure of the data and the computation, which it uses for extra optimizations. A DataFrame is a Dataset organized into named columns. The Dataset API is available in Scala and Java but not in Python.

Which capabilities does Spark provide?

Capability	What Spark provides
SQL and DataFrames	ANSI SQL plus DataFrame APIs in Python, Scala, Java and R, over structured tables and unstructured data such as JSON or images.
Adaptive Query Execution	On by default since Spark 3.2.0. It re-optimizes the plan with runtime statistics, for example by coalescing shuffle partitions, converting a sort-merge join to a broadcast join and splitting skewed partitions.
Structured Streaming	A scalable, fault-tolerant stream processing engine built on the Spark SQL engine. You express a streaming computation like a batch one, and Spark runs it incrementally and continuously.
Real-Time Mode	Added in 4.1.0 for continuous, sub-second latency streaming queries. The 4.1.0 notes list Scala stateless support, and 4.2.0 adds an RTM trigger for PySpark.
MLlib	Classification, regression, clustering and collaborative filtering algorithms, plus featurization, ML Pipelines and model persistence.
pandas API on Spark	Runs pandas workloads on Spark.
Spark Connect	A client-server architecture, introduced in Spark 3.4, that decouples client applications and allows remote connectivity to Spark clusters.
Spark Declarative Pipelines (SDP)	Added in 4.1.0. You define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries.

What does a Spark Declarative Pipeline look like?

You declare datasets instead of scripting jobs. These two statements come from the Spark programming guide:

CREATE STREAMING TABLE basic_st
AS SELECT * FROM STREAM samples.nyctaxi.trips;

CREATE MATERIALIZED VIEW basic_mv
AS SELECT * FROM samples.nyctaxi.trips;

A streaming table processes only new data as it arrives, and a materialized view always has exactly one batch flow writing to it. In Python, the same pipelines use from pyspark import pipelines as dp with decorators such as @dp.table and @dp.materialized_view.

How does Databricks approach Spark?

Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks. Lakeflow Pipelines are built on Spark Declarative Pipelines, run on the Databricks Runtime and share the same authoring model. The Databricks documentation describes these capabilities:

• AUTO CDC computes slowly changing dimensions (SCD) Type 1 and Type 2 from a change feed or database snapshots.
• Expectations validate each record. The action on an invalid record is warn (the default, which writes the record), drop, or fail the update.
• Pipeline modes. Triggered mode refreshes available data and stops. Continuous mode keeps tables fresh as data arrives.
• Incremental refresh recomputes only the affected results of a materialized view on a serverless pipeline. Databricks runs a cost analysis to choose between an incremental refresh and a full recompute.
• Automatic orchestration runs flows in the correct order with maximum parallelism and retries transient failures progressively, from the Spark task to the flow to the entire pipeline.
• Serverless pipelines run on Databricks-managed compute, and Databricks recommends serverless for new pipelines. They always use Unity Catalog.
• Lakeflow Pipelines Editor is an IDE with a multi-file code editor, an interactive pipeline graph, data previews and version control.
• Auto Loader incrementally processes new files as they arrive in cloud storage, through a Structured Streaming source called cloudFiles.

Common pipeline use cases are batch ingestion from cloud storage, incremental ingestion from message buses such as Apache Kafka, and incremental batch and streaming transformations.

FAQs

How does Spark handle streaming and batch?

Structured Streaming lets you express a streaming computation the same way you would express a batch computation on static data. By default, queries run on a micro-batch engine that processes data streams as a series of small batch jobs.

What limits Spark performance, and what helps?

Data skew can severely downgrade the performance of join queries. The Spark SQL performance tuning guide covers caching tables in memory with spark.catalog.cacheTable("tableName") or dataFrame.cache(), join strategy hints such as BROADCAST, and Adaptive Query Execution, which splits skewed tasks into roughly evenly sized ones.

Does Spark need a cluster?

No. It's easy to run locally on one machine with Java installed. For clusters, Spark runs by itself in Standalone mode or over Hadoop YARN or Kubernetes.

How do Spark Declarative Pipelines relate to Lakeflow Pipelines?

Lakeflow Pipelines share the same declarative authoring model as SDP and add production features such as AUTO CDC, data quality expectations and a queryable event log. The Databricks documentation says the transformation code you write stays portable to other SDP runtimes.