How do I define a materialized view in SDP?

Summary

• In Spark Declarative Pipelines (SDP), you define a materialized view with the @dp.materialized_view decorator in Python or CREATE MATERIALIZED VIEW ... AS SELECT in SQL.
• The Python function performs a batch read and returns a DataFrame. The view takes the function's name unless you pass name.
• The Spark guide describes a materialized view as "a view that is precomputed into a table" with exactly one batch flow writing to it.
• SDP analyzes the dependencies between your datasets and orders and parallelizes execution; you run the pipeline with spark-pipelines run.
• On Databricks, Lakeflow Pipelines extend SDP, publish materialized views to Unity Catalog, and can refresh them incrementally on serverless pipelines.

How do I define a materialized view in SDP?

In Spark Declarative Pipelines (SDP), you define a materialized view with the @dp.materialized_view decorator on a Python function that performs a batch read and returns a DataFrame, or with CREATE MATERIALIZED VIEW ... AS SELECT in a SQL file. The Spark guide describes it as "a view that is precomputed into a table" that "always has exactly one batch flow writing to it." SDP was added in Apache Spark 4.1.0, and pip install pyspark[pipelines] installs it.

How is a materialized view different from the other SDP datasets?

SDP has three dataset types. The choice depends on how the data is read and where the result is needed.

Dataset	Python	SQL	Per the Spark guide
Streaming table	@dp.table with spark.readStream	CREATE STREAMING TABLE ... AS SELECT ... FROM STREAM ...	A table with one or more streaming flows, for incremental processing of new data
Materialized view	@dp.materialized_view	CREATE MATERIALIZED VIEW ... AS SELECT ...	A view precomputed into a table, with exactly one batch flow
Temporary view	@dp.temporary_view	CREATE TEMPORARY VIEW ... AS SELECT ...	A view scoped to one execution of the pipeline, readable by flows in that pipeline

How do I write one in Python?

Import the module as dp, then decorate a function that returns a DataFrame. The view takes the function's name unless you pass name:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.materialized_view(name="trips_mv")
def basic_mv() -> DataFrame:
    return spark.table("samples.nyctaxi.trips")

Spark 4.2 injects spark into each pipeline file. On Spark 4.1, start the file with spark = SparkSession.active(). The guide calls that the only supported way to assign the session, because SparkSession.builder.config(...).getOrCreate() mutates session config, which SDP blocks.

How do I write one in SQL?

CREATE MATERIALIZED VIEW basic_mv
AS SELECT * FROM samples.nyctaxi.trips;

How do materialized views reference each other?

Refer to other datasets the way you would refer to any table. While the pipeline runs, SDP analyzes the dependencies and orders and parallelizes execution. This example assumes orders and customers are defined elsewhere in the pipeline:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame
from pyspark.sql.functions import col

@dp.materialized_view
def customer_orders() -> DataFrame:
    return (
        spark.table("orders")
        .join(spark.table("customers"), "customer_id")
        .select("customer_id", "state",
                col("order_datetime").cast("date").alias("order_date"))
    )

@dp.materialized_view
def daily_orders_by_state() -> DataFrame:
    return (
        spark.table("customer_orders")
        .groupBy("state", "order_date")
        .count()
        .withColumnRenamed("count", "order_count")
    )

What rules apply to the function body?

SDP evaluates the code that defines a pipeline multiple times during planning and runs, so dataset functions should hold only the code needed to define the view. The function must return a pyspark.sql.DataFrame, and it must never save or write data. The guide names collect(), toPandas(), save(), saveAsTable(), start() and toTable() among the operations to avoid. In SDP SQL, the PIVOT clause isn't supported.

How do I run and refresh the pipeline?

A pipeline project has a spec file, conventionally spark-pipeline.yml, with required name, libraries and storage fields plus optional catalog, database and configuration. spark-pipelines init --name my_pipeline generates a starter project.

Command	What it does
spark-pipelines dry-run	Reads and writes no data, but catches syntax errors, analysis errors and cyclic dependencies
spark-pipelines run	Runs the pipeline; with no refresh options it performs a default incremental update
spark-pipelines run --refresh daily_orders_by_state	Updates the listed datasets without clearing existing data
spark-pipelines run --full-refresh daily_orders_by_state	Clears the listed datasets' data and checkpoints, then recomputes them from scratch

How does Databricks approach materialized views?

Lakeflow Pipelines extend Apache Spark Declarative Pipelines and run on the Databricks Runtime. There, @dp.materialized_view accepts more arguments, such as comment, table_properties, partition_cols, cluster_by and schema. A pipeline configured with Unity Catalog publishes each materialized view to the catalog and schema you choose, and by default Databricks stores every streaming table and materialized view a pipeline creates as a Delta table. Creating one takes USE CATALOG on the catalog plus CREATE MATERIALIZED VIEW and USE SCHEMA on the schema, and by default only the pipeline owner can query the result until you grant access.
When the pipeline is triggered, Databricks keeps the view up to date, often by processing only the changed data. Incremental refresh is available only on serverless pipelines. On classic compute, a materialized view is always fully recomputed. The docs also say materialized views aren't designed for low-latency use cases.

FAQs

When should I use a materialized view instead of a streaming table?

Use a materialized view when the query performs aggregations or joins, or when source data changes through updates and deletes rather than only growing. Databricks docs describe a streaming table as designed for append-only sources, processing each record once.

Can a materialized view read a streaming source?

No. It comes from a batch read. Define a streaming table with @dp.table and spark.readStream for streaming sources.

Can other queries read a materialized view?

On Databricks, yes. A materialized view is a Unity Catalog managed table, so consumers outside the pipeline can query it. A temporary view can't be read outside the pipeline.