Why isn't my materialized view refreshing?

Summary

• In open-source Spark Declarative Pipelines (SDP), a pipeline execution refreshes a materialized view. spark-pipelines run launches an execution and monitors it until it completes, so first confirm that something launches it.
• spark-pipelines dry-run doesn't read or write any data. It catches many kinds of errors, such as syntax errors, analysis errors and graph validation errors like cyclic dependencies.
• With no refresh options, spark-pipelines run performs a default incremental update. --refresh updates named datasets without clearing existing data, and --full-refresh clears existing data and checkpoints before recomputing.
• On Databricks, pipeline updates can be scheduled through jobs, standalone materialized views support SCHEDULE and TRIGGER ON UPDATE, and a refresh can report "No change" when no source changes were detected.

Why isn't my materialized view refreshing?

In open-source Spark Declarative Pipelines (SDP), a materialized view is a view precomputed into a table, and it always has exactly one batch flow writing to it. A pipeline execution refreshes it. spark-pipelines run launches an execution and monitors its progress until it completes. So start with two questions: did an execution actually run, and did it finish without errors?
In SQL, the guide defines one like this. In Python, the @dp.materialized_view decorator wraps a function that performs a batch read.

CREATE MATERIALIZED VIEW basic_mv
AS SELECT * FROM samples.nyctaxi.trips;

How does refresh work in open-source SDP?

The Spark 4.2 programming guide documents these commands.

Command	What the Spark docs say it does
spark-pipelines run	Launches an execution of the pipeline. With no refresh options, a default incremental update is performed
spark-pipelines run --refresh a,b	Triggers an update for the listed datasets without clearing existing data
spark-pipelines run --full-refresh a,b	Resets the listed datasets: clears all existing data and checkpoints, then recomputes them from scratch
spark-pipelines run --full-refresh-all	Full graph reset and recompute, equivalent to --full-refresh for every dataset
spark-pipelines dry-run	Launches an execution that doesn't write or read any data but catches many kinds of errors

--full-refresh-all cannot be combined with --full-refresh or --refresh. You can combine --full-refresh and --refresh to give different datasets different behavior.

What should I check when a refresh doesn't happen?

Situation	What to check
The view never changes	Nothing launched an execution. The guide calls the spark-pipelines CLI the primary way to manage a pipeline and documents no built-in scheduler, so invoke spark-pipelines run from your own scheduler
A run fails	A syntax error, an analysis error (a table or column doesn't exist) or a graph validation error such as a cyclic dependency. Run spark-pipelines dry-run and fix what it reports
Python definitions have side effects	SDP evaluates the code that defines a pipeline multiple times during planning and runs. Keep dataset functions to the definition, return a DataFrame, and don't call collect(), count(), toPandas(), save(), saveAsTable(), start() or toTable()
You want a clean rebuild	Run spark-pipelines run --full-refresh my_view, which discards the dataset's existing data and checkpoints

How does Databricks approach materialized view refresh?

In Lakeflow Pipelines, a pipeline update starts a cluster, validates your source code and refreshes the tables and views in the pipeline. You can trigger an update manually, on a schedule through jobs, or programmatically. Updates started from Jobs or the Pipelines API use automatic retry and restart behavior, which retries execution on specific errors such as a failure to start a cluster.
A standalone materialized view created with CREATE MATERIALIZED VIEW can be refreshed manually with REFRESH MATERIALIZED VIEW, through a SCHEDULE or TRIGGER ON UPDATE clause, or from a SQL task in a job. The Databricks docs show this schedule:

CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.hourly_metrics
  SCHEDULE EVERY 1 HOUR
AS SELECT
  date_trunc('hour', event_time) AS hour,
  count(*) AS events
FROM catalog.schema.raw_events
GROUP BY 1;

A default refresh attempts an incremental refresh, and Databricks uses a cost model to choose between that and a full recompute. A refresh can also finish without changing anything: the pipeline's Tables panel lists a "No change" status when no source data changes were detected, so the materialized view was not updated. Incremental refresh runs only on serverless pipelines. On classic compute, a materialized view is always fully recomputed. To see what a refresh did, query the event log, where a NO_OP technique means no changes to the base table were detected:

SELECT timestamp, message
FROM event_log(TABLE(<fully-qualified-table-name>))
WHERE event_type = 'planning_information'
ORDER BY timestamp desc;

FAQs

Does a full refresh delete data?

In open-source SDP, --full-refresh clears all existing data and checkpoints for the datasets you name, then recomputes them from scratch. On Databricks, a full refresh of a materialized view recomputes the entire result and produces the same data as a default refresh. Databricks recommends running one only when necessary.

Can I refresh just one materialized view?

In open-source SDP, list it in --refresh, as in spark-pipelines run --refresh my_view. For a standalone Databricks materialized view, run REFRESH MATERIALIZED VIEW with its name.

Who can refresh a Databricks materialized view?

A user needs USE CATALOG and USE SCHEMA, plus ownership of the materialized view or the REFRESH privilege. The owner must have SELECT on the base tables.