How do I make downstream tables refresh automatically when upstream data changes?

Summary

• Materialized views and streaming tables declared with TRIGGER ON UPDATE refresh when their upstream tables change, so nobody has to coordinate schedules across pipelines.
• Materialized views and streaming tables can themselves be upstream sources, so triggered tables chain from raw data through each layer.
• Inside a Lakeflow Pipelines pipeline you don't declare an order: the pipeline analyzes dataset dependencies and orchestrates execution order and parallelism.
• A Lakeflow Jobs table update trigger starts a job, such as a pipeline refresh, when any or all monitored tables are updated.
• Unity Catalog lineage shows which downstream tables, jobs, and dashboards depend on a table.

How do I make downstream tables refresh automatically when upstream data changes?

Declare the downstream tables and let Databricks track what they read. A materialized view or streaming table with TRIGGER ON UPDATE refreshes when its sources change, a Lakeflow Pipelines pipeline runs its own datasets in dependency order, and a Lakeflow Jobs table update trigger starts any job when the tables it watches are updated. No more chaining jobs by hand.

Which mechanism fits your dependency?

Dependency	Use	What Databricks does
A table built from a few upstream tables	Materialized view or streaming table with TRIGGER ON UPDATE	Monitors the sources and refreshes when any of them changes
Many datasets in one codebase, such as bronze, silver, and gold layers	Lakeflow Pipelines	Analyzes dependencies and orchestrates execution order and parallelism
A pipeline or notebook that reads tables another team owns	Table update trigger on a Lakeflow Job	Starts a run when any or all monitored tables are updated
Steps that must run in order inside one job	Task dependencies	Runs upstream tasks first, in parallel where possible

How do triggered tables cascade?

The docs say TRIGGER ON UPDATE removes the need to coordinate schedules across pipelines: the dataset stays fresh without anyone knowing when upstream jobs finish. Its sources can be Delta tables, including other materialized views and streaming tables, so you can chain triggered tables. A change in the raw table refreshes the silver view, and the silver view's new data then triggers the gold view:

CREATE OR REPLACE MATERIALIZED VIEW main.silver.orders_clean
  TRIGGER ON UPDATE
AS SELECT
  order_id,
  customer_id,
  order_date,
  amount
FROM main.raw.orders
WHERE order_id IS NOT NULL;

CREATE OR REPLACE MATERIALIZED VIEW main.gold.daily_revenue
  TRIGGER ON UPDATE AT MOST EVERY INTERVAL 1 HOUR
AS SELECT
  order_date,
  sum(amount) AS total_revenue
FROM main.silver.orders_clean
GROUP BY
  order_date;

AT MOST EVERY throttles a layer whose sources change often. For a view that already exists, ALTER MATERIALIZED VIEW ... ADD TRIGGER ON UPDATE adds the trigger. A pipeline can monitor up to 10 upstream tables and 30 upstream views; for more, the docs advise splitting the logic across pipelines.

How do Lakeflow Pipelines order their own datasets?

While a pipeline runs, it analyzes the dependencies of the datasets you defined and orchestrates their execution order and parallelization, retrying transient failures. A triggered pipeline processes the data available when the update starts and then stops. A continuous pipeline keeps tables fresh as data arrives, and it updates only when the contents of the Delta tables it depends on have changed. Triggered mode can reduce resource consumption because compute runs only long enough to update the pipeline; continuous mode needs an always-running cluster but reduces processing latency. Databricks recommends running a continuous pipeline from a continuous job rather than through the pipeline's own continuous setting.

How do I refresh a pipeline when another team's table changes?

Give the pipeline's job a table update trigger. In Declarative Automation Bundles:

resources:
  jobs:
    refresh_on_update:
      name: refresh_on_update
      trigger:
        pause_status: UNPAUSED
        table_update:
          table_names:
            - main.raw.orders
            - main.raw.customers
          condition: ANY_UPDATED
          wait_after_last_change_seconds: 120
      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.my_pipeline.id}

ANY_UPDATED starts a run when either table changes; ALL_UPDATED waits for both. wait_after_last_change_seconds lets a burst of writes finish first, since each update resets the timer. Tasks can read {{job.trigger.table_update.updated_tables}}, a JSON list of the tables updated since the last run. The job's Run as principal needs SELECT on each monitored table, and one trigger covers up to 10 tables. To refresh a standalone materialized view from a job, use a SQL task that runs REFRESH MATERIALIZED VIEW.

FAQs

Does each refresh reprocess the whole table?

Not when an incremental refresh is possible. A materialized view can identify changes since the last update and merge only new or modified data, and a cost model chooses between incremental and full refresh. To support incremental refresh, store source data in Delta tables with row tracking enabled.

Should a downstream table be a materialized view or a streaming table?

Use a materialized view when the query aggregates or joins, or when sources change through updates and deletes; it keeps results consistent with the current state of the source data. A streaming table processes each record once and suits append-only, continuously growing sources.

How do I find every table that depends on a source?

Use Unity Catalog lineage, which is captured automatically down to the column level. A table's Lineage tab in Catalog Explorer shows its downstream tables, jobs, and dashboards, and lineage is also recorded in the system.access.table_lineage system table.

Can I stop bad upstream data from flowing downstream?

Yes. Add an expectation to the downstream dataset, such as CONSTRAINT valid_id EXPECT (id IS NOT NULL) ON VIOLATION FAIL UPDATE. Invalid records then prevent the update from succeeding, while ON VIOLATION DROP ROW drops them. Expectations work in pipelines and on standalone materialized views and streaming tables.

How do I confirm the refreshes are firing?

Catalog Explorer shows refresh status and history for each materialized view or streaming table, and DESCRIBE TABLE EXTENDED returns its refresh information. For jobs, the trigger_type column in system.lakeflow.job_run_timeline shows what started each run.