How do I keep a table fresh without guessing a schedule?

Summary

• Event-driven refresh replaces a guessed schedule: on Databricks, a materialized view or streaming table declared with TRIGGER ON UPDATE refreshes when its upstream tables change.
• Lakeflow Jobs table update triggers start a run when source tables are updated, and file arrival triggers start one when new files land in a Unity Catalog volume or external location.
• AT MOST EVERY caps how often a triggered table refreshes, and "Wait after last change" on a job trigger holds the run until a batch of updates or files has finished arriving.
• The Databricks docs call TRIGGER ON UPDATE the recommended approach for production workloads, especially when upstream dependencies don't run on predictable schedules.
• The trigger_type column in system.lakeflow.job_run_timeline shows what started each job run.

How do I keep a table fresh without guessing a schedule?

Refresh on the data event instead of the clock. On Databricks, a materialized view or streaming table declared with TRIGGER ON UPDATE refreshes when its upstream tables change, and Lakeflow Jobs can start a job when source tables are updated or when new files land in cloud storage. Tables refresh when new data exists, and nothing runs while the sources sit unchanged.
A cron schedule encodes a guess about when upstream data arrives. When the producer runs late, the table stays stale until the next slot. When nothing changed, the run repeats work.

Which refresh options respond to data instead of time?

Option	Starts work when	Configure it in
TRIGGER ON UPDATE on a materialized view or streaming table	An upstream source table changes	The CREATE or ALTER statement in Databricks SQL
Table update trigger on a job	Any or all monitored tables are updated	Job trigger settings: UI, Jobs API, or Declarative Automation Bundles
File arrival trigger on a job	New files arrive in a Unity Catalog volume or external location	Job trigger settings
Continuous job	The previous run completes or fails, so the job keeps running	Job trigger settings

Databricks recommends continuous mode for always-on streaming workloads. The other three options start work only when there's new data.

How does TRIGGER ON UPDATE work?

The table's pipeline monitors its source tables and refreshes when any of them changes. The docs say this removes the need to coordinate schedules across pipelines. AT MOST EVERY caps how often it refreshes when sources change constantly:

CREATE OR REPLACE MATERIALIZED VIEW main.analytics.daily_sales
  TRIGGER ON UPDATE AT MOST EVERY INTERVAL 5 MINUTES
AS SELECT
  date,
  sum(sales) AS sum_of_sales
FROM main.raw.sales
GROUP BY
  date;

For an existing view, ALTER MATERIALIZED VIEW ... ADD TRIGGER ON UPDATE adds the trigger. A few rules apply:

• Sources must be Delta tables, managed or external (other materialized views and streaming tables count), or managed views over supported tables.
• A pipeline can monitor up to 10 upstream tables and 30 upstream views, and the minimum trigger interval is 1 minute.
• A definition takes either a SCHEDULE clause or a TRIGGER ON UPDATE clause, not both.
• Refreshes run on a serverless pipeline instead of your SQL warehouse, and they can be incremental when the query and source data allow it.

How do job triggers handle files and source tables?

For notebooks or multi-step jobs, attach a trigger to the job. A table update trigger runs the job when one or all monitored tables are updated, so it runs "when new data is ready without the need for a continuously running cluster," in the docs' words. It supports Unity Catalog managed Delta and Iceberg tables, external Delta tables, materialized views, streaming tables, and views over supported tables.
A file arrival trigger watches the root or a subpath of a Unity Catalog volume or external location, subdirectories included. Only new files start runs; overwriting a file with the same name doesn't. Pair it with Auto Loader, which processes new files incrementally with exactly-once guarantees. In Declarative Automation Bundles:

resources:
  jobs:
    load_incoming_files:
      name: load_incoming_files
      trigger:
        pause_status: UNPAUSED
        file_arrival:
          url: /Volumes/main/raw/incoming/
          min_time_between_triggers_seconds: 300
          wait_after_last_change_seconds: 60
      tasks:
        - task_key: load_files
          notebook_task:
            notebook_path: ../src/load_files.ipynb

wait_after_last_change_seconds holds the run until files stop landing, because each new file resets the timer. min_time_between_triggers_seconds caps how often runs start. Both accept a minimum of 60 seconds.

What should you set up before relying on triggers?

• Permissions: the job's Run as principal needs SELECT on every table a table update trigger monitors. File arrival triggers need READ on the storage location and CAN MANAGE on the job.
• File events: Databricks recommends enabling file events on the external location. Locations without them have lower limits on file counts and triggered jobs.
• Overlapping runs: a job allows one active run by default, and runs beyond the limit are skipped. Turn on queueing so they wait instead.
• Alerts: configure notifications on job failure to hear about triggers that fail to evaluate.

FAQs

How quickly does a refresh start after new data lands?

File arrival triggers make a best effort to check for new files every minute, and cloud storage performance can affect that. TRIGGER ON UPDATE refreshes at most once a minute by default, or on the longer interval you set with AT MOST EVERY.

How do I see what started each run?

Query the jobs system table:

SELECT job_id, run_id, trigger_type, result_state, period_start_time
FROM system.lakeflow.job_run_timeline
WHERE period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
  AND result_state IS NOT NULL;

Values include TABLE, FILE_ARRIVAL, CRON, PERIODIC, and ONETIME. For a materialized view or streaming table, Catalog Explorer shows refresh status and history.

When is a fixed schedule still the right call?

When the requirement itself is time-based. The docs match SCHEDULE to predictable, time-based refresh requirements, and TRIGGER ON UPDATE to data freshness SLAs or unpredictable refresh periods.