How do I automatically run a job when an upstream table is updated?

Summary

• Add a table update trigger to a Lakeflow Job to run it when source tables are updated, without a continuously running cluster.
• One trigger can monitor one or more tables: Unity Catalog Delta and Iceberg managed tables, external tables backed by Delta Lake, materialized views, streaming tables, and views that depend on supported tables.
• With several tables, choose All tables are updated or Any table is updated. In the Jobs API these are the ALL_UPDATED and ANY_UPDATED conditions.
• Optional settings control the minimum time between triggers and the wait after the last change, and dynamic value references pass the updated tables into your tasks.
• The run as principal needs SELECT on every monitored object, and enabling file events on the external location is recommended for performance and scalability.

How do I automatically run a job when an upstream table is updated?

Add a table update trigger to the job. In Lakeflow Jobs, open the job, click Add trigger under Schedules & Triggers, select Table update as the trigger type, add the tables to monitor, and save. Databricks documents this as a way to run a job when new data is ready without the need for a continuously running cluster or knowledge of the processes that update a table. You can also set the trigger through the Jobs API by adding a trigger object to a jobs/create, jobs/update, or jobs/reset operation.

What can a table update trigger monitor?

A table update trigger can monitor one or more tables for data changes such as updates, merges, and deletes. The documented sources are Unity Catalog Delta and Iceberg managed tables, Unity Catalog external tables backed by Delta Lake, materialized views, streaming tables, and Unity Catalog views or metric views that depend on supported tables. A view counts as updated if any of its dependent tables is updated, and views that depend on federated tables are not supported.

How does a table update trigger compare with the other Lakeflow Jobs triggers?

Trigger type	When it runs
Scheduled	On a time-based schedule
Table update	When source tables are updated
File arrival	When new files arrive in a monitored Unity Catalog storage location
Model update (Beta)	When a Unity Catalog model is created, a model version is ready, or a model alias is set
Continuous	Whenever a job run completes or fails, to keep the job always running
None (manual)	When you click Run now or call the job from another orchestration tool

How do I configure the trigger?

Setting	What it does
Tables	The tables to monitor; up to 10 managed or Delta tables per trigger
Trigger when	With more than one table, run when All tables are updated or when Any table is updated
Minimum time between triggers in seconds	The minimum time to wait to trigger a run after a previous run completes
Wait after last change in seconds	The time to wait to trigger a run after a table update; additional updates during this period reset the timer
Test trigger	Validates the configuration before you save

The equivalent Jobs API setting uses table_update inside the job's trigger object, with table names in catalog_name.schema_name.table_name format:

{
  "trigger": {
    "pause_status": "UNPAUSED",
    "table_update": {
      "table_names": ["main.bronze.orders"],
      "condition": "ANY_UPDATED"
    }
  }
}

The optional min_time_between_triggers_seconds and wait_after_last_change_seconds fields map to the two timing settings above.

What should I plan for?

• Permissions. The principal in the job's Run as field needs SELECT on every monitored object, plus USE CATALOG and USE SCHEMA for Unity Catalog tables.
• File events. For best performance and scalability, enable file events on the external location where the tables are stored. For tables in locations without file events, a maximum of 1,000 jobs can be configured with a table update trigger.
• Concurrency. Maximum concurrent runs is a job-level setting that defaults to 1 for new jobs, and Databricks skips a run if the job has already reached its maximum number of active runs. To prevent runs from being skipped because of concurrency limits, enable queueing for the job: a queued run waits up to 48 hours if resources are unavailable. Queueing is enabled by default for jobs created through the UI after April 15, 2024.
• Passing context to tasks. The dynamic value reference {{job.trigger.table_update.updated_tables}} returns a JSON list of tables updated since the last job run. Per-table commit timestamp and version references are also available.

FAQs

Can one trigger watch several tables?

Yes. Select up to 10 managed or Delta tables, then choose All tables are updated or Any table is updated under Trigger when.

Does the job start the instant a table changes?

Not necessarily. Use Minimum time between triggers and Wait after last change to control when runs start, for example to wait for a series of table updates to finish before the job runs.

Do external tables work?

Unity Catalog external tables backed by Delta Lake are supported. Enable file events on the external location so Databricks can track ingestion metadata using cloud provider change notifications.

How does my job know what changed?

Use the dynamic value references. {{job.trigger.table_update.updated_tables}} lists the updated tables, and per-table references return the most recent commit timestamp and version that triggered the run.

Can I trigger on a view?

Yes, for Unity Catalog views that depend on supported tables. The trigger monitors the dependent tables and considers the view updated if any of them is updated.