How do I make a job wait until upstream data is ready before it runs?

Summary

• Start the job from the data instead of a schedule: a Lakeflow Jobs table update trigger starts a run when monitored tables are updated, and a file arrival trigger starts one when new files land.
• Set the table update trigger to fire only when all tables are updated, and use "Wait after last change" so the run starts after a burst of writes has finished.
• Inside the job, task dependencies run upstream tasks first, and an If/else condition task can skip downstream work when a check finds no new rows.
• The documented Lakeflow Jobs task types include no polling or wait task; triggers, task dependencies, If/else tasks, and Run Job tasks handle the waiting.

How do I make a job wait until upstream data is ready before it runs?

Start the job from the data instead of the clock. In Lakeflow Jobs, a table update trigger set to fire when all of its tables are updated, with a wait after the last change, starts the run only after every input has landed and the writes have stopped. File arrival triggers do the same for files. Inside the job, task dependencies and an If/else condition task keep downstream steps away from empty or partial data.

What are the ways to gate a job on upstream data?

Upstream data is	Mechanism	How the job waits
Unity Catalog tables written by other jobs	Table update trigger	No run starts until any or all monitored tables are updated
Files dropped by an external system	File arrival trigger	No run starts until new files arrive in a volume or external location
Produced earlier in the same job	Task dependencies	Downstream tasks start after their upstream tasks finish
Produced by another job you own	Run Job task	A parent job triggers that job as one of its tasks, and downstream tasks depend on it
Present but possibly empty	If/else condition task	Downstream tasks run only when the condition is true

How do table update triggers wait for every input?

A table update trigger checks the tables it monitors and runs the job when they're updated. With more than one table, you choose whether a run starts when any table is updated or only when all of them are. Two advanced options shape the wait:

• Wait after last change holds the run until updates stop, because each new update resets the timer. The docs suggest it when updates arrive in batches.
• Minimum time between triggers sets how long to wait after a previous run completes before another can start.

This job waits for both inputs, checks that rows arrived, and only then builds the mart:

resources:
  jobs:
    build_daily_mart:
      name: build_daily_mart
      queue:
        enabled: true
      trigger:
        pause_status: UNPAUSED
        table_update:
          table_names:
            - main.raw.orders
            - main.raw.payments
          condition: ALL_UPDATED
          wait_after_last_change_seconds: 300
      tasks:
        - task_key: check_inputs
          notebook_task:
            notebook_path: ../src/check_inputs.ipynb
        - task_key: has_new_rows
          depends_on:
            - task_key: check_inputs
          condition_task:
            op: GREATER_THAN
            left: '{{tasks.check_inputs.values.new_rows}}'
            right: '0'
        - task_key: build_mart
          depends_on:
            - task_key: has_new_rows
              outcome: 'true'
          notebook_task:
            notebook_path: ../src/build_mart.ipynb

The check_inputs notebook counts the rows it expects and publishes the count with dbutils.jobs.taskValues.set(key = "new_rows", value = new_rows). The condition task needs no cluster, and build_mart runs only when it evaluates to true. Downstream tasks default to the All succeeded Run if condition; set All done on a cleanup task that must run regardless of outcome, or At least one failed on an alerting task. The principal in the job's Run as field needs SELECT on every monitored table; without it, the trigger fails to evaluate and the job doesn't run.

How do I wait for a complete batch of files?

Use a file arrival trigger on a Unity Catalog volume or external location. Set Wait after last change longer than the gap between files in a batch and shorter than the gap between batches, so each run starts after a whole batch has landed. Only new files trigger runs, and Auto Loader can then process them incrementally with exactly-once guarantees.

What happens when runs overlap or data is late?

By default a job allows one active run, and runs beyond that limit are skipped. The queue setting above makes them wait instead. If the data never arrives, the trigger never fires, so no run processes partial data. Configure notifications on job failure to hear about triggers that fail to evaluate, and set timeout_seconds on tasks to cap how long each task run can take.

FAQs

Is there a task that polls until the data shows up?

No. The documented Lakeflow Jobs task types don't include a polling or wait task. Use triggers to start runs on data events, and If/else condition tasks to branch inside a run.

Does a waiting job use compute?

No. Nothing runs until the trigger fires. The docs describe table update triggers as a way to run a job when new data is ready without a continuously running cluster.

Can a task see which tables changed?

Yes. Table update triggers expose {{job.trigger.table_update.updated_tables}}, a JSON list of the tables updated since the last run, plus dynamic value references for each table's most recent commit timestamp and version.

How do I make one job wait for another?

Add a Run Job task to a parent job and make the downstream tasks depend on it, or put a table update trigger on the tables the first job writes. The trigger needs no knowledge of the process that updates those tables. Databricks doesn't support circular Run Job dependencies or nesting more than three Run Job tasks.

Can I start the job before the data is ready?

Yes. You can still start a run with Run now or the Jobs API, and the gating tasks run as part of it.