How do I stop paying for pipeline runs that fire when nothing has changed?

Summary

• Start runs from the data, not the clock: a Lakeflow Jobs table update trigger or file arrival trigger starts a run only after a monitored table changes or new files land.
• Two trigger options turn bursts of change into one run: "Minimum time between triggers in seconds" caps how often runs start, and "Wait after last change in seconds" waits until a batch finishes landing.
• Inside a run, incremental processing handles only new data, and an If/else condition task can skip downstream tasks when no new rows arrived.
• Keep a scheduled trigger only when something must happen at a fixed time; multiple triggers on one job (Beta) can pair a data trigger with a scheduled backup.

How do I stop paying for pipeline runs that fire when nothing has changed?

Start runs from the data instead of the clock. In Lakeflow Jobs, a table update trigger starts a run when a monitored table is updated, and a file arrival trigger starts a run when new files land in a Unity Catalog external location or volume. A scheduled trigger is time-based, so it fires at each scheduled time whether or not the source changed. The Databricks docs describe file arrival triggers as useful when "a scheduled job's efficiency is compromised by irregular new data arrivals."

Which triggers start a run only when something changed?

Trigger	Starts a run when	Limits to know
Scheduled	A time-based schedule fires (simple interval or Quartz cron)	Minimum 10 seconds between scheduled runs
Table update	A monitored table is updated (updates, merges, deletes)	Up to 10 managed or Delta tables per trigger; the Run as principal needs SELECT on each
File arrival	New files arrive in a monitored location	Best-effort check about every minute; no wildcards in the path
Continuous	A run completes or fails, so the next run starts	One running instance; no task dependencies

How do I find the scheduled jobs worth switching?

Databricks system tables record job runs and tasks across the account, so you can query them to analyze job performance and costs. This query lists the jobs with the most scheduled runs in the last 7 days:

SELECT workspace_id, job_id, COUNT(DISTINCT run_id) AS scheduled_runs
FROM system.lakeflow.job_run_timeline
WHERE trigger_type IN ('CRON', 'PERIODIC')
  AND period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
GROUP BY workspace_id, job_id
ORDER BY scheduled_runs DESC

Then check each job's source. If it is a Unity Catalog table, or files in a Unity Catalog external location or volume, a data trigger can replace the schedule.

How do I set up a trigger that fires only on change?

In the job, click Add trigger and choose Table update or File arrival. A table update trigger can fire when any or all of the monitored tables change, and its Test trigger button validates the configuration before you save. Two advanced options shape the pattern:

• Minimum time between triggers in seconds is a cooldown. Updates that arrive during the cooldown wait until it ends.
• Wait after last change in seconds is a debounce. Each new update resets the timer, so a batch of updates produces one run.

This is a table update trigger in the Jobs API job settings (the API minimum for both options is 60 seconds):

"trigger": {
  "pause_status": "UNPAUSED",
  "table_update": {
    "table_names": ["main.sales.orders_raw"],
    "condition": "ANY_UPDATED",
    "min_time_between_triggers_seconds": 900,
    "wait_after_last_change_seconds": 120
  }
}

How do I make each run cheap when little has changed?

• Process only new data. Auto Loader incrementally processes new files, and trigger(availableNow=True) processes the data available when the run starts.
• Let materialized views skip. When no source changes are detected, a materialized view refresh reports "No change" and does not update the view. Incremental refresh requires serverless compute.
• Prefer triggered pipelines. A triggered pipeline runs its cluster only long enough to update. Continuous mode needs an always-running cluster, which costs more but reduces latency.
• Skip downstream tasks. An If/else condition task can run transformation tasks only if the upstream ingestion task adds new data. The ingestion notebook sets a task value with dbutils.jobs.taskValues.set(key="new_rows", ...), and this bundle excerpt reads it:

tasks:
  - task_key: has_new_rows
    depends_on:
      - task_key: ingest
    condition_task:
      op: GREATER_THAN
      left: '{{tasks.ingest.values.new_rows}}'
      right: '0'
  - task_key: transform
    depends_on:
      - task_key: has_new_rows
        outcome: 'true'
    pipeline_task:
      pipeline_id: ${resources.pipelines.transform.id}

FAQs

What if I still need data by a fixed time?

Add a scheduled trigger beside the data trigger. Multiple triggers on one job is in Beta, allows up to five triggers, and a workspace admin turns it on from the Previews page. When any active trigger evaluates to true, it starts a run.

Why did my trigger not start a run?

The Run as principal needs SELECT on every monitored table, or the trigger fails to evaluate and the job does not run. For files, overwriting an existing file with the same name does not trigger a run. Add job failure notifications to hear when a trigger fails to evaluate.

Do these triggers add cost?

File arrival and table update triggers add no cost beyond cloud provider charges for listing files or tables and reading updates. On AWS, enabling file events on an external location creates an SNS topic and an SQS queue in your account, and those carry standard AWS messaging charges.

Can I still run the job by hand?

Yes. Besides the configured trigger, you can trigger job runs manually with the Run now button or through external orchestration tools.