I want to run my pipeline as soon as new data arrives instead of on a fixed schedule. What should I use?

Summary

• Replace the schedule with an event trigger on a Lakeflow Job: a file arrival trigger for files landing in a Unity Catalog volume or external location, or a table update trigger for changes to Unity Catalog tables.
• A file arrival trigger makes a best effort to check for new files every minute. A table update trigger can monitor up to 10 tables and start the job when any or all of them update.
• Minimum time between triggers sets a cooldown, and Wait after last change waits for a batch of arrivals to finish.
• For data that arrives continuously, use a continuous job. Auto Loader processes new files incrementally with exactly-once guarantees.

I want to run my pipeline as soon as new data arrives instead of on a fixed schedule. What should I use?

Use an event trigger on a Lakeflow Job instead of a schedule. Use a file arrival trigger when data lands as files in a Unity Catalog volume or external location, a table update trigger when it lands as updates to Unity Catalog tables, and a continuous trigger when data arrives continuously and you want the job always running. File arrival and table update triggers start a run only when the watched files or tables change.

Which trigger fits how your data arrives?

Data arrives as	Trigger	What it watches	Limits to know
New files in a volume or external location	File arrival	The root or a subpath of the location, including its subdirectories	Only new files trigger runs; paths cannot contain wildcards
Updates, merges or deletes in tables	Table update	One or more Unity Catalog tables, with a rule for any or all	Up to 10 tables per trigger; the Run as principal needs SELECT
A constant flow of data	Continuous	A new run starts when the previous run completes or fails	One running instance; no task dependencies
A fixed time	Scheduled	A simple interval, or an advanced schedule that can be edited as Quartz cron syntax	The scheduler is not intended for low-latency jobs

How do file arrival triggers work?

A file arrival trigger makes a best effort to check for new files every minute, and the performance of the underlying cloud storage can affect that. It needs Unity Catalog, a volume or external location, READ permission on the location and CAN MANAGE permission on the job. Databricks recommends enabling file events on the external location. To process the files that started a run, the file arrival docs show Auto Loader, which processes new files with exactly-once guarantees.
Two advanced options control run frequency:

• Minimum time between triggers in seconds limits the job to at most one run per interval, a cooldown between runs.
• Wait after last change in seconds waits after the most recent file arrives and resets on each new arrival, so a batch of files starts one run.

How do table update triggers work?

A table update trigger checks monitored tables for changes and starts the job when a table is updated. It can watch Unity Catalog managed Delta and Iceberg tables, external tables backed by Delta Lake, materialized views, streaming tables, and views that depend on supported tables. The job's Run as principal must have SELECT on every monitored object, or the trigger fails to evaluate. Dynamic value references such as {{job.trigger.table_update.updated_tables}} pass the updated tables to the job's parameters.

How do I set it up in code?

You can define the trigger in a Declarative Automation Bundle. This job runs a pipeline when new files arrive and waits for a batch to finish:

resources:
  jobs:
    ingest_on_arrival:
      name: ingest_on_arrival
      trigger:
        file_arrival:
          url: /Volumes/mycatalog/myschema/myvolume/
          wait_after_last_change_seconds: 60
      tasks:
        - task_key: run_pipeline
          pipeline_task:
            pipeline_id: 11111111-1111-1111-1111-111111111111

A table update trigger uses table_update with table_names and a condition of ANY_UPDATED or ALL_UPDATED.

How do pipelines fit in?

A Lakeflow pipeline runs in triggered or continuous mode. Triggered mode refreshes the data available when the update starts and then stops. Continuous mode keeps tables fresh as new data arrives. When a job runs a pipeline, the job decides the mode: a triggered or scheduled job runs its pipeline as a single update, and a continuous job runs it continuously.
For files in cloud object storage, Databricks recommends streaming tables for most ingestion use cases, with Auto Loader for incremental, idempotent loading. Databricks recommends running continuous pipelines with a continuous job rather than the pipeline's built-in continuous mode.

Can I combine a schedule with an event trigger?

Support for multiple triggers on one job is in Beta, and a workspace admin controls access from the Previews page. During the Beta a job can have up to five triggers, of the same or different types. For example, add a table update trigger to start the job when data is ready and a scheduled trigger as a backup. A continuous trigger cannot be combined with any other trigger.

FAQs

What happens when no data arrives?

File arrival and table update triggers start no run. For a minimum cadence, use the Beta multiple-triggers feature to add a scheduled trigger as a backup.

Can I filter which files start a run?

The path for a file arrival trigger cannot contain wildcards, and the trigger checks all subdirectories of the path. Point the trigger at a subpath that holds the files you care about, and use the Auto Loader pathGlobFilter option to select files by pattern when you load them.

How quickly does a run start after data arrives?

File arrival triggers make a best effort to check every minute. The Minimum time between triggers and Wait after last change options add delay by design, and the Jobs API accepts a minimum of 60 seconds for each. For lower latency, use a continuous job or a pipeline in continuous mode.