How do I trigger a job when a new file lands in cloud storage (S3/ADLS/GCS)?

Summary

• Add a file arrival trigger to a Lakeflow Job and point it at a Unity Catalog external location or volume path; the job runs when new files arrive in Amazon S3, Azure storage or Google Cloud Storage.
• Databricks makes a best effort to check for new files every minute, and enabling file events on the external location improves the trigger's performance and scale.
• Pair the trigger with Auto Loader so each run processes only the files that arrived since the last run.
• "Minimum time between triggers in seconds" and "Wait after last change in seconds" control how often runs start and let a batch of files finish landing first.

How do I trigger a job when a new file lands in cloud storage (S3/ADLS/GCS)?

Add a file arrival trigger to a Lakeflow Job and point it at a Unity Catalog external location or volume. The trigger checks the location, including its subdirectories, and starts a run when new files arrive. It works with external locations on Amazon S3, Azure storage and Google Cloud Storage.

What do I need before I add the trigger?

• Unity Catalog enabled on the workspace.
• A storage location that is a Unity Catalog volume or external location.
• READ permission on the storage location and CAN MANAGE permission on the job.
• Recommended: managed file events on the external location. You must own the location or hold the MANAGE privilege to enable them.

How do I add the trigger?

1. Click Jobs & Pipelines in the sidebar and open the job.
2. In the Job details pane, click Add trigger and select File arrival as the trigger type.
3. Enter the URL of the root or a subpath of the external location or volume.
4. Optionally set the advanced options, click Test connection, then click Save.

The same trigger as code, in a Declarative Automation Bundles job definition:

resources:
  jobs:
    ingest_on_arrival:
      name: ingest-on-arrival
      trigger:
        pause_status: UNPAUSED
        file_arrival:
          url: /Volumes/mycatalog/myschema/myvolume/mydirectory/
          wait_after_last_change_seconds: 60

How does Databricks notice the new file?

The trigger makes a best effort to check for new files every minute. When file events are enabled on the external location, Databricks tracks ingestion metadata by processing change notifications from the cloud provider. File events are enabled by default on new external locations. Existing triggers on a covered path start benefiting within minutes of enabling file events, and new triggers benefit within seconds.

Cloud	Storage	What Databricks configures in Automatic mode
AWS	Amazon S3	A managed SNS topic (csms-* prefix) and an SQS queue
Azure	Azure storage	Azure Event Grid and Azure Storage Queues
Google Cloud	Cloud Storage	File change notifications from your bucket

Without file events, a workspace can have at most 50 jobs with file arrival triggers on such locations, and the monitored location can hold up to 10,000 files. With file events, the number of files is not limited.

How do I process only the new files?

Use Auto Loader in the job's task. It incrementally and efficiently processes new files with exactly-once guarantees. Set file_location to the URL you gave the trigger, and keep checkpoint_location outside it. If you only need the URLs of new files for custom logic, use foreachBatch instead, which provides only at-least-once processing guarantees.

(spark.readStream.format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_location)
  .option("cloudFiles.useManagedFileEvents", "true")
  .load(file_location)
  .writeStream
  .option("checkpointLocation", checkpoint_location)
  .trigger(availableNow=True)
  .toTable(sink_table))

How do I control how often runs start?

• Minimum time between triggers in seconds is a cooldown: at most one run per interval. A value of 900 creates runs as files arrive, capped at one run every 15 minutes.
• Wait after last change in seconds waits after the most recent file arrives, and each new arrival resets the timer. If a batch starts about every 5 minutes, a value of 60 processes each batch in one run, as long as the files in a batch arrive within 60 seconds of each other.

The Jobs API accepts a minimum of 60 seconds for both options.

What are the limits?

• Only new files trigger runs. Overwriting an existing file with the same name does not.
• The path cannot contain wildcards such as * or ?, so watch a dedicated directory.
• The path must not contain external tables or managed locations of catalogs and schemas.
• With file events, a trigger on a subpath can exceed its processing time limit and enter an error state when there are too many changes outside that subpath.

FAQs

What if a file arrives while the job is running?

By default, only one run of a job is active at a time, and a run that exceeds the limit is skipped. Queueing, which is on by default for jobs created in the UI after April 15, 2024, holds such a run for up to 48 hours until capacity is available. Auto Loader keeps its progress in the checkpoint, so the next run picks up files it has not processed.

How do I find out that a trigger failed?

To be notified if a file arrival trigger fails to evaluate, configure email or system destination notifications on job failure.

Do file arrival triggers cost extra?

They do not incur additional costs other than cloud provider costs for listing files in the storage location.

What if the monitored directory does not exist?

On S3 and Google Cloud Storage, the trigger keeps evaluating with no error, because those stores do not differentiate between non-existent, deleted and empty directories. Check the path if no runs start.