What's the best way to build event-driven data pipelines?

Summary

• Match the mechanism to where the event appears: a file arrival trigger for new files, a table update trigger for changed tables, Structured Streaming for message buses such as Apache Kafka, and the Jobs REST API when an application must start a run.
• Chain stages with task dependencies, Run Job tasks or table update triggers, and keep bad data out with pipeline expectations.
• Checkpoints give Structured Streaming and Auto Loader their fault tolerance, and watermarks bound the state that late events would otherwise require.
• A triggered pipeline runs its cluster only long enough to update; a continuous pipeline needs an always-running cluster, which costs more and lowers latency.

What's the best way to build event-driven data pipelines?

Match the mechanism to where the event appears, then chain the stages. On Databricks, file arrival and table update triggers in Lakeflow Jobs start runs from file and table events instead of a clock. Structured Streaming reads message buses such as Apache Kafka. The Jobs REST API lets an application start a run. Lakeflow Pipelines transforms the data, and Unity Catalog records lineage across tables and jobs.

Which Databricks mechanism handles which event?

Event	Mechanism	Notes from the docs
New files in cloud storage	File arrival trigger, then Auto Loader	Best-effort check about every minute of an external location or volume
A Unity Catalog table changed	Table update trigger	Fires on updates, merges and deletes; any or all monitored tables
Messages on Apache Kafka	Structured Streaming kafka source, or read_kafka in a pipeline	Run continuously, or as incremental batches with availableNow
An event raised by an application	Jobs REST API run-now	The application calls POST /api/2.2/jobs/run-now to start a run of an existing job
An upstream job finished	depends_on or a Run Job task	No circular dependencies

How do I react to new files?

Add the trigger to the job, and let Auto Loader track which files it has already processed. Auto Loader stores file metadata in its checkpoint location, which ensures data is processed exactly once. If you use foreachBatch for custom logic, you get at-least-once guarantees instead.

trigger:
  file_arrival:
    url: /Volumes/mycatalog/myschema/myvolume/landing/
    wait_after_last_change_seconds: 60

Set wait_after_last_change_seconds when files arrive in batches so a whole batch lands before the run starts.

How do I process a continuous stream?

Structured Streaming lets you express computation on streaming data the way you express a batch computation on static data, with end-to-end fault tolerance and exactly-once processing guarantees. This reads a Kafka topic into a table:

df = (spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "<server:ip>")
  .option("subscribe", "<topic>")
  .option("startingOffsets", "latest")
  .load())

(df.writeStream
  .option("checkpointLocation", "/Volumes/catalog/schema/volume/path")
  .toTable("catalog.schema.table"))

The checkpoint records source offsets and commits, so a restarted query resumes where it left off without reprocessing data. Use .trigger(availableNow=True) for scheduled incremental batches, and run always-on streaming in a continuous job. For late events, withWatermark("event_time", "10 minutes") sets a lateness threshold. Records inside the threshold are always processed, and records outside it might still be processed, but that is not guaranteed.

How do I chain stages and keep bad data out?

• Chain with table changes. A table update trigger on a stage's output table starts the next job, and {{job.trigger.table_update.updated_tables}} tells the job which tables changed.
• Chain inside a job. Use depends_on between tasks, or a Run Job task to start another job. Run if conditions control what happens when an upstream task fails.
• Gate on quality. Pipeline expectations can drop invalid records (dp.expect_or_drop) or fail the update (dp.expect_or_fail). To stop downstream work on a failure, put validation in its own pipeline and make the downstream task depend on it.
• Pick the pipeline mode. Triggered mode refreshes the available data and stops. Continuous mode keeps tables fresh as data arrives and requires an always-running cluster.

What does an end-to-end flow look like?

1. Files land in a Unity Catalog volume, and a file arrival trigger starts the first job.
2. That job runs an Auto Loader task that appends the new files to a Delta table.
3. A table update trigger on that table starts a second job with a pipeline task.
4. The pipeline's expectations drop or fail invalid records, and later tasks run only if it succeeds.
5. Failure notifications alert the team, and lineage shows how the tables connect.

What about retries and lineage?

Set retries on a task to re-run it after an error. Continuous jobs retry automatically with exponential backoff. Add notifications for failures. Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, so you can trace which queries and files populate a table and which jobs and notebooks transform it. The lineage graph's nodes are tables, views, model versions, external assets and file paths.

FAQs

Can an external system start a job?

Yes. Call the Jobs REST API with POST /api/2.2/jobs/run-now and the job_id. The documented trigger types include scheduled, table update, file arrival and continuous, plus manual or programmatic runs.

When should I choose continuous over triggered?

The docs say triggered mode suits data updates that run every 10 minutes, hourly or daily, and continuous mode suits updates wanted every 10 seconds to a few minutes.

Do the triggers add cost?

File arrival and table update triggers do not incur additional costs other than cloud provider costs for listing files or tables and reading updates.