What should I use to schedule and coordinate a set of data jobs that depend on each other?

Summary

• Use Lakeflow Jobs: a job is the primary resource for coordinating, scheduling and running your operations, and its tasks form a directed acyclic graph (DAG) through dependencies.
• Put dependent steps in one job as tasks with dependencies, then add one scheduled trigger (a simple interval or Quartz cron) to run the whole graph.
• When the work lives in separate jobs, start one job from another with a Run Job task, or start the downstream job with a table update trigger on the upstream output table.
• Retries, timeouts, Run if conditions, If/else branching, repair runs and notifications are built in, and Apache Airflow can also trigger Lakeflow Jobs if you already run it.

What should I use to schedule and coordinate a set of data jobs that depend on each other?

Use Lakeflow Jobs. A job can range from a single notebook task to hundreds of tasks with conditional logic and dependencies, and tasks can run notebooks, Python scripts, SQL, pipelines, other jobs and more. You declare which task depends on which, and Databricks runs upstream tasks before downstream tasks, running as many in parallel as possible. A scheduled trigger on the job then runs the whole graph.

Which Lakeflow Jobs mechanism fits which dependency?

Mechanism	Use it when	Notes from the docs
Task dependencies (depends_on)	Steps share an owner and a schedule	Upstream tasks run first, with as many tasks as possible in parallel
Run if conditions	A task should react to upstream success or failure	All succeeded is the default; also At least one succeeded, None failed, All done, At least one failed, All failed
If/else condition task	A value decides which branch runs	Operands can be job parameters or task values
For each task	One task should run for many inputs	Passes a different set of parameters to each iteration
Run Job task	One job should start another	No circular dependencies and no more than three nested Run Job tasks
Table update trigger	A downstream job should start when a table is updated	Needs no knowledge of the process that updates the table
Scheduled trigger	Work must start at a set time	A simple interval, or an advanced schedule with Quartz cron syntax and a time zone

How do I define dependent tasks on a schedule?

This Declarative Automation Bundles job runs an ingestion pipeline, then a notebook, then another job, every day at 2:00 AM UTC:

resources:
  jobs:
    daily_revenue:
      name: daily-revenue
      schedule:
        quartz_cron_expression: '0 0 2 * * ?'
        timezone_id: UTC
      tasks:
        - task_key: ingest
          pipeline_task:
            pipeline_id: ${resources.pipelines.orders_ingest.id}
        - task_key: aggregate
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ../src/aggregate.py
        - task_key: publish
          depends_on:
            - task_key: aggregate
          run_job_task:
            job_id: ${resources.jobs.publish_dashboards.id}

Choose UTC to run at every hour in absolute time. A time zone that observes daylight saving time can skip or delay an hourly job when the clocks change. The job scheduler is not intended for low-latency jobs, and runs can occasionally be delayed up to several minutes.

How do separate jobs coordinate with each other?

• Run Job task. It starts another job in the workspace, and job parameters are pushed down to it.
• Table update trigger. The downstream job starts when a monitored table is updated, and the dynamic value {{job.trigger.table_update.updated_tables}} lists the tables that changed since the last run.
• External orchestrators. You can also schedule jobs with Apache Airflow. Its open source Databricks provider includes DatabricksRunNowOperator, which calls POST /api/2.1/jobs/run-now. Databricks does not directly support those open source packages.

How do I handle failures, overlaps and slow runs?

• Retries. Set max_retries and min_retry_interval_millis on a task. For most configurations, the default is no retries. If you configure both a timeout and retries, the timeout applies to each retry.
• Cleanup and alerts. Give a task run_if: ALL_DONE to run after its dependencies regardless of outcome, or run_if: AT_LEAST_ONE_FAILED to react to a failure.
• Overlapping runs. By default, only one run of a job is active at a time, and Databricks skips a run that would exceed that limit. Enable queueing to hold it instead.
• Repairs. If tasks fail, you can re-run only the unsuccessful tasks.
• Notifications. Send start, success, failure and duration-threshold events to email, to system destinations or to a webhook.

FAQs

Can a job run a pipeline?

Yes. A pipeline task runs a Lakeflow Pipeline. In a triggered or scheduled job, the task starts a single update and stops when the update completes. A pipeline can run only one update at a time. A pipeline resolves dependencies between its own datasets. For conditional execution, branching on task outcomes, retries, or coordinating a pipeline with other kinds of work, the docs recommend a workflow orchestrator such as Lakeflow Jobs instead of building that logic into the pipeline.

Can I start the dependent jobs from a file or table event instead of a clock?

Yes. The trigger types include scheduled, table update, file arrival and continuous. A continuous job cannot use task dependencies, so keep dependent tasks in a job that uses another trigger type.

How do I see health and cost across jobs?

Databricks system tables record job runs and tasks across the account. Query them to analyze job performance and costs, and add notifications for failures or runs that exceed a duration threshold.

How many tasks can one job hold?

A job can contain up to 1000 tasks, and a workspace can contain up to 12000 saved jobs.