What's the best tool to orchestrate data pipelines?

Summary

• The best orchestration tool runs next to your data, starts work from the events your pipelines depend on, and lets you recover a failed run without starting over.
• On Databricks, Lakeflow Jobs is the orchestrator: one job coordinates notebooks, Python, SQL, dbt projects and pipelines as tasks, on serverless compute by default.
• Lakeflow Pipelines handles declarative ETL inside a job, working out dependencies between streaming tables and materialized views and reprocessing only new or changed data when possible.
• Lakeflow Jobs can repair a run by re-running only the unsuccessful tasks and their dependents, and can backfill a job across a historical date range.

What's the best tool to orchestrate data pipelines?

The best orchestration tool runs next to your data, reacts to the events your pipelines depend on, and makes a failed run cheap to recover. For teams building on Databricks, that tool is Lakeflow Jobs, with Lakeflow Pipelines handling the declarative transformation steps inside it. Scheduling, run history, alerts and lineage then sit in the same workspace as the tables they describe.

What does a data pipeline orchestrator do?

An orchestrator decides when each step runs, in what order, on what compute, and what happens when a step fails. It starts runs on a schedule or an event, passes parameters between steps, retries transient errors, sends alerts and keeps an auditable history. Transformation code defines what a step computes; the orchestrator owns when, and whether, it runs.

What should you look for in an orchestration tool?

Capability	Why it matters
Event-based triggers	Starting a run when files land or a source table changes takes the guesswork out of schedules.
Dependencies and branching	Real workflows fan out, wait on several upstream steps and branch on results.
Partial reruns and backfills	Re-running only failed steps, or replaying a past date range, saves time and compute after an incident.
Managed compute	Provisioned, autoscaled compute per task takes cluster management out of the job.
Pipelines as code	Jobs in source control can be reviewed, tested and promoted between environments.
Alerts and run history	Teams need to hear about failures and slow runs, and to see what ran, when and why.
Lineage and cost visibility	Knowing which tables a job touches and what each run costs is how you trace impact and control spend.

How does Databricks approach pipeline orchestration?

Lakeflow Jobs is the orchestrator. Databricks describes it as workflow automation that lets you coordinate and run multiple tasks as part of a larger workflow. Tasks can be notebooks, Python scripts or wheels, SQL, JARs, dbt projects, pipelines, or another job. If/else tasks branch on job parameters or task values, and For each runs a nested task once per input.
Jobs start from triggers: a time-based schedule, file arrival in a Unity Catalog storage location, a table update on source tables, or continuous mode, which starts a new run whenever the previous one completes or fails. You can also start runs with Run now or from other orchestration tools.
Serverless compute for jobs is the default for every task that supports it. Databricks manages the compute, autoscaling and Photon are enabled automatically, and auto-optimization retries failed tasks by default. Build jobs in the UI or define them as code with Declarative Automation Bundles (YAML or Python), the Databricks SDKs, the CLI or the Jobs REST API.
Lakeflow Pipelines covers declarative ETL. You define streaming tables and materialized views in SQL or Python, and the pipeline analyzes their dependencies, runs them in the correct order with maximum parallelism, and retries transient failures at the Spark task, flow and pipeline level. Materialized views stay current incrementally, reprocessing only new or changed source data when possible. Pipelines run in triggered or continuous mode, and expectations check each record: you choose whether invalid records are kept, dropped, or fail the update. A pipeline task runs a pipeline as one step of a job.

How do you handle failures, reruns and backfills?

• Dependencies: by default a task runs only when all of its dependencies have run and succeeded. Other Run if conditions, such as All done or At least one failed, add cleanup and alerting branches.
• Repair runs: Repair run re-runs only the unsuccessful tasks and any tasks that depend on them. Successful tasks aren't re-run, and you can change job or task settings first.
• Backfills: Run backfill, in the Run now menu, generates runs across a historical date range and interval, and can pass values such as {{backfill.iso_datetime}} into job parameters, which the job must use to process the right data. A backfill runs the entire job, not a subset of tasks.
• Notifications: jobs and tasks can notify email addresses, Slack or any webhook-based service on start, success, failure, or when a run's duration crosses a threshold.

How do you monitor runs, lineage and cost?

The Runs tab lists running and recently completed runs for all jobs and pipelines in a workspace. A job's matrix view shows each task's run history, and the timeline view helps you find slow tasks. With Unity Catalog enabled, you can view lineage for the Unity Catalog tables in a workflow, captured automatically down to the column level. For account-wide reporting, join system.lakeflow.jobs and system.lakeflow.job_run_timeline with system.billing.usage to see cost per job and per run.

FAQs

Can another orchestrator trigger Lakeflow Jobs?

Yes. Runs can start manually with Run now or programmatically from other orchestration tools, so an existing scheduler can hand Databricks work to a job.

Can Lakeflow Jobs run dbt?

Yes. The dbt task runs a dbt Core project from a workspace folder or a Git provider, and the SQL dbt generates runs on a SQL warehouse.

How do I see what my pipelines cost?

Query the billing and Lakeflow system tables, or import the jobs and pipelines monitoring dashboard Databricks publishes as a JSON file on GitHub.