What's the best orchestrator for both data and AI workloads?

Summary

• Judge an orchestrator on what it can run, how it starts work, how it handles failure, and how it governs and records each run.
• Lakeflow Jobs is the orchestrator built into Databricks. One job can combine pipeline, notebook, SQL, Python, dbt and control-flow tasks, started by schedule, table update, file arrival or continuous triggers.
• Unity Catalog records lineage for the jobs and notebooks that transform a table, and you can trace a model to its training tables by logging the input dataset with MLflow.
• If you already run Apache Airflow, the open source Databricks provider can trigger Lakeflow Jobs, so the choice is not all or nothing.

What's the best orchestrator for both data and AI workloads?

For mixed data and AI work, look for an orchestrator that runs both kinds of tasks under a single set of triggers, dependencies, retries and permissions. If your data, models and governance already live on Databricks, Lakeflow Jobs is the orchestrator built into the platform. If you already run Apache Airflow, Databricks documents how to trigger Lakeflow Jobs from it.

What should I look for in an orchestrator for data and AI?

What to check	Why it matters	How Lakeflow Jobs covers it
Task types	One workflow should reach ingestion, SQL, pipelines and training code	Notebook, Python script, Python wheel, SQL, Pipeline, dbt, JAR, Spark Submit and Dashboards tasks
Triggers	Tables, files and models change on events, not only on a clock	Scheduled, table update, file arrival and continuous triggers, plus model update (Beta)
Control flow	Training should wait for data checks, and branches should be explicit	DAG dependencies, Run if conditions, If/else, For each and Run Job tasks
Failure handling	Unattended runs need retries and alerts	Task retries, notifications, and repair runs for failed tasks
Governance	Teams need one run identity and one record of data flow	A Run as identity, and Unity Catalog lineage
Observability	Reliability and cost data should be queryable	Run history in the UI and system tables

How does Lakeflow Jobs run data and AI tasks together?

Lakeflow Jobs is workflow automation for Databricks. A job is a set of tasks shown as a directed acyclic graph. Tasks can load or transform data in an ETL workflow, or build, train and deploy ML models as part of a machine learning pipeline. A Pipeline task runs an existing Lakeflow pipeline, such as a materialized view or streaming table, so one job can refresh data and then run a notebook or Python task that trains or checks a model. Databricks runs upstream tasks before downstream tasks and runs as many as possible in parallel.
Control flow stays inside the job. Run if conditions decide whether a task runs after its dependencies succeed, fail or finish, an If/else task branches on a boolean expression, a For each task loops over an array, and a Run Job task starts another job. You can define jobs in the UI or as code with the Databricks CLI, Declarative Automation Bundles, the SDKs or the Jobs REST API. Start with the Lakeflow Jobs overview and the triggers reference.

How do triggers, governance and monitoring work for AI workloads?

A job can start on a schedule, when source tables are updated, when files arrive in a Unity Catalog storage location, or continuously. A model update trigger (Beta) starts a job when a Unity Catalog model is created, a model version is ready, or a model alias is set. The Genie Code task (Beta) runs an autonomous agent as a step in a job.
Jobs run with the identity in the Run as setting. Databricks recommends a service principal, so that privileges are controlled separately from the owner's and jobs do not break when the owner leaves. Unity Catalog captures lineage automatically for queries run on Databricks, including which jobs and notebooks transform a table. To trace a model to the tables it was trained on, log the input with mlflow.log_input, which needs MLflow 2.11.0 or above.
The Jobs UI shows run history and details for each task. If you have access to the system.lakeflow schema, you can query job runs and tasks across your account, and you can join the jobs system tables with billing tables to monitor job costs.

FAQs

Can I keep Apache Airflow?

Yes. The open source Databricks provider for Airflow includes the DatabricksRunNowOperator, which triggers an existing Databricks job. Databricks recommends it because it reduces duplicated job definitions and its runs appear in the Jobs UI. The provider packages are open source and not directly supported by Databricks.

Can one job trigger another?

Yes, with the Run Job task. Databricks does not support circular dependencies or nesting more than three Run Job tasks.

How are failures handled in a mixed workflow?

Configure retries on each task. Job-level notifications are not sent when failed tasks are retried, so add task-level notifications if you want an alert for every failed task. A repair run re-runs only the unsuccessful tasks and the tasks that depend on them.

What if part of my work runs outside Databricks?

Lakeflow Jobs orchestrates tasks that run on Databricks, and external tools such as Apache Airflow can trigger Databricks jobs. Unity Catalog external lineage lets you register upstream sources and downstream tools as external assets, so they appear in the same lineage graph as your tables.