What's the best way to orchestrate dbt models in production?

Summary

• Run dbt as a task inside the orchestrator that runs the rest of your data workflow: dbt builds and tests models in dependency order, and the orchestrator decides when it runs and what follows a failure.
• dbt Core is a command-line tool. dbt's own docs describe production as replacing manual command-line runs with scheduling, from the dbt platform or another orchestration tool.
• Use dbt build so tests gate downstream models, and state selection (--select "state:modified+" with --state) to rebuild only what changed since a previous run.
• On Databricks, the Lakeflow Jobs dbt task runs a dbt Core project, executes the generated SQL on a SQL warehouse, and archives each run's logs, results, manifests and configuration automatically.

What's the best way to orchestrate dbt models in production?

Run dbt as a native task in the orchestrator that already runs your ingestion and downstream steps, so the whole workflow shares one schedule, one failure policy and one run history. Let dbt do what it's built for: compiling the project, building models in dependency order and running tests. Let the orchestrator decide when dbt runs and what happens after a failure. On Databricks, that's the dbt task in Lakeflow Jobs.

Why does dbt need an orchestrator?

The dbt engine compiles your project, executes your transformation graph and produces metadata, but it doesn't schedule itself. dbt's deployment docs describe moving from manual command-line runs to scheduled jobs, with the dbt platform's scheduler or with other orchestration tools. The split of responsibilities:

Concern	dbt handles	The orchestrator handles
Build order	dbt build runs models, tests, snapshots and seeds in DAG order	Running dbt after ingestion and before downstream tasks
Data quality	A failing test makes downstream models skip	Stopping or branching the workflow, and alerting
Selection	--select by tag or by state	Running different selections in different jobs
Environments	Targets in profiles.yml, picked with --target	Which workspace, catalog and schema a run writes to
Artifacts	manifest.json and run_results.json in target/	Keeping each run's artifacts and logs
Timing	Not part of dbt Core	Schedules, event triggers, retries and timeouts

How do you orchestrate dbt with Lakeflow Jobs?

The dbt task runs a dbt Core project as a task in a job. The project comes from a workspace folder or a Git provider, and by default the task runs dbt deps, dbt seed and dbt run in sequential order; you can add, remove or edit commands. You also choose:

• A SQL warehouse that runs the SQL dbt generates. Serverless or pro SQL warehouses are required.
• dbt CLI compute that runs dbt Core itself. Databricks recommends serverless compute for jobs or a single-node classic jobs cluster.
• A warehouse catalog and schema. If the catalog is unset, the workspace default is used, and the schema defaults to default.

Lakeflow Jobs archives each run's artifacts automatically, including logs, results, manifests and configuration. databricks jobs get-run-output <task_run_id> returns inline dbt logs and a download link for the packaged artifacts, and the SQL warehouse query history shows the SQL dbt generated. One limitation: you can't run dbt Python models in a dbt task that uses a SQL warehouse.
As code, a dbt task in Declarative Automation Bundles looks like this example from the Databricks docs:

resources:
  jobs:
    my-dbt-job:
      name: my-dbt-job
      tasks:
        - task_key: my-dbt-task
          dbt_task:
            commands:
              - 'dbt deps'
              - 'dbt seed'
              - 'dbt run'
            project_directory: /Users/someone@example.com/Testing
            warehouse_id: 1a111111a1111aa1
          libraries:
            - pypi:
                package: 'dbt-databricks>=1.0.0,<2.0.0'

Place the dbt task after ingestion and before the tasks that read its tables. By default a task runs only when all of its dependencies succeeded, so a dashboard refresh that depends on a failed dbt task doesn't start. Retries, duration thresholds and notifications live in the task's advanced settings. Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, so the tables dbt builds show up in lineage next to the jobs and dashboards that use them.

What are the best practices for production dbt?

• Gate with tests. dbt build runs tests inside the graph, and a failing test causes downstream resources to skip. Set a test's severity to warn when it shouldn't block.
• Rebuild only what changed. --select "state:modified+" selects modified nodes and everything downstream of them, compared against the manifest you pass with --state.
• Pin packages. Declare versions or version ranges in packages.yml. dbt deps records what it installed in package-lock.yml and reinstalls from it while the package files are unchanged.
• Separate environments. Keep dev and prod as targets in profiles.yml and choose one with --target. On Databricks, give each job's dbt task its own catalog and schema, or point profiles_directory at a custom profiles.yml.
• Plan for schema changes. For incremental models, on_schema_change decides what happens when columns change: ignore (the default), fail, append_new_columns or sync_all_columns. --full-refresh drops and rebuilds the model.

FAQs

Should I run several dbt commands in one task or split them?

One dbt task runs its commands in sequence. Splitting dbt run and dbt test into separate tasks gives each its own retries and Run if condition, and shows each step's status in the job's matrix view.

How do I debug a failed dbt run?

Start with the inline logs and artifacts from get-run-output, then find the failing statement in the SQL warehouse query history. Locally, dbt writes logs to logs/ and artifacts to target/, both relative to dbt_project.yml.

How do I re-run after fixing a failure?

Use Repair run. It re-runs the unsuccessful tasks and any tasks that depend on them with the current settings, and doesn't repeat tasks that already succeeded, such as ingestion.

Can some models run on their own schedule?

Yes. Tag them and select by tag, for example dbt run --select "tag:nightly", then give each selection its own job and schedule.