How do I trigger a job only when new data passes my quality checks?

Summary

• Lakeflow Jobs triggers start runs on a schedule or on data events (table update, file arrival), so the quality gate belongs in the job's task graph, not in the trigger.
• Put expectations with ON VIOLATION FAIL UPDATE in a separate validation pipeline and make the downstream task depend on the validation task. If the validation update fails, the downstream task does not run.
• To branch on a computed value such as a bad-record count, use an If/else condition task that reads a task value.
• Expectations enforce data quality, not orchestration: a validation table does not make other datasets wait for it.

How do I trigger a job only when new data passes my quality checks?

Start the job from the data, then gate the downstream work on a validation step inside the job. The documented trigger types include scheduled, table update, file arrival and continuous. These react to time, table updates or new files, not to data quality: a table update trigger, for example, starts a run when a monitored table is updated. The quality gate is a task dependency, so the downstream task runs only if the validation task succeeds.

Check	Where to run it	How the gate works
Record-level rules, such as nulls or ranges	Pipeline expectations	drop removes bad rows, and fail stops the update
Dataset-level rules, such as row counts or key uniqueness	A validation table in a separate pipeline	The downstream task depends on the validation task
A computed metric, such as a bad-record count	A notebook task plus an If/else task	A task value drives the branch

What can pipeline expectations do, and what can't they do?

Expectations are optional clauses on pipeline materialized views, streaming tables and views. Each applies a SQL Boolean check to every record. In Python, import the module with from pyspark import pipelines as dp.

Action	SQL syntax	Python syntax	Result
warn (default)	EXPECT	dp.expect	Invalid records are written to the target
drop	EXPECT ... ON VIOLATION DROP ROW	dp.expect_or_drop	Invalid records are dropped before the write
fail	EXPECT ... ON VIOLATION FAIL UPDATE	dp.expect_or_fail	Invalid records prevent the update from succeeding

Expectations decide which records reach a dataset. They do not decide whether other parts of the pipeline run. Reading a validation table from another dataset does not make that dataset wait for the result, so a failed validation does not stop downstream tables from updating. In a triggered pipeline, a failed flow rolls back only its own update while the other flows continue. In a continuous pipeline, a failed expectation stops the flow and all dependent flows.

How do I gate a downstream task on a validation pipeline?

The Databricks docs say to split the validation logic and the downstream work into separate pipelines and orchestrate them with a job. Because a pipeline task fails when its update fails, a downstream task that depends on it does not run unless validation succeeds.

CREATE OR REFRESH MATERIALIZED VIEW orders_validation(
  CONSTRAINT no_missing_ids EXPECT (missing_ids = 0) ON VIOLATION FAIL UPDATE
) AS SELECT COUNT(*) AS missing_ids FROM main.raw.orders WHERE order_id IS NULL;

resources:
  jobs:
    validated_load:
      name: validated-load
      trigger:
        table_update:
          table_names:
            - main.raw.orders
      tasks:
        - task_key: validate_orders
          pipeline_task:
            pipeline_id: ${resources.pipelines.validate_orders.id}
        - task_key: load_marts
          depends_on:
            - task_key: validate_orders
          pipeline_task:
            pipeline_id: ${resources.pipelines.load_marts.id}

The default Run if condition is All succeeded, and a task whose condition is unmet is marked Upstream failed. A failed FAIL UPDATE expectation rolls the update back atomically, so the table that carries the expectation is not updated by that run. If upstream tables update in batches, set Wait after last change in seconds on the trigger so validation sees the whole batch.

How do I branch on a quality metric instead?

Use an If/else condition task when the decision is a computed value. A notebook task counts bad records and calls dbutils.jobs.taskValues.set(key="bad_records", value=n). The condition task reads it with {{tasks.check_quality.values.bad_records}}:

- task_key: has_bad_records
  depends_on:
    - task_key: check_quality
  condition_task:
    op: GREATER_THAN
    left: '{{tasks.check_quality.values.bad_records}}'
    right: '0'
- task_key: load_marts
  depends_on:
    - task_key: has_bad_records
      outcome: 'false'

EQUAL_TO and NOT_EQUAL compare strings, so 12.0 == 12 is false. GREATER_THAN and the other ordering operators compare numbers. A condition task needs no cluster and does not support retries or notifications.

What happens when the gate stops a run?

Fix the pipeline code to handle the invalid data, then re-run. For a job with several tasks, you can re-run only the unsuccessful tasks. To act on a failure, give a task run_if: AT_LEAST_ONE_FAILED or run_if: ALL_DONE, or add job or task notifications.

FAQs

Can I drop bad records instead of stopping the run?

Yes. dp.expect_or_drop drops invalid records and logs the dropped count with the dataset metrics. You can also quarantine invalid records without failing or dropping data.

Can a constraint query another table or call a service?

No. A constraint cannot contain custom Python functions, external service calls or subqueries that reference other tables. Compute the check in a validation table, then apply expect_or_fail to it.

Where do I see the quality results?

The pipeline UI shows a Data quality tab for warn and drop actions, and you can query the pipeline event log. A fail action records no metrics, because the update fails.

What permissions does the table update trigger need?

The principal in the job's Run as field needs SELECT on every table the trigger monitors, plus USE CATALOG and USE SCHEMA on its catalog and schema. Without SELECT, the trigger fails to evaluate and the job does not run.