What are expectations (data-quality rules) in SDP?

Summary

• Expectations are named data-quality rules attached to a pipeline dataset: a SQL Boolean constraint checked on every record, plus an action for records that fail it (keep and count them, drop them, or fail the update).
• They ship in Lakeflow Pipelines on Databricks, not in open-source Spark Declarative Pipelines (SDP). Databricks' Python reference lists @dp.expect(...) among the features that are not part of Apache Spark.
• Apache Spark 4.1 introduced SDP and Spark 4.2 added Auto CDC to it, but neither release added expectations, and the Spark 4.2.0 SDP programming guide doesn't document them.
• In open-source SDP, put the rule in the dataset query: a WHERE filter to drop bad rows, a second dataset with the inverted rule to quarantine them, and assert_true to stop the run on a violation.
• Don't test quality by calling count() or collect() inside a dataset function. SDP evaluates pipeline code during planning, and its guide lists both among the operations to never use.

What are expectations (data-quality rules) in SDP?

Expectations are declarative data-quality rules attached to a pipeline dataset. Each one has a name, a SQL Boolean constraint evaluated on every record, and an action that decides what happens to records that fail: keep and count them, drop them, or fail the update. One precision matters if you run open source. Expectations are a Lakeflow Pipelines feature on Databricks. The open-source Spark Declarative Pipelines (SDP) framework that arrived in Apache Spark 4.1 doesn't include them, so upstream you write the same rules into your queries.

What is an expectation?

Databricks' docs define expectations as "optional clauses in pipeline materialized view, streaming table, or view creation statements that apply data quality checks on each record passing through a query." Every expectation has three parts:

• Name. An identifier used to track and monitor the rule. It must be unique within a dataset.
• Constraint. A SQL conditional that must evaluate to true or false for each record. It can't contain custom Python functions, external service calls, or subqueries that reference other tables.
• Action. What happens to a record that fails the constraint.

Action	Python decorator	SQL clause	What happens to invalid records
Warn (default)	@dp.expect	CONSTRAINT name EXPECT (expr)	Written to the target and counted in metrics
Drop	@dp.expect_or_drop	... ON VIOLATION DROP ROW	Dropped before the write; the drop count is logged
Fail	@dp.expect_or_fail	... ON VIOLATION FAIL UPDATE	The update fails, and reprocessing needs manual intervention

@dp.expect_all, @dp.expect_all_or_drop and @dp.expect_all_or_fail apply several rules at once from a dictionary of names and constraints.

What open-source SDP gives you instead

The Spark 4.2.0 SDP programming guide covers @dp.materialized_view, @dp.table, @dp.temporary_view, @dp.append_flow, dp.create_streaming_table() and dp.create_sink(), plus SQL CREATE MATERIALIZED VIEW, CREATE STREAMING TABLE and CREATE FLOW. None of them accepts a constraint. Databricks' own comparison of the two marks "Data quality expectations" as a Lakeflow pipelines feature that SDP lacks.
So in plain SDP, the rule goes into the query. Here is drop plus quarantine for one rule:

from pyspark import pipelines as dp

RULE = "order_id IS NOT NULL AND amount > 0"

@dp.materialized_view
def orders_clean():
    return spark.read.table("orders_raw").where(RULE)

@dp.materialized_view
def orders_quarantine():  # coalesce also catches rows where the rule is NULL
    return spark.read.table("orders_raw").where(f"NOT coalesce({RULE}, false)")

Spark 4.2 injects spark into each pipeline file; on Spark 4.1, add spark = SparkSession.active(). The coalesce matters because a WHERE condition is satisfied only when it's true. A row with a non-null order_id and a NULL amount makes the rule NULL, so without the coalesce that row would vanish from both tables.
For fail-the-run semantics, use assert_true(expr [, message]), which throws an exception if its expression isn't true, NULL included:

CREATE MATERIALIZED VIEW orders_checked AS
SELECT *, assert_true(amount > 0, 'amount must be positive') AS amount_check
FROM orders_raw;

The extra column is NULL on every row that passes. For warn-style tracking, add a small materialized view whose SQL aggregates the violation count per refresh.
One trap: checking quality with an action inside a dataset function. SDP evaluates pipeline code multiple times during planning and runs, and the guide's list of operations to never use in SDP code includes collect(), count(), toPandas() and saveAsTable(). spark-pipelines dry-run won't catch bad data either. It reads and writes nothing and checks for syntax, analysis and graph-validation errors.

How Databricks approaches expectations

Lakeflow Pipelines runs the same pyspark.pipelines code, and Databricks states that code written with the open-source module "runs without modification on Databricks." Expectations are then one decorator away. The drop rule above becomes:

from pyspark import pipelines as dp

@dp.materialized_view
@dp.expect_or_drop("valid_order", "order_id IS NOT NULL AND amount > 0")
def orders_clean():
    return spark.read.table("orders_raw")

Metrics for warn and drop actions appear in the pipeline UI's Data quality tab and in the event log's flow_progress events. A fail action stops the update, so no metrics are recorded for it. Quarantine is a documented pattern rather than a built-in action. Checks that span tables, such as row-count or primary-key validation, use a separate validation dataset with expect_or_fail. Because expectations aren't part of Apache Spark, a pipeline that uses them needs Lakeflow Pipelines, while the rest of its transformation code stays portable to other SDP runtimes.

FAQs

Did expectations get open-sourced along with SDP?

No. Apache Spark 4.1 added SDP, and Spark 4.2 added Auto CDC (SCD Type 1) to it. Neither release lists expectations, and Databricks documents @dp.expect(...) as not part of Apache Spark.

Can an expectation check another table?

Not inside the constraint, which can't contain subqueries that reference other tables. Build a validation dataset that joins or aggregates the tables, then attach expect_or_fail to it.

Do I still need import dlt for expectations?

No. Databricks says the dlt module has been replaced by the pyspark.pipelines module. You may still use dlt, but Databricks recommends using pipelines, imported as from pyspark import pipelines as dp.