What are the best practices for data-quality expectations?

Summary

• An expectation is a named data-quality rule: a SQL Boolean constraint checked on every record, plus an action for records that fail it (keep and count, drop, or fail the update).
• Open-source Spark Declarative Pipelines (SDP) has no expectations API as of Spark 4.2.0; open-source projects use table constraints such as Delta Lake NOT NULL and CHECK, or filters in the dataset query.
• @dp.expect, @dp.expect_or_drop, @dp.expect_or_fail, the expect_all variants and the SQL CONSTRAINT ... EXPECT clause are Lakeflow Pipelines features on Databricks.
• Choose each action by the cost of bad data, keep rules in one shared table or module, and check row counts and key uniqueness in validation tables.
• Expectations don't orchestrate: to stop downstream work when a check fails, run validation as its own pipeline and make the downstream task depend on it.

What are the best practices for data-quality expectations?

Write each expectation as a small, named SQL Boolean rule, decide per rule whether a failing record is kept, dropped or stops the update, and store the rules in one place so every dataset applies the same logic. Then watch the pass, fail and drop counts.
In open-source Apache Spark, Spark Declarative Pipelines (SDP) doesn't include expectations. The Spark 4.2.0 SDP programming guide and the pyspark.pipelines API reference cover datasets and flows (@dp.table, @dp.materialized_view, @dp.temporary_view, @dp.append_flow, dp.create_streaming_table, dp.create_sink) and nothing about data quality. Expectation decorators and the SQL CONSTRAINT ... EXPECT clause belong to Lakeflow Pipelines, the Databricks product that extends and interoperates with SDP.

How do I enforce data quality in open-source Spark?

Use table constraints, filters or both. Delta Lake, an open-source table format, supports NOT NULL column constraints and CHECK constraints, a Boolean expression that must be true for each input row. When a write violates one, Delta Lake throws an InvariantViolationException and the new data isn't added. ALTER TABLE ... ADD CONSTRAINT first verifies that every existing row satisfies the rule, and adding a constraint upgrades the table's writer protocol version.

ALTER TABLE default.people10m ADD CONSTRAINT dateWithinRange CHECK (birthDate > '1900-01-01');
ALTER TABLE default.people10m DROP CONSTRAINT dateWithinRange;

To drop bad rows instead, filter them out in the dataset's query, and add a second dataset with the inverse condition to review the rejects.

What actions can an expectation take?

In Lakeflow Pipelines, each expectation has a name, a constraint and an action:

Action	SQL syntax	Python syntax	What happens to an invalid record
warn (default)	EXPECT	dp.expect	Written to the target; valid and invalid counts are logged
drop	EXPECT ... ON VIOLATION DROP ROW	dp.expect_or_drop	Dropped before the write; the dropped count is logged
fail	EXPECT ... ON VIOLATION FAIL UPDATE	dp.expect_or_fail	The update fails, and a table update rolls back atomically

Only Python can group rules under one action: expect_all, expect_all_or_drop and expect_all_or_fail take a dictionary of expectation names and constraints. Expectation decorators go after @dp.table, @dp.materialized_view or @dp.temporary_view and before the function:

from pyspark import pipelines as dp

valid_orders = {
    "valid_order_id": "order_id IS NOT NULL",
    "non_negative_price": "price >= 0",
}

@dp.table
@dp.expect_all_or_drop(valid_orders)
@dp.expect_or_fail("valid_purchase_date", "date <= current_date()")
def clean_orders():
    return spark.readStream.table("raw_orders")

In SQL, one drop rule reads CONSTRAINT non_negative_price EXPECT (price >= 0) ON VIOLATION DROP ROW inside CREATE OR REFRESH STREAMING TABLE, with several constraints separated by commas.

Which practices hold up in production?

1. Name rules for what they check. Names must be unique within a dataset and are how you track rules in metrics, so prefer valid_customer_age to check_1.
2. Keep constraints to plain SQL. No custom Python functions, external service calls or subqueries that reference other tables.
3. Match the action to the cost of bad data. Warn for visibility without losing rows, drop when a bad row is useless downstream, and fail only when invalid records are unacceptable: a failed update needs a code fix before re-running.
4. Quarantine when you need to see bad rows. The documented pattern adds an is_quarantined flag, applies expect_all so metrics still record, and splits valid and invalid rows into separate views.
5. Store rules apart from pipeline code. Databricks recommends keeping expectation definitions in a rules table or Python module and tagging related rules. SQL can't load rules dynamically from a file.
6. Check whole datasets with validation tables. For row counts, missing records or primary-key uniqueness, define a separate materialized view (for example, num_entries = 1 after grouping by key) with expect_or_fail.
7. Don't use expectations to gate other tables. A validation table doesn't make downstream datasets wait, and in a triggered pipeline a failed flow doesn't fail the other flows. Put validation in its own pipeline and make the downstream task depend on it in a job.

How Databricks approaches expectations

Lakeflow Pipelines records metrics for warn and drop actions (a fail action stops the update, so it records none). You see them on a dataset's Data quality tab in the pipeline UI, or in the event log, a hidden Delta table by default. SELECT * FROM event_log(<pipelineId>) returns flow_progress events whose details:flow_progress.data_quality.expectations object holds passed_records and failed_records per rule. Expectations work on streaming tables, materialized views and temporary views, but not on sinks or with AUTO CDC FROM SNAPSHOT.

FAQs

Does open-source Spark Declarative Pipelines support expectations?

Not as of Spark 4.2.0: neither the SDP programming guide nor the pyspark.pipelines API reference lists an expectation decorator. Use table constraints such as Delta Lake CHECK, or filters.

How do I find the record that broke a fail expectation?

Read the error. Lakeflow Pipelines reports an EXPECTATION_VIOLATION error that names the violated expectation and, for many queries, shows the input record that caused it.

Can I reuse one rule set across tables?

Yes. Pass the same dictionary, loaded from a rules table or a Python module, to @dp.expect_all_or_drop or another expect_all decorator on each similar dataset.