What tools can automatically detect and remediate data quality issues in my warehouse?

Summary

• Databricks detects data quality issues with data quality monitoring. Anomaly detection (Public Preview) checks the freshness and completeness of tables in a schema, and data profiling tracks summary statistics and drift over time.
• It enforces rules at write time with Delta table constraints (NOT NULL, CHECK) and with Lakeflow Pipelines expectations that keep, drop, or fail on invalid records.
• The remediation that is safe to automate is prevention and isolation: reject or drop bad rows, or quarantine them for review. Deciding the correct value stays with a person.
• Anomaly detection reports problems. It does not modify the tables it monitors.
• Route findings to people with data quality monitoring alerts (Beta), Databricks SQL alerts, and job notifications.

What tools can automatically detect and remediate data quality issues in my warehouse?

Detection can be automatic, and so can prevention. Correcting bad data is different. The tools below detect problems, reject or isolate bad records, and re-run work, but deciding the right value stays with your team.

Which Databricks tools detect and enforce data quality?

Tool	What it does	Stage or note
Anomaly detection	Learns each table's commit history and row counts, then marks tables stale or incomplete. Results appear in Catalog Explorer and in system.data_quality_monitoring.table_results.	Public Preview. Views and foreign tables are not supported.
Data profiling	Computes summary statistics over time, including drift against an optional baseline table, into metric tables and a dashboard.	Snapshot, time series, or inference profiles.
Lakeflow Pipelines expectations	SQL conditions checked on each record. The action is warn (the default), drop, or fail the update.	Metrics for warn and drop actions appear in the pipeline UI and event log.
Delta table constraints	NOT NULL and CHECK constraints are enforced: a write that violates one fails the transaction.	Primary and foreign keys are informational only.
Databricks SQL alerts	Run a query on a schedule and notify you when a condition on the result is met.	Can be paired with data quality monitoring.

How do anomaly detection and data profiling work?

You enable anomaly detection on a schema, which needs the MANAGE privilege on the schema or its catalog. Databricks then runs a background job that checks freshness and completeness. A table is stale when a commit is unusually late, and incomplete when the rows committed over the last 24 hours fall below the predicted range. New tables show Training status until the model has enough history. For an unhealthy table, the Root Cause column points to the most likely reason, such as a job failure or an upstream table health issue. The job runs on serverless compute.
Data profiling works differently. You attach a profile to a table, and it computes metrics over time into a profile metrics table and a drift metrics table, both Delta tables. You can build dashboards and alerts on them.

What can be remediated automatically?

Reject bad writes. A violated CHECK or NOT NULL constraint fails the transaction:

ALTER TABLE main.default.people_demo ADD CONSTRAINT dateWithinRange CHECK (birthDate > '1900-01-01');

Drop or fail on bad records in a pipeline. This expectation drops rows that fail the check, and the count of dropped records is logged with the dataset metrics:

CREATE OR REFRESH STREAMING TABLE customers(
  CONSTRAINT valid_customer_age EXPECT (age BETWEEN 0 AND 120) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM(datasets.samples.raw_customers);

With ON VIOLATION FAIL UPDATE instead, the update fails and rolls back, and manual intervention is required before reprocessing.
Quarantine instead of dropping. The documented quarantine pattern combines expectations with a temporary table and views, so valid and invalid records follow separate processing paths.
Re-run what failed. Task retries in Lakeflow Jobs run automatically. Repair run re-runs failed tasks and their dependents on demand, and Refresh failed tables does the same for failed pipeline tables and their downstream dependencies.
Expectations control which records reach a dataset, not which other tables run. To stop downstream processing when validation fails, put validation and downstream work in separate pipelines and make the downstream task depend on the validation task in a job.

How do I get alerted?

• Data quality monitoring alert rules (Beta) notify workspace users by email, and can send to Slack or webhook destinations, when a monitored table in a catalog or schema becomes unhealthy.
• Databricks SQL alerts evaluate a query on a schedule, and the docs list pairing them with anomaly detection and profiles.
• Job notifications cover failed runs of the jobs that load your tables.

FAQs

Do I have to write a rule for every check?

No for freshness and completeness, because anomaly detection builds per-table models from history. Yes for business rules, because expectations and constraints are SQL conditions you write. Expectation constraints cannot contain custom Python functions, external service calls, or subqueries that reference other tables.

Can anomaly detection fix the data?

No. It marks table health and points to a likely root cause. It does not modify the tables it monitors.

How fast does it detect problems?

Databricks scans each table at the same frequency it is updated, and intelligent scanning prioritizes high-impact tables and scans less critical ones less often. Health indicators for a table skipped in the first scan can take up to two weeks to appear.

Where do I see results across all schemas?

Use the Data Quality Monitoring dashboard, or query system.data_quality_monitoring.table_results. By default only account admins can access that table, and they can grant access to others.