My data freshness is bad, how do I triage?

Summary

• Triage freshness backward from the stale table: confirm its last write, find the job or pipeline that writes it, then check whether that producer started late, failed, ran slowly, or waited on stale inputs.
• On Databricks, table history shows the last write and the job behind it, lineage shows upstream producers, and run records in the Jobs UI and system.lakeflow tables show state, timing, and termination codes.
• Data quality monitoring (Public Preview) can flag a stale table and point to a likely upstream cause, and duration warnings tell you when a run exceeds its expected time.
• Check the consumer last. A dashboard can look stale because its refresh schedule is paused while the table is current.

My data freshness is bad, how do I triage?

Work backward from the stale table. Confirm when it last received a write and which job wrote it, then ask four questions about that producer: did it start, did it succeed, was it slow, and were its inputs fresh? Check the consumer last, because a stale dashboard does not always mean a stale table.

Where does freshness usually break?

Symptom	Likely cause	Where to check
No recent write	Trigger did not fire, schedule paused, or the run was skipped	Job triggers and run list; termination code MAX_CONCURRENT_RUNS_EXCEEDED
Run failed	Task error, cluster or cloud failure, permissions	Task run details and termination_code; Repair run
Run finished late	Queue, setup, or execution time grew	Matrix and timeline views; duration columns in system.lakeflow.job_run_timeline
Pipeline update slow	Materialized view fell back to a full recompute, streaming backlog, or datasets queued behind the concurrency limit	Event log and backlog metrics
Upstream table stale	The delay started upstream	Lineage and the data quality monitoring root cause
Table fresh, dashboard stale	Dashboard schedule paused or too infrequent	Dashboard schedules and run history

How do I triage step by step?

1. Confirm the last write. Every operation that modifies a Delta table creates a version, including OPTIMIZE, so read the operation and timestamp, not only the latest version. The job column is filled for commits written from a Lakeflow job.

SELECT version, timestamp, operation, job
FROM (DESCRIBE HISTORY main.sales.orders)
ORDER BY version DESC
LIMIT 5

2. Find the producer and its inputs. Lineage shows which jobs and notebooks transform a table and which tables feed it. Check each input's freshness before blaming the producer.
3. Check whether it started. File arrival triggers make a best-effort check for new files every minute. A table update trigger can wait by design: Minimum time between triggers and Wait after last change both delay a run, and the trigger fails to evaluate if the Run as principal lacks SELECT on a monitored table. By default only one run of a job can be active, and runs beyond the configured maximum concurrency are skipped.
4. Check whether it was slow. Compare recent runs with their baseline:

SELECT to_date(period_start_time) AS run_date,
       avg(queue_duration_seconds) AS avg_queue_s,
       avg(setup_duration_seconds) AS avg_setup_s,
       avg(execution_duration_seconds) AS avg_execution_s
FROM system.lakeflow.job_run_timeline
WHERE job_id = '<job_id>'
  AND result_state IS NOT NULL
  AND period_start_time > CURRENT_TIMESTAMP() - INTERVAL 14 DAYS
GROUP BY ALL
ORDER BY run_date

These tables are typically available within one hour, so use the Jobs UI for the newest runs. For pipelines, the docs name three degradation signals: update duration trending up, materialized views falling back to full recomputes, and extra datasets waiting behind the pipeline's concurrency limit. For streaming sources, backlog metrics such as numBytesOutstanding (Delta and Auto Loader) or maxOffsetsBehindLatest (Kafka) show whether a query is falling behind.
5. Check for dropped data. Expectations that drop rows appear in the event log's data quality metrics, and a watermark can drop records that arrive after its threshold.
6. Check the consumer. Paused dashboard schedules do not update the dashboard, and the schedule list shows the outcome of recent runs.

How do I catch it earlier?

Anomaly detection in data quality monitoring (Public Preview) builds a per-table model of commit times and marks a table stale when a commit is unusually late. It uses lineage to name a likely cause, such as a job failure or an upstream table. Set an expected completion time on a job to get a duration warning notification, and configure failure notifications. Streaming backlog thresholds on tasks (Public Preview) can also notify you, though not for pipeline tasks. A Databricks SQL alert can check max(event_time) on a schedule.

FAQs

What if the delay is in a source system I do not own?

Use lineage and the data quality monitoring root cause to see which upstream job or table is late, then raise it with its owner. Alert on that upstream table's freshness so you hear about the delay before your stakeholders do.

Does a successful run mean the data is fresh?

Not by itself. Run state says the task finished, while table history says whether a write happened and when.

Should I auto-retry failed runs?

Retries depend on the job type. For most job configurations the default is no retries, serverless jobs auto-optimize retries, continuous jobs use exponential backoff, and scheduled pipeline updates retry recoverable failures while manual updates do not. Retry transient failures, and alert on repeated ones instead of hiding them.