One of my team's biggest challenges is keeping track of SLAs for continuous jobs, what engine and technology makes this the easiest for us to do?

Summary

• SLA tracking for continuous jobs means watching availability, backlog, data quality, and table freshness, because an always-running job never finishes.
• On Databricks, run always-on streams as Lakeflow Jobs in continuous mode, which Databricks recommends for always-on streaming and which retries the whole job on failure.
• Job notifications cover failures, duration warnings, and streaming backlog thresholds, and the pipeline event log records backlog and expectation metrics you can alert on with Databricks SQL alerts.
• Anomaly detection in data quality monitoring (Public Preview) marks a table stale when a commit is unusually late for its history.

One of my team's biggest challenges is keeping track of SLAs for continuous jobs, what engine and technology makes this the easiest for us to do?

Keep the streaming engine, the orchestrator, and the SLA telemetry on one platform. On Databricks, run the stream with Lakeflow Pipelines or Structured Streaming, orchestrate it as a continuous Lakeflow Job, and track SLAs with signals the platform already records: job notifications, backlog and data quality metrics in the pipeline event log, run history in the system.lakeflow system tables, and table freshness checks. Define each SLA once and alert on the metric instead of polling job status.

Why are SLAs harder to track for continuous jobs than for batch jobs?

A batch job has an end and a status, so "finished before the business day" is easy to check. A continuous job never finishes. It can show as running while it falls behind its source or passes bad records downstream. So SLA tracking shifts from job status to three questions:

• Is it keeping up? Backlog, the data still waiting at the source, tells you whether latency is drifting.
• Is the output fresh? Consumers query tables, not jobs, so measure freshness at the table.
• Is the data correct? A job can meet its latency target and still publish invalid records.

Which SLA signals should you track, and where does Databricks expose them?

SLA signal	What it tells you	Where to get it on Databricks
Availability	Whether the stream is up or restarting	Continuous trigger retries, Failure notifications
Backlog	Whether the job keeps up with its source	Streaming backlog notifications; backlog_bytes, backlog_records, backlog_seconds, and backlog_files in the pipeline event log
Run duration	Whether triggered work finishes in its window	Duration warning notifications; system.lakeflow.job_run_timeline
Data quality	Whether invalid records reach consumers	Expectations, with passed and failed record counts
Freshness and completeness	Whether the output table updated as expected	Anomaly detection in data quality monitoring (Public Preview)

How do you set up SLA tracking on Databricks?

1. Run the stream as a continuous job. A continuous job keeps one run active and retries the entire job on failure using an exponential backoff algorithm. For pipelines, Databricks recommends a continuous job over setting the pipeline's own mode to continuous.
2. Add notifications for each SLA event. Select Start, Success, Failure, Duration warning, or Streaming backlog, and send them to email or to admin-configured destinations such as Slack, PagerDuty, or a webhook. Duration warnings need an expected duration set on the job. Streaming backlog notifications need the Jobs service to track the active streaming query, so don't use awaitTermination() in jobs.
3. Query backlog history from the pipeline event log. Each pipeline tracks its backlog in the details:flow_progress.metrics.backlog_bytes object. After you create an event_log_raw view on the event log, this query from the Databricks docs returns backlog for the latest update:

with latest_update as (
  SELECT origin.update_id as id
    FROM event_log_raw
    WHERE event_type = 'create_update'
    ORDER BY timestamp DESC
    limit 1
)
SELECT
  timestamp,
  Double(details :flow_progress.metrics.backlog_bytes) as backlog
FROM
  event_log_raw,
  latest_update
WHERE
  event_type ='flow_progress'
  AND
  origin.update_id = latest_update.id;

Backlog metrics might not be available for every source type and Databricks Runtime version.
4. Alert on the query. Databricks SQL alerts run a query on a schedule and notify you when a condition you define is met, so any SLA query can become an alert.
5. Watch freshness at the table. Anomaly detection (Public Preview) predicts each table's next commit from its history and marks the table stale if a commit is unusually late. It also marks a table incomplete when recent row counts fall below the expected range.

Which engine makes SLA tracking easiest?

Lakeflow Pipelines suits teams that want declarative tables: data quality checks, progress, and backlog land in the event log without extra code, and serverless pipelines autoscale as workload volume changes. Structured Streaming in a job suits teams that want code-level control: name each stream with .queryName(<query-name>), read backlog metrics such as numBytesOutstanding for Delta and Auto Loader or avgOffsetsBehindLatest for Kafka, and push metrics to external services with StreamingQueryListener. The job run page also charts backlog through streaming observability (Public Preview).

What are common questions about SLAs for continuous jobs?

How do I measure end-to-end latency for a stream?

Carry an event timestamp from the source and compare it with the time each row lands in the output table. Where supported, the backlog_seconds event log field reports the maximum backlog seconds across a flow's inputs.

Why don't backlog notifications fire for my pipeline task?

They aren't supported for pipeline tasks. Query the backlog fields in the pipeline event log and alert on them with a Databricks SQL alert.

Can the platform absorb load spikes on its own?

Partly. Enhanced autoscaling adds and removes compute as workload volume changes, and it's always on for serverless pipelines. It won't fix an upstream delay or a bad record, so alert first and investigate.