How do I monitor a serverless application?

Summary

• Use built-in observability. Job run status, task-level metrics, run history, and alerts via email, Slack, and webhooks give you monitoring for serverless jobs and pipelines.
• Query system tables. System tables capture billing and usage, compute events, job and pipeline runs, and audit logs for account-wide observability and custom dashboards.
• Monitor Databricks Apps. Operational monitoring for Databricks Apps captures logs, traces, and metrics that you can analyze with SQL in Unity Catalog.
• Track cost. The serverless cost observability dashboard and Budget alerts help you monitor serverless usage and spend.
• Detect data issues. Lakehouse Monitoring tracks data and model quality on production tables and logs metrics for alerting.

How do I monitor a serverless application?

Monitoring a serverless application on Databricks combines built-in run observability, account-wide system tables, application-level telemetry, and cost tracking. Because Databricks manages the serverless compute, you focus on observing your workloads -- jobs, pipelines, apps, and model endpoints -- through the platform's native monitoring, alerting, and logging rather than on managing infrastructure metrics.

Why monitor serverless applications on the Databricks Platform

• Built-in run observability and alerts. Job monitoring and observability reports run status and task-level metrics with run history in the UI, and you can send notifications and alerts by email, Slack, and custom webhooks to diagnose failures and understand performance.
• Account-wide observability with system tables. system tables capture billing and usage, compute and warehouse events, job and pipeline run history, and audit logs. You can query them to build custom dashboards and set alerts for failures or cost anomalies.
• Application telemetry for Databricks Apps. operational monitoring for Databricks Apps captures logs, traces, and metrics as governed telemetry in Unity Catalog, so you can run SQL-native incident analysis under the same access controls as your other data.
• Cost and budget monitoring. The serverless cost observability dashboard and Budgets let you track serverless usage and spend, apply filters by team, project, or workspace, and get spend alerts.
• Data and model quality. Lakehouse Monitoring watches critical production tables to track metrics and detect anomalies, logging results into system tables you can alert on.
• Operational best practices. Follow the observability recommendations and set up monitoring, alerting, and logging from the Databricks Well-Architected guidance.

Getting started

• Turn on system tables for your metastore and build cost, performance, and reliability dashboards.
• Configure job monitoring and alerts for your serverless jobs and pipelines.
• Enable operational monitoring for Databricks Apps to capture app logs, traces, and metrics.
• Set Budgets and use the serverless cost observability dashboard to watch spend.
• Review Lakeflow Jobs reliable orchestration with unified observability.

FAQs

How do I monitor serverless jobs and pipelines?

Use built-in job monitoring for run status and task-level metrics, and configure email, Slack, or webhook alerts on failures. System tables add account-wide run history for custom dashboards.

How do I monitor a Databricks App?

Enable operational monitoring for Databricks Apps, which captures logs, traces, and metrics as governed telemetry in Unity Catalog for SQL-native analysis.

How do I track the cost of serverless workloads?

Query the system.billing.usage system table, use the serverless cost observability dashboard, and set Budgets with spend alerts filtered by team, project, or workspace.