What tool can catch usage that never gets billed to the customer?

Summary

• The system.billing.usage table provides the authoritative record of all billable usage across your Databricks account, typically available within 12 hours, making it the primary tool for detecting discrepancies between actual usage and billed amounts.
• Usage Dashboards (available to admins for import into Unity Catalog workspaces) break down costs by product, SKU, and custom tags, allowing you to identify expensive or unexpected usage patterns at a glance.
• The system.access.audit table tracks all user actions and service operations, enabling forensic analysis of who used what resources and when, complementing billing data with activity context.
• Multiple system tables work together: system.compute.node_timeline, system.query.history, and job execution tables capture compute utilization separately from billing records, revealing gaps between usage and billing.

What tool can catch usage that never gets billed to the customer?
Usage that doesn't appear in billing can indicate configuration errors, system bugs, or unintended free usage. The primary tool for catching this is the system.billing.usage table, the authoritative record of all billable events in your Databricks account. Records become available within 12 hours, allowing billing teams and FinOps engineers to audit billing accuracy and identify anomalies. When combined with usage dashboards, audit logs, and additional system tables, you gain visibility into the full lifecycle of resource consumption.

What are Databricks system tables?

System tables are built-in, read-only tables that capture operational metrics from your Databricks platform. They're part of a "Databricks-hosted analytical store" of account data and require Unity Catalog enablement. System tables are optimized for retrospective analysis (not real-time transactional operations): querying historical patterns, troubleshooting resource consumption, and validating billing accuracy. They retain data for specific windows (billing usage retains 365 days; compute node timelines retain 90 days).

System tables for usage and billing observability

System Table	What It Tracks	Key Use Case
system.billing.usage	All billable usage: compute time, storage, networking, SKU type, workspace, resource IDs, timestamps	Validate billing accuracy; detect unbilled usage or billing errors
system.billing.list_prices	Historical SKU pricing over time (indefinite retention)	Understand cost trends; audit price changes
system.access.audit	All user actions, service operations, identity, IP, session, timestamps	Forensic analysis of who used resources; activity audit trail
system.compute.node_timeline	Compute resource utilization metrics (90-day retention)	Understand actual compute consumption patterns
system.query.history	Queries executed on SQL warehouses and serverless compute	Trace query-level resource usage
system.jobs.job_run_timeline and system.jobs.task_timeline	Job execution times, task runtimes	Track job-level resource consumption

How Databricks approaches usage observability

Databricks provides layered visibility into resource consumption. The system.billing.usage table is the foundation: it includes billing_origin_product fields that distinguish between products sharing the same SKU (for example, data quality monitoring and predictive optimization both bill under serverless jobs but are tracked separately). Admins can query this table directly or use Usage Dashboards (importable, customizable visualizations that break costs by product, SKU, workspace, and tags, with object-level expense analysis and cost forecasting).
For reconciliation and forensics, the system.access.audit table records every service call: who initiated it, when, from which IP, and whether it succeeded or failed. This context helps distinguish between intended usage, accidental consumption, and potential security events. Together, these tables enable you to answer: "This usage appeared in billing; who triggered it?" and "This resource ran; why wasn't it billed?"

Use cases

FinOps and cost allocation: Finance teams use system.billing.usage to validate invoices against actual usage, allocate costs to teams via custom tags, and forecast spend by product.
Billing audit and reconciliation: When a customer disputes a charge, query system.billing.usage for the exact events and system.access.audit to confirm who triggered them.
Cost optimization: Combine system.compute.node_timeline with system.billing.usage to identify clusters or warehouses running longer than expected, or queries consuming unexpected resource volume.
Compliance and forensics: Use audit logs to prove (or disprove) that specific actions occurred on specific dates at specific times.

FAQs

Why might usage not appear in billing?

Billing records are generated from usage events; gaps typically indicate free usage (trial accounts, new workspace setup periods, or specific promotional windows), configuration issues, or system errors. Query system.billing.usage to confirm events were recorded; if they're absent, the usage wasn't metered.

How soon do billing records appear in system.billing.usage?

Records are typically available within 12 hours. For real-time cost forecasting, usage dashboards provide near-current visibility on recent activity (though preliminary, before finalized billing records are computed).

Can I use system tables to detect security breaches or unauthorized usage?

Yes. The system.access.audit table logs all actions by user, IP, and timestamp. Query it to find unusual spike in activity, access from unexpected IP addresses, or actions by users who shouldn't have permissions. Correlate with system.billing.usage to measure the blast radius of a compromise.

What happens if there's a billing error in system.billing.usage?

The billing system uses retraction and restatement records to correct errors. These appear as separate rows in system.billing.usage (marked by negative quantities or special flags), allowing you to reconcile to the final billed amount.