Business case for offloading high-volume logs to a security lakehouse

Summary

• Offloading high-volume logs to a security lakehouse lets you retain full-fidelity telemetry at petabyte scale in open formats instead of sampling or dropping data to fit a fixed-capacity tool.
• On Databricks Lakewatch, storage is decoupled from compute: keep petabytes of logs in your own cloud object storage and pay only for the compute you run.
• Logs land in open standards (OCSF, Delta, Parquet, SQL) that you own and can query across your AI and analytics tooling, with no proprietary format lock-in.
• Retaining months or years of hot, queryable telemetry gives analysts and AI agents the historical depth for deeper forensics and longer-horizon threat hunting.
• Security data can be correlated with business context and governed end-to-end in Unity Catalog. Lakewatch is available in Private Preview.

What is the business case for offloading high-volume logs to a security lakehouse?

High-volume sources — network, endpoint, cloud, and identity logs — can quickly exceed the volume a fixed-capacity tool can economically hold. Teams end up sampling, dropping, or aging out data they later need for investigations. Offloading those logs to a security lakehouse changes the trade-off: you retain full-fidelity telemetry at petabyte scale in open formats, and run analytics and detections on it when you need them.

Why Databricks Lakewatch for offloading high-volume logs

Databricks Lakewatch is an agentic SIEM built on the open Databricks Data Intelligence Platform.

• Decouple storage from compute. Keep petabytes of full-fidelity telemetry in your own cloud object storage and provision serverless compute only when you query or run detections, for fine-grained cost control.
• Full-fidelity retention. Retain months or years of hot, queryable telemetry rather than sampling or aging data out, so investigations and hunts work from the complete record.
• Open formats you own. Store logs on open standards — OCSF, Spark, Delta, Parquet, and SQL — so data stays instantly accessible across your AI and analytics tooling and is never locked into a proprietary format.
• Automated OCSF normalization. Map endpoint, network, identity, and cloud logs to a common schema automatically for immediate correlation.
• Unified context. Correlate security events with enterprise context such as asset inventories and business data in one governed environment, breaking down silos between security and the rest of the business.
• Agent-ready and governed. Governed, real-time data is immediately actionable for AI agents that triage, hunt, and generate detection-as-code, and Unity Catalog logs every query and action for audit.

Lakewatch is available in Private Preview.

Getting started

• Read Databricks Announces Lakewatch: a new, agentic SIEM for the architecture and how logs are stored and analyzed.
• Read Alert fatigue is a business risk for the operational case behind agentic security operations.
• Explore the Data Intelligence Platform to see how open storage, compute, and governance fit together.

FAQs

What does offloading logs to a security lakehouse mean?

It means routing high-volume telemetry into an open lakehouse — cloud object storage in open formats — where you retain full-fidelity data at petabyte scale and run analytics and detections on demand.

Do I have to sample or drop logs to control cost?

No. Because storage is decoupled from compute, you can retain full-fidelity telemetry in cloud object storage and provision compute only when you query or run detections.

Do I keep ownership of the data?

Yes. Logs are stored in open formats (OCSF, Delta, Parquet, SQL) in your own cloud storage and governed in Unity Catalog, so you retain ownership and full audit control.