How to keep security logs for years without spending too much

Summary

• Archive infrequently-accessed logs to cheaper storage tiers (cloud object storage) after a retention window, significantly reducing long-term storage costs.
• Use a lakehouse to unify log ingestion, tagging, and lifecycle policies in a single system, avoiding separate data warehouses or SIEM appliances.
• Compress and partition logs by time and severity to optimize both storage and query performance, keeping hot logs fast and cold logs queryable.
• Apply tiered retention policies: hot storage for recent logs, cold storage for compliance holds, with automatic lifecycle rules that move data without manual intervention.

How to keep security logs for years without spending too much
Keeping security logs for years is a compliance requirement, but storing gigabytes or terabytes of log data at high performance tiers gets expensive fast. The solution is a tiered storage strategy: keep hot logs (recent, frequently queried) in fast, indexed storage, then move older logs to cheaper archival tiers while keeping them queryable.
A lakehouse approach unifies log ingestion, retention policies, and cost controls in one system. Instead of managing separate log databases, data warehouses, and archive buckets, you route all security events into a single Delta Lake table, partition by date and severity, and apply automatic lifecycle rules to move cold data to cost-effective object storage.

What are tiered retention policies?

Tiered retention means applying different storage and access patterns based on how old data is. Recent logs stay in fast, indexed storage for quick searches during investigations. Older logs move to cheaper storage tiers but remain queryable. Logs beyond the compliance requirement can be compressed and moved to long-term archival storage or deleted.

Key capabilities to evaluate

Capability	Why it matters
Automated tiering	Manually moving logs is error-prone and expensive. Automatic lifecycle rules move data based on age without manual intervention.
Unified ingestion	A single entry point for all log sources (firewalls, DNS, endpoints, cloud API logs) simplifies operations and reduces visibility gaps.
Partitioning by time and severity	Query recent or high-severity logs directly instead of scanning all data.
Compression	Reduces storage footprint of older logs significantly without sacrificing queryability.
SQL-based access	Query archived logs in place without copying or restoring, keeping costs predictable.
Audit trail	Immutable records of what was accessed, when, and by whom ensure compliance and forensics.

How Databricks approaches long-term log retention

Lakewatch (in preview) is built to handle security log ingestion, retention, and analysis at scale. It provides:

• Auto Loader integration: Stream security events from cloud log services (AWS CloudTrail, Azure audit logs, or custom webhooks) directly into Delta Lake tables.
• Lifecycle management: Define retention schedules per log type. Logs older than the retention window automatically move from managed tables to cloud object storage (S3, Azure Blob, GCS), reducing storage cost while keeping them queryable.
• Partitioning and clustering: Partition logs by date, severity, and source. Query only the partitions you need.
• Compression: Delta Lake's default compression significantly reduces the disk footprint of archived logs.
• Unified SQL access: Query hot and cold logs with the same SQL; the system handles retrieval from the appropriate tier.

This approach avoids separate data warehouses or SIEM appliances, cutting infrastructure costs and reducing operational complexity.

Use cases

Financial services and regulated industries: Keep audit logs queryable for the full compliance period without the cost of keeping everything in high-performance storage. Run forensics queries on archived logs when investigating past transactions.
Large enterprises: Ingest logs from thousands of endpoints, applications, and cloud services into one place. Apply retention policies consistently across the entire organization.
Startups scaling fast: Start with hot storage, move logs to cold tiers automatically as they age, and pay only for what you use.

FAQs

How much can tiered storage save?

Tiered storage significantly reduces long-term log storage costs. The exact savings depend on your query patterns, log volume, and how quickly you move logs to cold storage.

Can I query archived logs as fast as hot logs?

Archived logs are queryable with SQL even when stored in object storage. For compliance audits and forensics, retrieval times are acceptable. For real-time monitoring, keep hot logs in fast storage.

What happens if I need a log that's been archived?

Archived logs remain accessible via SQL. The system fetches them on demand from cold storage. You don't need to manually restore or copy them.

How do I ensure archived logs meet compliance requirements?

Define retention policies that match your regulatory requirements (GDPR, SOC 2, HIPAA, etc.). Lakewatch's audit trail records all access, creating a forensic record of who accessed what and when.

Can I delete logs after the retention period?

Yes. Retention policies can include automatic deletion after the required hold period. Make sure your policy is documented and approved by legal and compliance teams.