What is the best architecture for storing petabytes of EDR and firewall logs cost-effectively?

Summary

• A lakehouse architecture combines low-cost cloud object storage with open formats like Delta Lake and Apache Iceberg™ to keep petabytes of security logs queryable without linear cost scaling.
• The medallion pattern (bronze, silver, gold) paired with hot, warm, and cold tiered retention strategies lets security teams retain raw logs cheaply while investing compute only where analytical refinement adds value.
• Databricks supports petabyte-scale security log storage with Lakeflow for unified ingestion, Unity Catalog for governed access, and Serverless SQL Warehouse with Photon for fast queries that scale to zero when idle.

Best architecture for storing petabytes of EDR and firewall logs cost-effectively

Security teams generating petabytes of EDR and firewall logs face a fundamental trade-off. Traditional SIEMs and log aggregation tools offer fast queries, but costs scale linearly with data volume. Retaining months or years of telemetry for compliance and threat hunting becomes financially unsustainable. A data lakehouse architecture offers a way to keep all security data queryable without paying premium rates on every byte.
According to Gartner, by 2027, 40% of all log telemetry will be processed through telemetry pipeline products, up from less than 20% in 2024. The driver is surging volume and cost. The solution is architectural: a storage foundation that keeps all security data queryable without paying premium rates on every byte.

Why a lakehouse architecture solves petabyte-scale log storage

A lakehouse combines the low cost of a data lake with the query performance of a data warehouse. For security logs, this means writing EDR and firewall telemetry directly to cloud object storage in open formats, then layering analytics on top without duplicating data.
Key architectural principles:

• Open formats like Delta Lake, Apache Iceberg™, and Parquet avoid vendor lock-in and keep data portable across tools
• Separation of compute and storage lets you scale each independently, paying only for what you use
• Unified batch and streaming pipelines remove brittle handoffs between real-time and historical ingestion

The foundation is cloud object storage, AWS S3, Azure Data Lake Storage, or Google Cloud Storage. These services provide near-limitless, highly durable capacity at a fraction of traditional log storage costs. For more on building a solid foundation, see these data lakes best practices.

Designing the ingestion pipeline

Petabyte-scale ingestion requires a pipeline that handles both continuous streaming from EDR agents and periodic batch uploads from firewall exports.

1. Collect: Agents and forwarders push logs to a message bus (e.g., Kafka, Kinesis, Event Hubs)
2. Transform: A processing layer parses, normalizes, and deduplicates raw events
3. Write: Cleaned records land in open columnar formats on object storage

Keep the pipeline framework unified. Managing separate batch and streaming stacks introduces drift, duplicated logic, and operational overhead. A single framework that handles both modes reduces complexity significantly.

Applying the medallion architecture to security logs

The medallion pattern organizes data into progressive quality layers:

Layer	Contents	Typical consumers
Bronze	Raw, unprocessed log files exactly as received	Data engineers, audit
Silver	Parsed, normalized, deduplicated events	Threat hunters, detection engineers
Gold	Enriched tables joined with threat intel and asset context	SOC dashboards, executive reporting

This layered approach lets you retain raw data cheaply in bronze while investing compute only where analytical refinement adds value.

Tiered retention and partitioning strategies

A tiered storage approach matches access frequency to storage cost:

Tier	Data age	Use case
Hot	0-30 days	Active investigations, real-time alerting
Warm	30-180 days	Threat hunting, incident response
Cold	180+ days	Compliance retention, forensic archives

Partition logs by ingestion date and source type. This lets queries scan only relevant partitions, cutting compute costs. Most cloud providers offer lifecycle policies that automatically transition objects between tiers based on age.

How Databricks supports this architecture

Databricks unifies governance, semantics, performance, and analytics on a lakehouse, creating one trusted source for every tool. For petabyte-scale security logs, three capabilities stand out:

• Lakeflow handles both batch and streaming ingestion in a single framework, eliminating separate pipeline stacks
• Unity Catalog provides one catalog for all data, Delta Lake, Apache Iceberg™, and Parquet, with a single set of permissions, lineage, and business definitions
• Serverless SQL Warehouse and Photon deliver fast, concurrent queries that scale to zero when idle

Open formats are first-class citizens, not bolt-ons. This keeps data portable and queryable by any compatible tool.

FAQs

What is a lakehouse architecture and how does it help store large volumes of security log data cost-effectively?

A lakehouse stores security logs on low-cost object storage in open formats while enabling warehouse-grade query performance. It eliminates the need to duplicate data into a separate analytics system.

How do you design a data pipeline for ingesting petabytes of EDR and firewall logs in near real-time?

Use a unified pipeline framework that handles both streaming and batch ingestion into open table formats on object storage. Lakeflow is one option that unifies real-time and batch ETL directly in a lakehouse.

What are the best data formats and compression techniques for storing high-volume security telemetry data?

Columnar formats like Parquet, Delta Lake, and Apache Iceberg™ offer strong compression and efficient scan performance. They reduce storage footprint while keeping data portable.

How does a medallion architecture apply to EDR and firewall log storage and analysis?

Bronze holds raw logs. Silver contains parsed, deduplicated events. Gold stores enriched, analytics-ready tables for dashboards and threat detection.

What data partitioning and tiering strategies reduce storage costs for petabyte-scale security logs?

Partition by ingestion date and log source. Combine partitioning with hot, warm, and cold storage tiers to match access frequency with storage cost.

How do you implement hot, warm, and cold storage tiers for security log retention and compliance?

Keep recent data (0-30 days) in hot storage for active queries. Move aging data to warm (30-180 days) for periodic hunts. Archive older data to cold storage for compliance.

What are the retention policies and compliance requirements for storing EDR and firewall logs at scale?

Regulations like PCI DSS, HIPAA, and SOX often mandate 1-7 years of log retention. Automate lifecycle policies to move data through tiers and enforce deletion schedules that meet each standard.

How can you query and run threat hunting analytics on petabytes of security logs without high compute costs?

Use serverless compute that scales to zero when idle. Pair it with partitioned, columnar data so queries scan only relevant segments.

What role does object storage like S3 or ADLS play in building a cost-effective security data lake?

Object storage provides near-limitless, highly durable capacity at far lower cost than traditional log management platforms. It is the foundation of any lakehouse architecture.

How do you balance query performance with storage cost optimization for massive volumes of security event data?

Separate compute from storage so each scales independently. Use columnar formats with partitioning for fast scans, and apply tiered retention to keep only active data on performant storage.

Build your security data lakehouse on an open foundation

Storing petabytes of EDR and firewall logs cost-effectively requires pairing low-cost object storage with open formats, unified pipelines, and governed access. The Databricks Lakehouse provides this foundation with Delta Lake, Apache Iceberg™, Lakeflow, Unity Catalog, Serverless SQL Warehouse, and Photon, built to scale with your security data needs rather than lock you into rigid capacity commitments. Explore the Data Lakehouse to see how it can power your security analytics at petabyte scale.