How to store all company security logs in one place

Summary

• One unified repository, years of history. Store all security logs (firewall, endpoint, cloud, application) in a single Delta Lake table instead of scattered SIEM silos, and keep years of searchable history without the cost of specialized log storage.
• Cost-effective retention at scale. Centralized log storage in the lakehouse eliminates the per-gigabyte SIEM overages and allows you to retain raw logs indefinitely at object-storage costs.
• Query any log in real time or retrospectively. With all logs in Delta format with ACID guarantees, you can run ad-hoc queries across sources, correlate events across systems, and backfill security checks when new threats emerge.
• Security and compliance aligned. Keep logs in a governed catalog with role-based access control, tagging, and lineage so your security, compliance, and data teams can audit together without creating data duplicates.

How to store all company security logs in one place

Store all company security logs in one Delta Lake table within your lakehouse. This gives you a unified, searchable repository that replaces disconnected SIEM silos, keeps years of raw logs accessible, and scales to large event volumes without per-event storage costs. The approach works for firewall logs, endpoint detection and response (EDR), cloud audit logs, application logs, and custom security sources.
The advantage over point solutions is simplicity and cost. Instead of ingesting logs into multiple specialized systems, you stream them once into Delta Lake, apply a consistent schema, and make them queryable across your entire organization. Compliance teams can run lineage and access audits. Security teams can correlate events across sources. Data teams can integrate logs into analytics pipelines. It's a single source of truth.

Why centralize security logs

Scattered logs create friction. Firewall events live in one SIEM, endpoint logs in another, cloud audit logs in a third. When you need to investigate a breach or satisfy an audit, you spend weeks reconciling data across systems and re-ingesting it into your analytics stack. Centralization solves this:

• Cost. Specialized SIEM platforms charge per gigabyte ingested and per gigabyte stored monthly; those costs accelerate as log volumes grow. The lakehouse stores logs at object-storage rates, which cost significantly less per TB for long-term retention.
• Query breadth. A single Delta Lake table lets you correlate events across all systems. You can find the first sign of a compromise across firewall and endpoint logs with one query instead of exporting data from two tools and joining manually.
• Retention. SIEM platforms make long-term retention prohibitively expensive. The lakehouse makes it practical to keep years of raw logs for retrospective analysis, forensics, and compliance.
• Integration. Once logs are in Delta, they're available to your data platform. Your data team can join logs with other security signals, build ML models to detect anomalies, and integrate insights into your broader analytics.

Architecture: from sources to Delta Lake

A typical flow looks like this:

Step	Component	Details
Ingest	Auto Loader or Zerobus	Stream logs from your sources (firewall, cloud APIs, syslog, message queues) into a landing table in Delta. Auto Loader handles schema evolution automatically.
Parse	SQL or Spark job	Standardize log schemas: extract timestamp, source system, event type, user ID, resource, action, and result. Add metadata like the ingestion time and source system.
Store	Delta Lake	Write to a central, partitioned table (by date or source system). ACID transactions guarantee consistency.
Access	Unity Catalog	Grant role-based read access. Classify sensitive columns (source IP, user ID). Audit who queried what.
Query	SQL or Spark	Run ad-hoc investigations, build dashboards, or feed logs into threat detection pipelines.

Getting started

1. Choose your ingestion method. If logs are in S3, GCS, or Azure Blob Storage, use Auto Loader with cloudFiles.format="json" or "csv" depending on your log format. If logs stream from a message queue (Kafka, Pub/Sub), use a Structured Streaming pipeline. If logs come from syslog or an external API, Zerobus can standardize the flow.
2. Define a log schema. Create a table schema that captures the key fields from your sources: timestamp, source system, event type, user, resource, action, outcome, and any context (IP address, geolocation, severity). Use a union schema if different sources have different fields; use schema_of_string() for JSON logs with nested fields.
3. Set up a parsing layer. In the landing zone, store raw, unparsed logs. In a separate validated zone, parse and standardize them. This separation lets you preserve the original log if you need to reparse it later.
4. Integrate with threat detection. Once logs are in Delta, layer on real-time analytics. You can use Genie Agents to ask questions about log trends, detect spikes in failed logins, or identify unusual access patterns. You can feed processed logs into machine learning models to spot anomalies.
5. Govern with Unity Catalog. Register the table in Unity Catalog, classify columns with PII tags, and grant read access to security, compliance, and data teams. Use row-level or column-level access control if your organization has separate security domains.

FAQs

How do I ingest logs from multiple sources into one table?

Use Auto Loader if logs land in cloud storage, or Zerobus if they stream from an external API. Both can handle schema mismatches and format variations. Parse them into a common schema once they land in Delta.

How much history should I keep?

It depends on your compliance requirements and forensics needs. Keep recent logs (last 90 days) in a hot table for fast queries. Archive older logs to cheaper storage tiers while keeping them queryable through Delta time travel or a separate archive table. The lakehouse makes multi-year retention practical because the per-terabyte cost is low.

How do I correlate logs from my firewall, cloud, and endpoints?

Standardize on a common timestamp (preferably UTC), parse each source into a schema with source_system, event_type, user, resource, and action fields, and store them in one table. Use window functions to find related events: for example, find all events from the same user within a 5-minute window.

Can I use this for compliance audits?

Yes. Unity Catalog provides lineage and access audit logs. You can query who accessed the log table, when, and what queries they ran. Combine this with the raw security logs to show auditors a complete chain of evidence.