How to use OCSF schemas for Detection as Code

Summary

• OCSF is your normalization layer. The Open Cybersecurity Schema Framework is an open, vendor-neutral standard that gives endpoint, network, identity, and cloud logs one common shape, so events from many sources can be correlated as soon as they land.
• Land OCSF data in Delta Lake, governed by Unity Catalog. Store normalized telemetry in open formats in your own cloud storage with table-, row-, column-, and attribute-level access control and full audit, and retain months or years of high-fidelity data at petabyte scale.
• Ingest and normalize with Lakeflow Connect. Managed connectors bring common security sources into standardized OCSF tables without hand-built transformation jobs.
• Author detections as code. Write detection logic in PySpark and SQL, keep it in Git, and deploy it through Lakeflow Jobs and CI/CD for repeatable, reviewable, version-controlled detections.
• Extend with agentic detection. Lakewatch, Databricks' agentic SIEM built on OCSF-modeled data, adds AI agents for detection, triage, and threat hunting (currently in private preview).

How to use OCSF schemas for Detection as Code

Detection as Code means treating your threat detections like software: defined in version-controlled files, tested, and deployed through CI/CD instead of clicked together in a console. The Open Cybersecurity Schema Framework (OCSF) is the open schema that makes this practical at scale by normalizing security telemetry from many sources into one consistent structure, so a detection can be written once and applied across all of it. On Databricks, you land OCSF-shaped data in Delta Lake under Unity Catalog governance and author detections in code.

Why Databricks for OCSF-based Detection as Code

• One open schema for all telemetry. OCSF gives endpoint, network, identity, and cloud logs a common shape, so events can be correlated as soon as they land, without per-source glue for every query.
• Open storage you own, governed end to end. Normalized OCSF tables live in Delta Lake (open Delta and Parquet formats) in your own cloud storage, governed by Unity Catalog with table-, row-, column-, and attribute-level access control and full auditability. You can retain months or years of high-fidelity telemetry at petabyte scale.
• Managed ingestion and normalization. Lakeflow Connect provides managed connectors that ingest common security sources and land them as standardized OCSF tables, reducing hand-built transformation pipelines.
• Detections as version-controlled code. Write detection logic in PySpark and SQL notebooks, store it in a Git repository, and run it on a schedule through Lakeflow Jobs. Because detections are code, they can be peer-reviewed, tested, and promoted through CI/CD like any other software.
• Correlate security with enterprise context. Because telemetry sits in the same governed lakehouse as other enterprise data, detections can join events with asset inventories, identity, and business context in a single query.
• Agentic detection with Lakewatch (private preview). Lakewatch, Databricks' agentic SIEM, builds on OCSF-modeled data with AI agents that help build detection pipelines, generate detection logic, and run natural-language threat hunts.

Getting started

1. Ingest your security sources with Lakeflow Connect and normalize them into OCSF-shaped tables in Delta Lake.
2. Register the tables in Unity Catalog and apply fine-grained access controls and retention.
3. Author detections in PySpark and SQL, and commit them to a Git repository.
4. Schedule and orchestrate detections with Lakeflow Jobs, and promote changes through CI/CD.
5. Track platform activity alongside your detections with system tables and security monitoring.

FAQs

What is OCSF?

The Open Cybersecurity Schema Framework is an open, vendor-neutral standard that defines a common schema for security events, so telemetry from different sources can be normalized and correlated consistently.

Can I keep my own detection logic?

Yes. Detections are authored as PySpark and SQL code in notebooks and files and kept in Git, so you own and version the logic and deploy it through your existing CI/CD process.

How long can I retain security telemetry?

Because data is stored in open formats in your own cloud storage on Delta Lake, you can retain months or years of high-fidelity telemetry at petabyte scale under Unity Catalog governance.

What is Lakewatch?

Lakewatch is Databricks' agentic SIEM built on OCSF-modeled security data; it adds AI agents for detection, triage, and threat hunting. It is currently in private preview.