How does Detection as Code work in a security lakehouse?

Summary

• Detections are code. Security teams write detection rules as YAML with SQL queries or Python notebooks — open, standard technologies rather than a proprietary detection language.
• Software-engineering rigor. Rules are unit-tested, version-controlled, and deployed through automated CI/CD pipelines, and can be backtested against historical telemetry before going live.
• Runs where the data lives. Detections execute as code directly against petabytes of security telemetry stored in open formats and normalized to OCSF in the lakehouse.
• Broad source coverage. Normalized telemetry arrives through 100+ out-of-the-box connectors into the gold layer of a medallion architecture, governed by Unity Catalog.
• AI-assisted tuning. AI workflows help operationalize detection content, tune logic, and reduce noisy alerts. Lakewatch, the agentic SIEM built on this lakehouse, is in Private Preview.

How does Detection as Code work in a security lakehouse?

Detection as Code applies software-engineering practices — version control, testing, and CI/CD — to threat detection. In a security lakehouse, detection logic lives as code alongside the telemetry it runs on, so security engineering teams manage detections the way developers manage software. Lakewatch, Databricks' agentic SIEM built on the security lakehouse, is currently in Private Preview.

Why Databricks Lakewatch enables Detection as Code

• Author detections in open languages. Engineers define detection rules in YAML with SQL queries or in Python notebooks, using standard open technologies that integrate natively with the platform.
• Version control and testing. Detections are written, unit-tested, and version-controlled, then deployed through automated CI/CD pipelines — bringing DevOps rigor to detection engineering.
• Backtest before production. Because full-fidelity telemetry is retained in the lakehouse, teams can backtest new rules against years of historical data to validate them before deployment.
• Execute at scale on normalized data. Detections run as code directly against petabytes of telemetry, normalized to the Open Cybersecurity Schema Framework (OCSF) in the gold layer of a medallion architecture.
• Broad source coverage. Normalized security data flows in through 100+ out-of-the-box connectors across identity, endpoint, email, cloud, and network sources.
• AI-assisted operations. AI workflows help bring detection content from idea to orchestrated detection, continuously tune logic, and reduce false positives and noisy alerts.

Getting started

1. Store and normalize security telemetry to OCSF gold tables in the lakehouse, governed by Unity Catalog.
2. Author detection rules as YAML with SQL or as Python notebooks in version control.
3. Unit-test and backtest rules against historical data, then deploy through CI/CD.
4. Explore the Databricks cybersecurity solution and Lakewatch (Private Preview).

FAQs

What language are detections written in?

Detection rules are defined in YAML with SQL queries or in Python notebooks — open, standard technologies rather than a proprietary detection language.

How are detections tested and deployed?

They are unit-tested, version-controlled, and deployed via automated CI/CD pipelines, and can be backtested against years of historical telemetry retained in the lakehouse.

Where do detections run?

They run as code directly against security telemetry stored in open formats and normalized to OCSF in the lakehouse, so there is no need to move data to a separate engine.