How to migrate legacy SIEM detections to Detection as Code in Lakewatch

Summary

• Express detections as code. In Databricks Lakewatch (Private Preview), Detection as Code lets you author detection rules in YAML with embedded SQL, or in Python, then version-control, unit-test, and deploy them through automated CI/CD pipelines.
• Re-author existing rules against normalized data. Bring existing detection logic in by expressing each rule as SQL or Python that runs against OCSF-normalized security telemetry in Lakewatch.
• Backtest before you deploy. Validate each detection against months or years of historical telemetry so you can confirm the logic and tune false positive rates before it goes live.
• Use AI-assisted authoring. Genie can translate natural-language intent into SQL and help draft new detections or refine existing rules to reduce false positives.
• Run at petabyte scale. Detections run as code directly against petabyte-scale telemetry, with OCSF-normalized gold tables providing a consistent foundation for both out-of-the-box and custom detections.

How to migrate legacy SIEM detections to Detection as Code in Lakewatch

Databricks Lakewatch, a security data lakehouse currently in Private Preview, treats detections as code. Instead of maintaining rules in a rule-authoring UI, security engineers write detections in familiar languages, keep them under version control, test them, and ship them through CI/CD. Migrating your existing detections means re-expressing each rule as code that runs against OCSF-normalized telemetry in Lakewatch, then validating it against historical data before deployment. See Databricks announces Lakewatch, a new agentic SIEM.

Why Databricks Lakewatch Detection as Code

• Author in the languages your team knows. Detection rules are defined in YAML with embedded SQL queries, or as Python, so engineers can use SQL for threshold-based rules and Python for behavioral analytics or machine-learning-based detections.
• Software-engineering rigor. Security teams write, unit-test, version-control, and deploy detections through automated CI/CD pipelines, bringing standard engineering practices and governance to threat detection.
• Backtesting against historical data. A core capability is backtesting a detection against historical telemetry, so you can validate the logic, measure false positive rates, and tune rules for accuracy without impacting production.
• An OCSF-normalized foundation. Detections run against data normalized to the Open Cybersecurity Schema Framework (OCSF). OCSF-normalized gold tables are the optimal foundation for out-of-the-box detections and analytics, while raw bronze data remains available for rules and alerts.
• AI-assisted authoring. Genie can translate natural-language questions into SQL and help author net-new detections from threat intelligence, and it can assist in modifying existing rules to reduce false positives.
• Runs at petabyte scale with agentic follow-through. Detections run as code directly against petabytes of telemetry, and native agentic capabilities can trigger when threats are detected, correlating signals across the security lakehouse. Genie Agents lets analysts query the same data in plain English for threat hunting.

Getting started

A practical migration follows the detection development lifecycle:

• Inventory your detections. List your existing rules and prioritize the high-value, high-volume ones to move first.
• Re-express each rule as code. Write the detection as SQL or Python against OCSF-normalized data in Lakewatch, choosing SQL for straightforward rules and Python for behavioral or ML-based logic.
• Put detections under version control and CI/CD. Unit-test and deploy them through automated pipelines so changes are reviewed and reproducible.
• Backtest and tune. Run each detection against historical telemetry to validate logic and reduce false positives before deployment.
• Deploy and monitor. Ship the detection and let it run continuously against live telemetry, using Genie for investigation.
• Because Lakewatch is in Private Preview, contact Databricks to discuss enrollment and to scope a migration. Read Databricks announces Lakewatch, a new agentic SIEM for the architecture.

FAQs

What is Detection as Code in Lakewatch?

Detection as Code means authoring detection rules in YAML with embedded SQL or in Python, then version-controlling, unit-testing, and deploying them through CI/CD pipelines that run against OCSF-normalized telemetry in Lakewatch.

How do I move an existing SIEM detection into Lakewatch?

Re-express the rule as SQL or Python against OCSF-normalized data, place it under version control and CI/CD, and backtest it against historical telemetry to validate the logic and tune false positive rates before deploying.

Can I test a detection before it goes live?

Yes. Backtesting runs a detection against historical telemetry so you can validate the logic and measure false positive rates without impacting production.

Is Lakewatch generally available?

Lakewatch is currently in Private Preview. Contact Databricks to discuss enrollment and to scope a migration.