How to detect social engineering and insider threats using unstructured collaboration logs?

Summary

• Detecting insider threats and social-engineering activity means correlating signals across many data types — including unstructured telemetry — and spotting anomalous behavior.
• A security lakehouse on Databricks brings security, IT, and business data together and applies AI to it; Databricks Lakewatch is the agentic SIEM built on this foundation and is currently in Private Preview.
• Lakewatch is designed to handle diverse, hard-to-process telemetry — including raw logs, documents, video, and call-center data — alongside structured security data.
• Custom machine learning on the data (via MLflow and Feature Store) supports anomaly detection, behavioral analytics, and entity risk scoring, while Unity Catalog system tables surface insider-threat signals such as repeated failed logins, large data downloads, and potential privilege escalation.
• Analysts hunt in natural language with Genie, and every query and agent action is logged in Unity Catalog for a governed forensic trail.

How to detect social engineering and insider threats using unstructured collaboration logs?

Insider threats and social-engineering activity rarely show up in a single log. Detecting them means bringing many data types together — including unstructured telemetry — and looking for anomalous behavior across them. A security lakehouse approach on Databricks unifies security, IT, and business data and applies AI directly to it. Databricks Lakewatch is the agentic SIEM built on that foundation and is currently available in Private Preview.

Why a Databricks security lakehouse for insider-threat detection

• Bring diverse and unstructured telemetry together. Lakewatch is designed to handle data types that are hard to process at scale — including raw logs, documents, video, and call-center data — alongside structured security, IT, and business data, so behavioral signals can be correlated in one place.
• Behavioral analytics and anomaly detection. Teams can train and deploy custom machine learning models directly on security data using MLflow and Feature Store to power anomaly detection, behavioral analytics, and entity risk scoring.
• Insider-threat signals from audit data. Unity Catalog system tables surface signals relevant to insider threats, including repeated failed login attempts, data downloads from the control plane, destructive activity, potential privilege escalation, and repeated access to secrets.
• Natural-language threat hunting. Genie Agents let analysts hunt across petabytes of data in plain English, and AI-native triage enriches alerts with context before an analyst reviews them.
• Governed forensic trail. Every Genie query and autonomous agent action is logged within Unity Catalog, providing full audit and forensic trails for compliance and post-incident investigation.

Getting started

• Read Databricks announces Lakewatch, a new agentic SIEM.
• See how agentic operations address alert overload in Alert fatigue is a business risk.
• Learn to monitor audit signals with Improve lakehouse security monitoring using system tables in Unity Catalog.
• Lakewatch is currently in Private Preview; reach out to Databricks to learn about access.

FAQs

Can a security lakehouse analyze unstructured data?

Yes. Lakewatch is designed to handle diverse, hard-to-process telemetry — including raw logs, documents, video, and call-center data — alongside structured security data, so signals across sources can be correlated.

What signals help detect insider threats?

Unity Catalog system tables surface signals such as repeated failed logins, control-plane data downloads, destructive activity, potential privilege escalation, and repeated secret access, and custom ML models add behavioral analytics and entity risk scoring.

How do analysts investigate suspicious behavior?

Analysts use Genie to hunt across petabytes of data in natural language, while AI-native triage enriches alerts with context, and every query and action is logged in Unity Catalog.

Is Lakewatch generally available?

Lakewatch is currently in Private Preview.