Step-by-step guide to onboarding cloud logs (AWS, Azure, GCP) into Lakewatch

Summary

• Lakewatch ingests security logs from AWS, Azure, and GCP cloud storage through configurable datasources that authenticate using Unity Catalog credentials.
• Onboarding involves connecting to your cloud storage (S3 for AWS, ADLS for Azure, GCS for GCP), configuring access permissions, and defining which log streams to ingest.
• The process requires cross-cloud IAM setup: AWS IAM roles, Azure RBAC or shared access keys, and GCP service account permissions to read logs from your cloud accounts.
• Lakewatch consolidates logs into a unified schema, enabling you to apply security rules, create alerts, and investigate events across all cloud sources in one platform.

Step-by-step guide to onboarding cloud logs (AWS, Azure, GCP) into Lakewatch

Lakewatch is a unified security monitoring and incident response platform that centralizes logs from your cloud infrastructure. To use Lakewatch for security analysis, you need to ingest logs from AWS, Azure, or GCP. The process involves configuring cloud storage access, setting up authentication credentials, and pointing Lakewatch to your log streams.

What is cloud logs onboarding?

Cloud logs onboarding is the process of connecting Lakewatch to your cloud provider's log storage and configuring which logs to ingest. Instead of managing logs in separate cloud consoles, Lakewatch brings them into a centralized lakehouse, where you can query, analyze, and investigate security events alongside all your other data. You're essentially telling Lakewatch where your logs live in the cloud and granting it read access.

High-level onboarding steps

Phase	What happens	Owner
1. Identify log sources	Determine which logs you want to ingest: VPC Flow Logs, CloudTrail (AWS); Activity Logs, Diagnostic Logs (Azure); VPC Flow Logs, Cloud Audit Logs (GCP).	Security team
2. Configure cloud access	Create IAM roles/policies (AWS), RBAC assignments (Azure), or service account permissions (GCP) to allow Lakewatch to read logs from cloud storage.	Cloud operations team
3. Set up cloud credentials	Create Unity Catalog credentials that store the cloud authentication tokens, allowing Lakewatch to securely authenticate to your cloud account.	Data platform team
4. Configure datasources	In Lakewatch, create a datasource that points to your cloud storage location and attach the UC credentials for authentication.	Lakewatch administrator
5. Define log ingestion	Specify which log types, time ranges, and storage paths to ingest into Lakewatch's unified schema.	Security operations team
6. Validate ingestion	Verify logs are arriving in Lakewatch and write queries against the ingested data to confirm completeness.	Security team

Cloud-specific prerequisites

AWS:

• CloudTrail, VPC Flow Logs, or other log sources writing to an S3 bucket in your AWS account
• IAM role with permissions to read from the S3 bucket and describe log files
• AWS access keys (or temporary credentials) configured in a Unity Catalog credential

Azure:

• Activity Logs or Diagnostic Logs exported to an Azure Data Lake Storage (ADLS) Gen2 container or Blob Storage
• Either a shared access key (SAS) or managed identity with Storage Blob Data Reader role
• Azure credentials configured in Unity Catalog

GCP:

• Cloud Audit Logs or VPC Flow Logs exported to a Google Cloud Storage (GCS) bucket
• A GCP service account with Storage Object Viewer role on the bucket
• Service account key configured in a Unity Catalog credential

How Lakewatch approaches cloud logs ingestion

Lakewatch connects to your cloud storage through datasources, which are logical connections that combine authentication (UC credentials) with a storage path. When you create a datasource, you specify where your logs are stored and how to authenticate. Lakewatch then reads logs from that location on a schedule or on-demand, parses them into a standardized schema, and loads them into your lakehouse.
You control the ingestion scope: you can ingest all logs from a storage location, filter by time range, or specify particular log types. Once ingested, logs are stored as Delta tables in your lakehouse, where you can query them with SQL, apply Lakewatch's built-in detection rules, and correlate them with your operational and application data.
For organizations spanning multiple clouds, you can configure multiple datasources (one per cloud) and Lakewatch will ingest and unify them into a single security monitoring interface.

Key configuration considerations

Storage Path: Know the exact S3 prefix (AWS), ADLS container path (Azure), or GCS bucket path (GCP) where your logs are written.
Permissions: Ensure the cloud identity (IAM role, service account, managed identity) has minimal permissions following least-privilege principles.
Credential Rotation: Store credentials in Unity Catalog and keep access keys/tokens rotated per your organization's security policy.
Log Format: Confirm the log format (JSON, CSV, or cloud-provider-specific format) so Lakewatch can parse correctly.
Retention Policy: Define how long logs should be retained in your lakehouse to balance compliance, cost, and query performance.

Common use cases after onboarding

Security Investigations: Query ingested logs to investigate suspicious activity, track user actions, and correlate events across cloud accounts.
Compliance Reporting: Generate audit reports from centralized logs to meet regulatory requirements (SOC 2, HIPAA, PCI DSS).
Threat Detection: Apply Lakewatch's built-in detection rules to ingested logs to identify anomalies and security events automatically.
Incident Response: Use Lakewatch cases and notebooks to investigate and remediate security incidents without switching between cloud consoles.

FAQs

How long does cloud logs onboarding take?

The time depends on your cloud setup complexity and approval processes. Cloud access configuration can take hours to days if you need IAM approvals. Once credentials are set up, creating a datasource and verifying ingestion typically takes under an hour.

Can I ingest logs from multiple cloud accounts at once?

Yes. Create separate datasources for each cloud account using different UC credentials. Lakewatch can ingest from and query across all of them in a single interface.

What happens if logs are delayed or stop arriving?

Lakewatch will alert you if ingestion fails or if no new logs arrive within an expected window. Check datasource configuration, verify cloud credentials are still valid, and confirm IAM permissions haven't been revoked.

Do I need to pre-process logs before ingesting them?

No. Lakewatch handles parsing and schema mapping. However, you should verify that logs are being written to the cloud storage location you specified before creating the datasource.