Console
    Start free

    Ingest and parse log data

    Supported in:

    This guide helps Security Engineersefficiently ingest and parse log data using Google Security Operations. It explains how to consolidate logs from various sources, normalize them into the Unified Data Model (UDM), and make them available for threat detection and analysis. By following these methods, you can improve visibility across your environments, reduce operational latency, and ensure data is consistently formatted for searching and examination, ultimately enhancing your security posture.

    Google SecOps ingests telemetry from diverse sources, including multi-cloud, on-premises, and SaaS environments. It normalizes this data into UDM, enabling real-time threat detection and historical analysis.

    Common use cases

    The following use cases highlight how Google SecOps log ingestion and parsing capabilities can enhance your security operations.

    Consolidate fragmented data

    • Objective: Bring logs from all your security tools and environments (multi-cloud, on-premises, SaaS) into a single platform.
    • Value: Centralized analysis and visibility, breaking down data silos.

    Ensure schema consistency

    • Objective: Automatically normalize diverse log formats into the UDM.
    • Value: Enables consistent searching and rule application across all data sources. For example, a failed login event from AWS will have the same UDM structure as one from an on-premises Windows server.

    Reduce operational latency

    • Objective: Utilize direct ingestion methods for real-time data flow.
    • Value: Minimizes the time from event occurrence to detection and response.

    Key terminology

    • UDM : A standardized schema that Google SecOps uses to normalize log data from various sources. This allows for consistent analysis and threat detection across different log types and vendors.

    Before you begin

    • Permissions: Make sure you have the necessary Identity and Access Management (IAM) roles:

      • roles/chroniclesm.admin : Typically required at the organization level for multi-project ingestion.
      • roles/securitycenter.adminEditor : For some configurations.

      For more information, see Grant a single IAM role

    • Google Cloud: Google SecOps requires Security Command Center (Standard or Premium) for asset metadata ingestion.

    Configure log ingestion and parsing

    Choose the ingestion method based on your log sources. The primary methods are Direct Ingestion for Google Cloud, the Bindplane agent for on-premises or pre-processing needs, and Data Feeds for cloud-to-cloud or API integrations.

    Choose an ingestion method

    1. Direct ingestion (Google Cloud):This is the most common ingestion method for Google Cloud telemetry. It uses Cloud Logging filters to forward logs directly to Google SecOps.

      • Supported types:Cloud Audit Logs, Cloud DNS, Cloud NAT, Cloud Next Generation Firewall, Cloud IDS, Cloud Load Balancing, and more. For more information, see Ingest Google Cloud data to Google Security Operations .
      • Enablement:
        • Go to the Google SecOps Ingestion Settingsin the Google Cloud console.
        • Click the Sending data to Google Secopstoggle.
        • Configure Logging sinks and filters as needed.

    2. Bindplane agent:Use Bindplane for on-premises servers (Windows or Linux) or to filter or transform cloud data before it reaches Google SecOps.

      • Editions:
        • Google edition:All customers receive this edition.
        • Enterprise edition:Offers advanced filtering, PII masking, and more.
      • Architecture:Can run in Gateway mode , where multiple collectors forward logs to a central Bindplane gateway, simplifying firewall rules and offloading processing.
      • Enablement:Install the agent using the shell or PowerShell scripts that the Google SecOps console provides. For more information, see Use Bindplane with Google SecOps .

    3. Data feeds:Use Data Feeds for cloud-to-cloud ingestion from object stores or third-party APIs.

      • Supported sources:Includes Azure Event Hub, Amazon S3, Azure Blob Storage, Google Cloud Storage, Microsoft 365, CrowdStrike Falcon, and ~50 others. For more information, see Source Types
      • Security:Google Cloud Secret Manager securely stores credentials for data feeds.
      • Enablement:Configure feeds through the Google SecOps console, providing the necessary credentials and source details. For more information, see Feed management documentation .

    4. Other methods

      For details on these methods, refer to the specific documentation for each integration.

    General configuration steps

    1. Grant IAM permissions:Ensure the appropriate service accounts or user roles have roles/chroniclesm.admin and other necessary permissions, as described in Before you begin .
    2. Connect to source:
    3. Customize filters: Apply exclusion filters to reduce log volume and cost. For example, exclude high-volume, low-value operations.
    4. Monitor health:Use the Health Hub dashboard in the Google SecOps console to track ingestion rates, check for errors ( Anomalous Sources ), and identify parser issues.

    Troubleshooting

    This section helps you diagnose and resolve common issues with log ingestion and parsing.

    Latency, service quota, and limits

    • Ingestion delays:Large files (exceeding 5-10 GB) can cause ingestion delays. Split these files to avoid delays.
    • Encoding:Google SecOps only supports UTF-8 encoding for ingested logs.
    • Google Cloud direct ingestion:Google SecOps does not support custom Namespace or Label assignment during ingestion.
    • Data feed log line size:Google SecOps limits individual log lines within data feeds to 4 MB.
    • Burst limits:If ingestion rates exceed tenant burst limits, you might see a Threshold reachedstatus. If this happens, refine export filters or contact Google SecOps support for potential quota increases.

    Error remediation

    Symptom Potential cause Fix
    Missing identity relationships
    		 Context ingestion is desynchronized. To force a context refresh, disable and then re-enable direct Google Cloud ingestion.
    Threshold reached status
    		 Ingestion rate exceeds tenant burst limits. Refine export filters to exclude less critical logs. Contact Google SecOps support to request a quota increase, if necessary.
    Bindplane agent won't connect
    		 Your firewall is blocking required ports or endpoints. Ensure that firewall rules allow egress to malachiteingestion-pa.googleapis.com on port 443 . If you use Gateway Mode, ensure that your firewall allows internal traffic on port 3001 (default).
    Parser errors
    		 The log format changed, or you applied an incorrect parser. Review logs in Health Hub . Check if the default parser is appropriate, or if you need a custom parser. Validate that the log format meets parser expectations.

    Validation and testing

    • Use the Health Hub dashboard in the Google SecOps console to monitor ingestion rates, check for errors ("Anomalous Sources"), and identify parser issues. Health Hub is the primary tool to validate that data is flowing correctly and that Google SecOps parses it as expected.
    • Use UDM Search in Google SecOps to query for ingested logs and ensure that they are present and Google SecOps normalizes them correctly.

    Support

    Need more help? Get answers from Community members and Google SecOps professionals.

    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: