Monitor ingestion data

This guide is for security engineers who want to monitor data ingestion health and troubleshoot issues within Google Security Operations. It explains how to use the Health Hub dashboard and configure Cloud Monitoring alerts to track the status of data sources and parsers. By following this monitoring workflow, engineers can quickly identify, diagnose, and remediate data pipeline problems, distinguishing between user-fixable (actionable) and support-required (non-actionable) issues. Successful completion ensures reliable data flow, which is crucial for effective security operations and analysis.

The Health Hubdashboard is the central place in Google SecOps to monitor all configured data sources. This guide details how to interpret the metrics and take action on any failures.

Common use cases

This section covers common use cases.

Proactive health check

• Objective: Regularly review the status and health of all configured data sources.
• Value: Early detection of potential ingestion problems before they impact security visibility.

Alert response

• Objective: Investigate and fix a data source or parser issue flagged by an automated alert.
• Value: Minimize data loss or delays, ensuring data is available for timely threat detection and response.

Key terminology

• Health Hub:The central dashboard within Google SecOps for monitoring the status and health of all configured data sources and parsers.
• Cloud Monitoring:Google Cloud service used to create alerting policies based on metrics, including those from Chronicle ingestion.
• Actionable issue:An ingestion problem that the user can typically resolve themselves through configuration changes (for example, updating credentials).
• Non-actionable issue:An ingestion problem that likely requires assistance from Google support to resolve (for example, an internal system error).
• Parser:A component that normalizes raw log data into the Unified Data Model (UDM) structure.

Before you begin

• Permissions: Ensure you have the necessary IAM roles and permissions to access the Google SecOps instance, view the Health Hub, and configure alerts in Cloud Monitoring.

Monitor and troubleshoot data ingestion

This section details how to monitor data ingestion.

Monitor using the Health Hub dashboard

1. In the Google SecOps side navigation menu, click Health Hub.
2. Review the "Big Number" widgets for Failed Sourcesand Failed Parsersto identify components requiring immediate attention.
3. Inspect the Health Status by Data Sourcetable. Check the Latest Issue Detailscolumn for error descriptions, such as, "Config credential issue"or "Normalization issue".
4. Click the Edit Data Sourceor Edit Parserlinks provided in the table to navigate directly to the respective configuration pages for remediation.
5. Check timestamps, such as Last Event Timeand Last Ingested, to verify data was ingested as expected.
6. After applying a fix, monitor the Health Hub dashboard for the specific data source or parser.

Set up automated ingestion alerts

1. In the Health Hub dashboard, click the Set Up Alertslink, which directs you to the Cloud Monitoring interface.
2. Create an alerting policy:
   • Select Metrics : Choose metrics under Chronicle Collector > Ingestion, such as Total ingested log countor Total ingested log size.
   • Add filters for collector\\_id or log\\_type to narrow the alert scope to specific sources.
3. To detect silent forwarders, select Metric absenceas the condition type. Configure it to trigger an alert if logs stop flowing for a specified duration (for example, 60 minutes).

Troubleshooting

Investigate common issues

• Pipeline latency investigation:A significant delay between the Last Event Timeand Last Ingestedtimestamp indicates potential latency. Health Hub exposes the 95th percentile of this delta. High values suggest pipeline latency, while normal values might mean the source is sending historical data.
• Ingestion surges or drops:The system uses z-score standardization to flag anomalies. A drop is flagged if both daily and weekly standardized differences are less than -1.645.
• Parsing failures:An alert is triggered if the proportion of parser errors relative to total ingested events increases by 5 percentage points or more compared to the previous day. Investigate the parser configuration using the link in Health Hub.

Latency, service quota, and limits

• Data Refresh:Information on the Health Hub and Data Ingestion dashboards refreshes approximately every 15 minutes.

Error remediation

For a full list of error messages and solutions, see Troubleshoot ingestion .

Need more help? Get answers from Community members and Google SecOps professionals.