Ingestion metrics schema

component

STRING

Forwarder , is the ingestion service or pipeline type ingesting log entities into the platform.

collector_id

STRING

The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

log_type

STRING

The log source type identifying log entries in a batch. For example, WINDOWS_DNS .

input_type

STRING

This field is populated if the ingestion source is the Google SecOps forwarder. Based on the data that the forwarder sends, this field contains pcap , syslog , or splunk .

drop_reason_code

STRING

The reason for dropping the log.

last_heartbeat_time

TIMESTAMP

The last timestamp at which the forwarder or API feed was active, in microseconds. This field is populated if the ingestion source is the Google SecOps forwarder or Chronicle API feed.

When the feed is active, it populates the last_heartbeat_time field, and the log_count and log_volume fields remain empty.

log_volume

FLOAT64

The volume of logs during the interval, in bytes.

The log_volume field remains empty or is populated in the following cases:

• This field is populated when the Google SecOps forwarder or the feed sends data. The last_heartbeat_time field remains empty.
• If the feed is inactive, no entry is made in the ingestion metrics table.
• When a feed is active, the last_heartbeat_time , log_count , or log_volume field is populated.

drop_count

FLOAT64

The number of logs dropped for the customer.

log_count

FLOAT64

The number of logs ingested during the interval.

The log_count field remains empty or is populated in the following cases:

• This field is populated when the Google SecOps forwarder or the feed sends data. The last_heartbeat_time field remains empty.
• If the feed is inactive, no entry is made in the ingestion metrics table.
• When a feed is active, the last_heartbeat_time , log_count , or log_volume field is populated.

memory_used

FLOAT64

The percentage of memory used by the forwarder container.

disk_used

FLOAT64

Percentage of disk used by the forwarder container.

cpu_used

FLOAT64

The percentage of CPU used by the forwarder container.

buffer_used

FLOAT64

Percentage of buffer used by the forwarder buffer type.

buffer_type

STRING

The type of buffer used in the forwarder.

component

STRING

Normalizer , is the ingestion service or pipeline type ingesting log entities into the platform.

collector_id

STRING

The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

state

STRING

The final status of the event or log. The status is one of the following:

• parsed . The log is successfully parsed.
• validated . The log is successfully validated.
• failed_parsing . The log has parsing errors.
• failed_validation . The log has validation errors.
• failed_indexing . The log has batch indexing errors.

log_type

STRING

The log source type identifying log entries in a batch. For example, WINDOWS_DNS .

event_type

STRING

The event type determines which fields are included with the event. The event type includes values, such as EDR , WEBPROXY , and ASSET_CONTEXT .

drop_reason_code

STRING

The reason for dropping the log.

log_volume

FLOAT64

The total size of the log entries received for parsing, in bytes.

log_count

FLOAT64

The number of log entries received for parsing.

event_count

FLOAT64

The number of events generated during the interval.

latency_count

FLOAT64

The number of values in a latency distribution, of the difference in time between ingestion and normalization.

buckets

FLOAT64

The number of values in each bucket in a latency distribution, of the difference in time between ingestion and normalization.

bucketer_num_finite_buckets

FLOAT64

The number of buckets in a latency distribution, of the difference in time between ingestion and normalization.

bucketer_growth_factor

FLOAT64

The bucketer growth factor in a latency distribution, of the difference in time between ingestion and normalization.

bucketer_scale_factor

FLOAT64

The bucketer scale factor in a latency distribution, of the difference in time between ingestion and normalization.

latency_overflow

FLOAT64

The overflow bucket in a latency distribution, of the difference in time between ingestion and normalization.

latency_underflow

FLOAT64

The underflow bucket in a latency distribution, of the difference in time between ingestion and normalization.

last_event_time

TIMESTAMP

This field is in preview.
The epoch timestamp of the latest log entry that was successfully fetched from a source.

last_normalization_time

TIMESTAMP

This field is in preview.
The epoch timestamp when a log entry for a given log type was successfully parsed.

health_status

STRING

This field is in preview.
The overall health of the ingestion pipeline. Values: HEALTHY , ANOMALOUS , CRITICAL .

issue

STRING

This field is in preview.
The details explaining the specific issue found in the pipeline.

component

STRING

Out-of-band processor (Chronicle API feed), is the ingestion service or pipeline type ingesting log entities into the platform.

feed_id

STRING

The id of the specific Xenon/Gopher feed that a log belongs to.

log_type

STRING

The log source type identifying log entries in the batch. For example, WINDOWS_DNS .

last_heartbeat_time

TIMESTAMP

The epoch timestamp of the successful ingestion of the log entry, in seconds.

When the feed is active, the last_heartbeat_time field is populated, and the log_count and log_volume fields remain empty.

log_volume

FLOAT64

The size of the logs received in the out-of-band processor, in bytes.

The log_volume field remains empty or is populated in the following cases:

• This field is populated when the feed sends data. The last_heartbeat_time field remains empty.
• If the feed is inactive, no entry is made in the ingestion metrics table.
• When the feed is active, the last_heartbeat_time , log_count , or log_volume field is populated.

log_count

FLOAT64

The number of logs processed in the out-of-band processor.

The log_count field remains empty or is populated in the following cases:

• This field is populated when the feed sends data. The last_heartbeat_time field remains empty.
• If the feed is inactive, no entry is made in the ingestion metrics table.
• When a feed is active, the last_heartbeat_time , log_count , or log_volume field is populated.

last_collection_time

TIMESTAMP

This field is in preview.
The epoch timestamp when a log entry was successfully fetched from source.

health_status

STRING

This field is in preview.
The overall health of the ingestion pipeline. Values: HEALTHY , ANOMALOUS , CRITICAL .

issue

STRING

This field is in preview.
The details explaining the issue in the pipeline.