Configure the Logging agent

This page provides details about the Cloud Logging agent's default and custom configurations.

Most users don't need to read this page. Read this page if:

• You're interested in learning deep technical details of the Cloud Logging agent's configuration.

• You want to change the configuration of the Cloud Logging agent.

Default configuration

The Logging agent google-fluentd is a modified version of the fluentd log data collector. The Logging agent comes with a default configuration; in most common cases, no additional configuration is required.

In its default configuration, the Logging agent streams logs, as included in the list of default logs , to Cloud Logging. You can configure the agent to stream additional logs; for details, go to Customizing the Logging agent configuration on this page.

The Logging agent uses fluentd input plugins to retrieve and pull event logs from external sources, such as files on disk, or to parse incoming log records. Input plugins are bundled with the agent or can be installed separately as Ruby gems; review the list of bundled plugins .

The agent reads log recordsstored in log files on the VM instance via fluentd 's built-in in_tail plugin. Each log record is converted to a log entry structure for Cloud Logging. The content of each log record is mostly recorded in the payloadof the log entries, but log entries also contain standard elements like a timestampand severity. The Logging agent requires every log record to be tagged with a string-format tag; all of the queries and output plugins match a specific set of tags. The log name usually follows the format, projects/[PROJECT-ID]/logs/[TAG] . For example, this log name includes the tag structured-log :

 projects/my-sample-project-12345/logs/structured-log 

The output plugin transforms each internalized structured message to a log entry in Cloud Logging. The payload becomes the text or JSON payload.

The following sections on this page discuss the default configuration in detail.

Default configuration definitions

The following sections describe the default configuration definitions for syslog, the forward input plugin, input configurations for third-party application logs, such as those in the list of default logs , and our Google Cloud fluentd output plugin.

Root configuration file location

• Linux: /etc/google-fluentd/google-fluentd.conf

  This root configuration file imports all configuration files from the /etc/google-fluentd/config.d folder as well.

• Windows: C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf

  If you are running a Logging agent before v1-5, the location is: C:\GoogleStackdriverLoggingAgent\fluent.conf

Syslog configuration

• Configuration file locations: /etc/google-fluentd/config.d/syslog.conf

• Description: This file includes the configuration to specify syslog as a log input.

• Review the config repository .

in_forward input plugin configuration

• Configuration file locations: /etc/google-fluentd/config.d/forward.conf

• Description: This file includes the configuration to configure the in_forward fluentd input plugin. The in_forward input plugin allows you to pass in logs via a TCP socket.

• Review the detailed fluentd documentation for this plugin and the config repository .

Third-party application log input configuration

• Configuration file locations: /etc/google-fluentd/config.d/[APPLICATION_NAME].conf

• Description: This directory includes configuration files to specify third-party applications' log files as log inputs. Each file, except syslog.conf and forward.conf , represents one application (e.g., apache.conf for the Apache application).

• Review the config repository .

¹ If you are using the <parse> stanza, then specify the format of the log by using @type .

Google Cloud fluentd output plugin configuration

• Configuration file locations:

  • Linux: /etc/google-fluentd/google-fluentd.conf

  • Windows: C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf

    If you are running a Logging agent before v1-5, the location is: C:\GoogleStackdriverLoggingAgent\fluent.conf

• Description: This file includes configuration options to control the Google Cloud fluentd output plugin's behavior.

• Go to the config repository .

Monitoring configuration

Output plugin telemetry

The enable_monitoring option controls whether the Google Cloud fluentd output plugin collects its internal telemetry. When set to true , the Logging agent keeps track of the number of log entries requested to be sent to Cloud Logging and the actual number of log entries successfully ingested by Cloud Logging. When set to false , no metrics are collected by the output plugin.

The monitoring_type option controls how this telemetry is exposed by the agent. See the following for the list of metrics.

When set to prometheus , the Logging agent exposes metrics in Prometheus format on the Prometheus endpoint ( localhost:24231/metrics by default; see prometheus and prometheus_monitor plugin configuration for details on customizing this). On Compute Engine VMs, in order for those metrics to be written to the Monitoring API, the Monitoring agent has to be installed and running as well.

When set to opencensus (default since v1.6.25 ), the Logging agent directly writes its own health metrics to the Monitoring API. This requires the roles/monitoring.metricWriter role to be granted to the Compute Engine default service account , even if the Monitoring agent is not installed.

The following metrics are written to the Monitoring API by both the Monitoring agent and the Logging agent in opencensus mode:

• agent.googleapis.com/agent/uptime with a version label: Uptime of the Logging agent.
• agent.googleapis.com/agent/log_entry_count with a response_code label: Count of log entries written by the Logging agent.
• agent.googleapis.com/agent/log_entry_retry_count with a response_code label: Count of log entries written by the Logging agent.
• agent.googleapis.com/agent/request_count with a response_code label: Count of API requests from the Logging agent.

These metrics are described in more detail on the Agent metrics page.

In addition, the following Prometheus metrics are exposed by the output plugin in prometheus mode:

• uptime with a version label: Uptime of the Logging agent.
• stackdriver_successful_requests_count with grpc and code labels: The number of successful requests to the Logging API.
• stackdriver_failed_requests_count with grpc and code labels: The number of failed requests to the Logging API, broken down by the error code.
• stackdriver_ingested_entries_count with grpc and code labels: The number of log entries ingested by the Logging API.
• stackdriver_dropped_entries_count with grpc and code labels: The number of log entries rejected by the Logging API.
• stackdriver_retried_entries_count with grpc and code labels: The number of log entries that failed to be ingested by the Google Cloud fluentd output plugin due to a transient error and were retried.

prometheus and prometheus_monitor plugin configuration

• Configuration file locations: /etc/google-fluentd/google-fluentd.conf

• Description: This file includes configuration options to control the behavior of the prometheus and prometheus_monitor plugins. The prometheus_monitor plugin monitors Fluentd's core infrastructure. The prometheus plugin exposes the metrics including the ones from the prometheus_monitor plugin and the ones from the google_cloud plugin above via a local port in Prometheus format. See more details at https://docs.fluentd.org/deployment/monitoring-prometheus.

• Go to the config repository .

For monitoring Fluentd, the built-in Prometheus http metrics server is enabled by default. You can remove the following section from the configuration to stop this endpoint from starting:

 # Prometheus monitoring.
<source>
  @type prometheus
  port 24231
</source>
<source>
  @type prometheus_monitor
</source> 

Processing payloads

Most of the supported logs under the default configuration of the Logging agent are from log files and are ingested as unstructured (text) payloads in the log entries.

The only exception is that the in_forward input plugin, which is also enabled by default, only accepts structured logs and ingests them as structured (JSON) payloads in the log entries. For details, read Streaming structured (JSON) log records via in_forward plugin on this page.

When the log line is a serialized JSON object and the detect_json option is enabled, the output plugin transforms the log entry into a structured (JSON) payload. This option is enabled by default in VM instances running on App Engine flexible environment and Google Kubernetes Engine. This option isn't enabled by default in VM instances running on App Engine standard environment. Any JSON parsed with the detect_json option enabled is always ingested as jsonPayload .

You can customize the agents's configuration to support ingesting structured logs from additional resources. See Streaming structured (JSON) log records to Cloud Logging for details.

The payload of log records streamed by a custom-configured Logging agent can be either a single unstructured text message ( textPayload ) or a structured JSON message ( jsonPayload ).

Special fields in structured payloads

When the Logging agent receives a structured log record, it moves any key that matches the following table into the corresponding field in the LogEntry object. Otherwise, the key becomes part of the LogEntry.jsonPayload field. This behavior lets you set specific fields in the LogEntry object, which is what is written to the Logging API. For example, if the structured log record contains a key of severity , then the Logging agent populates the LogEntry.severity field.

Time-related fields

In general, time-related information about a log entry is stored in the timestamp field of the LogEntry object:

 {
insertId: "1ad8d08f-6529-47ea-832e-467f869a2da4"
...
resource: {2}
timestamp: "2023-10-30T16:33:15.505196Z"
} 

When the source for a log entry is structured data, the Logging agent uses the following rules to search the fields in the jsonPayload entry for time-related information:

1. Search for a timestamp field that is a JSON object that includes the seconds and nanos fields, representing, respectively, a signed number of seconds from the UTC epoch and a nonnegative number of fractional seconds:

    jsonPayload: {
     ...
     "timestamp": {
       "seconds": CURRENT_SECONDS,
       "nanos": CURRENT_NANOS
     }
   } 
   

2. If the previous search fails, then search for a pair of timestampSeconds and timestampNanos fields:

    jsonPayload: {
     ...
     "timestampSeconds": CURRENT_SECONDS,
     "timestampNanos": CURRENT_NANOS
   } 
   

3. If the previous search fails, then search for a time field that is a string in RFC 3339 format:

    jsonPayload: {
     ...
     "time": CURRENT_TIME_RFC3339
   } 
   

When time-related information is found, the Logging agent uses that information to set the value of the LogEntry.timestamp , and it doesn't copy that information from the structured record into the LogEntry.jsonPayload object.

Time-related fields that aren't used to set the value of the LogEntry.timestamp field are copied from the structured record into the LogEntry.jsonPayload object. For example, if the structured record contains a timestamp JSON object and a time field, then the data in the timestamp JSON object is used to set the LogEntry.timestamp field. The LogEntry.jsonPayload object contains a time field, because this field wasn't used to set the LogEntry.timestamp value.

Customizing agent configuration

Besides the list of default logs that the Logging agent streams by default, you can customize the Logging agent to send additional logs to Logging or to adjust agent settings by adding input configurations.

The configuration definitions in these sections apply to the fluent-plugin-google-cloud output plugin only and specify how logs are transformed and ingested into Cloud Logging.

• Main configuration file locations:

  • Linux: /etc/google-fluentd/google-fluentd.conf

  • Windows: C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf

    If you are running a Logging agent before v1-5, the location is: C:\GoogleStackdriverLoggingAgent\fluent.conf

• Description: This file includes configuration options to control the fluent-plugin-google-cloud output plugin's behavior.

• Review the config repository .

Streaming logs from additional inputs

You can customize the Logging agent to send additional logs to Logging by adding input configurations.

Streaming unstructured (text) logs via log files

1. From the Linux command prompt, create a log file:

    touch  
   /tmp/test-unstructured-log.log 
   

2. Create a new configuration file labeled test-unstructured-log.conf in the additional configuration directory /etc/google-fluentd/config.d :

    sudo  
   tee  
   /etc/google-fluentd/config.d/test-unstructured-log.conf  
   <<EOF
   < source 
   >  
   @type  
   tail  
   <parse>  
    # 'none' indicates the log is unstructured (text). 
     
   @type  
   none  
   </parse>  
    # The path of the log file. 
     
   path  
   /tmp/test-unstructured-log.log  
    # The path of the position file that records where in the log file 
     
    # we have processed already. This is useful when the agent 
     
    # restarts. 
     
   pos_file  
   /var/lib/google-fluentd/pos/test-unstructured-log.pos  
   read_from_head  
    true 
     
    # The log tag for this log input. 
     
   tag  
   unstructured-log
   </source>
   EOF 
   

   An alternative to creating a new file, is to add the configuration information to an existing configuration file.

3. Restart the agent to apply the configuration changes:

    sudo  
   service  
   google-fluentd  
   restart 
   

4. Generate a log record into the log file:

     echo 
     
    'This is a log from the log file at test-unstructured-log.log' 
    >> 
   /tmp/test-unstructured-log.log 
   

5. Check the Logs Explorer to see the ingested log entry:

    {
     insertId:  "eps2n7g1hq99qp"
     labels: {
     compute.googleapis.com/resource_name:  "add-unstructured-log-resource"
     }
     logName:  "projects/my-sample-project-12345/logs/unstructured-log"
     receiveTimestamp:  "2018-03-21T01:47:11.475065313Z"
     resource: {
     labels: {
       instance_id:  "3914079432219560274"
       project_id:  "my-sample-project-12345"
       zone:  "us-central1-c"
     }
     type:  "gce_instance"
     }
     textPayload:  "This is a log from the log file at test-unstructured-log.log"
     timestamp:  "2018-03-21T01:47:05.051902169Z"
   } 
   

Streaming structured (JSON) logs via log files

You can configure the Logging agent to require each log entry for certain log inputs to be structured . You can also customize the Logging agent to ingest JSON formatted content from a log file. When the agent is configured to ingest JSON content, the input mustbe formatted so that each JSON object is on a newline:

{"name" : "zeeshan", "age" : 28}
    {"name" : "reeba", "age" : 15}

To configure the Logging agent to ingest JSON formatted content, do the following:

1. From the Linux command prompt, create a log file:

    touch  
   /tmp/test-structured-log.log 
   

2. Create a new configuration file labeled test-structured-log.conf in the additional configuration directory /etc/google-fluentd/config.d :

    sudo  
   tee  
   /etc/google-fluentd/config.d/test-structured-log.conf  
   <<EOF
   < source 
   >  
   @type  
   tail  
   <parse>  
    # 'json' indicates the log is structured (JSON). 
     
   @type  
   json  
   </parse>  
    # The path of the log file. 
     
   path  
   /tmp/test-structured-log.log  
    # The path of the position file that records where in the log file 
     
    # we have processed already. This is useful when the agent 
     
    # restarts. 
     
   pos_file  
   /var/lib/google-fluentd/pos/test-structured-log.pos  
   read_from_head  
    true 
     
    # The log tag for this log input. 
     
   tag  
   structured-log  
   </source>  
   EOF 
   

   An alternative to creating a new file, is to add the configuration information to an existing configuration file.

3. Restart the agent to apply the configuration changes:

    sudo  
   service  
   google-fluentd  
   restart 
   

4. Generate a log record into the log file:

     echo 
     
    '{"code": "structured-log-code", "message": "This is a log from the log file at test-structured-log.log"}' 
    >> 
   /tmp/test-structured-log.log 
   

5. Check the Logs Explorer to see the ingested log entry:

    {
     insertId:  "1m9mtk4g3mwilhp"
     jsonPayload: {
     code:  "structured-log-code"
     message:  "This is a log from the log file at test-structured-log.log"
     }
     labels: {
     compute.googleapis.com/resource_name:  "add-structured-log-resource"
     }
     logName:  "projects/my-sample-project-12345/logs/structured-log"
     receiveTimestamp:  "2018-03-21T01:53:41.118200931Z"
     resource: {
     labels: {
       instance_id:  "5351724540900470204"
       project_id:  "my-sample-project-12345"
       zone:  "us-central1-c"
     }
     type:  "gce_instance"
     }
     timestamp:  "2018-03-21T01:53:39.071920609Z"
   } 
   

   In the Logs Explorer, filter by your resource type and a logNameof structured-log .

For additional options to customize your log input format for common third-party applications, see Common Log Formats and How To Parse Them .

Streaming structured (JSON) logs via in_forward plugin

Additionally, you can send logs via the fluentd in_forward plugin. fluentd-cat is a built-in tool that helps easily send logs to the in_forward plugin. The fluentd documentation contains more details for this tool.

To send logs via the fluentd in_forward plugin, read the following instructions:

1. Execute the following command on the VM with the Logging agent installed:

     echo 
     
    '{"code": "send-log-via-fluent-cat", "message": "This is a log from in_forward plugin."}' 
     
    | 
     
   /opt/google-fluentd/embedded/bin/fluent-cat  
   log-via-in-forward-plugin 
   

2. Check the Logs Explorer to see the ingested log entry:

    {
     insertId:  "1kvvmhsg1ib4689"
     jsonPayload: {
     code:  "send-log-via-fluent-cat"
     message:  "This is a log from in_forward plugin."
     }
     labels: {
     compute.googleapis.com/resource_name:  "add-structured-log-resource"
     }
     logName:  "projects/my-sample-project-12345/logs/log-via-in-forward-plugin"
     receiveTimestamp:  "2018-03-21T02:11:27.981020900Z"
     resource: {
     labels: {
       instance_id:  "5351724540900470204"
       project_id:  "my-sample-project-12345"
       zone:  "us-central1-c"
     }
     type:  "gce_instance"
     }
     timestamp:  "2018-03-21T02:11:22.717692494Z"
   } 
   

Streaming structured (JSON) log records from application code

You can enable connectors in various languages to send structured logs from application code; for more information, review the fluentd documentation . These connectors are built based on the in_forward plugin.

Setting log entry labels

The following configuration options let you override LogEntry labels and MonitoredResource labels when ingesting logs to Cloud Logging. All log entries are associated with monitored resources; for more information, review the list of Cloud Logging monitored resource types .

The Logging agent output plugin supports three ways to set LogEntry labels:

• Dynamically, replacing specific labels in a structured entry with different labels. For details, go to Setting labels in structured log entries on this page.
• Statically, attaching a label to any occurrence of a value. For details, go to Setting labels statically on this page.

Setting labels in structured log entries

Suppose you wrote a structured log entry payload like this:

 { "message": "This is a log message", "timestamp": "Aug 10 20:07:00", "env": "production" } 

And suppose you want to translate the payload field env to a metadata label environment . To do this, add the following to your output plugin configuration in the main configuration file ( /etc/google-fluentd/google-fluentd.conf on Linux or C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf on Windows):

  # Configure all sources to output to Cloud Logging 
<match  
**>  
@type  
google_cloud  
label_map  
 { 
  
 "env" 
:  
 "environment" 
  
 } 
  
...
</match> 

The label_map setting here replaces the env label in the payload with environment , so the resulting log entry has a label environment with the value production .

Setting labels statically

If you don't have this information in the payload, and simply want to add a static metadata label called environment , add the following to your output plugin configuration in the main configuration file ( /etc/google-fluentd/google-fluentd.conf on Linux or C:\Program Files (x86)\Stackdriver\LoggingAgent\fluent.conf on Windows):

  # Configure all sources to output to Cloud Logging 
<match  
**>  
@type  
google_cloud  
labels  
 { 
  
 "environment" 
:  
 "production" 
  
 } 
  
...
</match> 

In this case, instead of using a map to replace one label with another, we use a labels setting to attach a label with a given literal value to a log entry, regardless of whether the entry already has a label or not. This approach can be used even if you are sending unstructured logs.

For more on how to configure labels , label_map and other Logging agent settings, go to Setting Log Entry Labels on this page.

Modifying Log Records

Fluentd provides built-in filter plugins that can be used to modify log entries.

The most commonly used filter plugin is filter_record_transformer . It enables you to:

• Add new fields to log entries
• Update fields in log entries
• Delete fields in log entries

Some output plugins also let you modify log entries. The fluent-plugin-record-reformer output plugin provides functionality similar to the filter_record_transformer filter plugin, except that it also allows you to modify log tags. More resource usage is expected with this plugin: each time a log tag is updated, it generates a new log entry with the new tag. Note that the tag field in the configuration is required; we also recommend that you modify this field to avoid entering a dead loop.

The fluent-plugin-detect-exceptions output plugin scans a log stream, either unstructured (text) or JSON-format log records, for multi-line exception stack traces. If a consecutive sequence of log entries forms an exception stack trace, the log entries are forwarded as a single, combined log message. Otherwise, the log entry is forwarded as it was.

Advanced (non-default) configuration definitions

If you want to customize the configuration of your Logging agent, beyond its default configuration , continue to read this page.

Buffer-related configuration options

The following configuration options let you adjust the Logging agent's internal buffering mechanism.

Project- and monitored resource-related configuration options

The following configuration options let you manually specify a project and certain fields from the MonitoredResource object. These values are automatically gathered by the Logging agent; it is not recommended that you manually specify them.

Other output plugin configuration options

¹ This feature is enabled by default in VM instances running on App Engine flexible environment and Google Kubernetes Engine.

Applying customized agent configuration

Customizing the Logging agent allows you to add your own fluentd configuration files:

For more information on fluentd configuration files, see fluentd's Configuration File Syntax documentation .