OTLP log record to log entry mapping

This document describes how Google Cloud Observability determines the fields of a LogEntry from the OTLP log record, when that record is sent to Google Cloud by using the Telemetry API.

General structure of OTLP-formatted log data

When log data is sent to Google Cloud by using the Telemetry API, this data must be in a format that is consistent with OTLP. The general structure for this data is as shown:

 "resourceLogs": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeLogs": [
        {
          "logRecords": [...]
        }
      ]
    }
] 

Notice that OpenTelemetry batches individual logs, each of which is represented by a logRecord structure, with information about the source of those logs, which is represented by the resource structure.

When Google Cloud Observability receives a resourceLogs object, it constructs one LogEntry for each logRecord . Unlike OTLP which batches source information with a collection of individual logs, each LogEntry structure contains information about the source of the log and the log itself.

To learn more about the structure of OTLP-formatted log data, see the OpenTelemetry logs.proto .

How LogEntry fields are set

Google Cloud Observability uses the following rules to determine the values for LogEntry fields:

HttpRequest field

Google Cloud Observability maps OTLP attributes that apply to HTTP requests to LogEntry fields. The following sections describe how the system maps flat and nested attributes.

Flat attributes

The following table describes how Google Cloud Observability maps flat attributes that apply to HTTP requests to LogEntry fields. For example, the value from the attribute http.request.method: "GET", is set as the value of the httpRequest.requestMethod field in a log entry:

Nested attributes

This section describes how Google Cloud Observability maps nested OTLP attributes that apply to HTTP requests to fields in a LogEntry . The following example illustrates a log record that contains two attributes, each of which contains at least one other attribute:

log_record {
  attributes: {
    gcp.http_request {
      "requestMethod": "GET",
      "requestUrl": "some-URL",
    }
    http_request {
      "requestMethod": "GET",
    }
  }
}

In the table, the nested attributes are shown by using braces. For example, gcp.http_request {requestMethod} means that the attribute gcp.http_request contains the attribute requestMethod . The value from the innermost attribute is assigned to the value of the log entry field:

Labels field

To determine what labels to attach to the log entry, the system performs the following actions:

1. It removes from the OTLP log record any attributes that have been mapped to specific LogEntry fields.

   For example, suppose there are the attributes attached to a log record:

    attributes: {
       "log_array_attr: ["value1", "value2"],
       "log_json_attr": {"json_key": "json_value"}
       "log-string-attr": "string",
       "code.file.path": "my-file.cc",
       "code.function.name: "my-func",
       "code.line.number": 123,
       "gcp.http_request": {
           "requestMethod": "GET",
           "requestUrl": "my-URL",
     },
   } 
   

   After removing fields that are mapped to specific LogEntry fields, the following fields remain:

    attributes: {
       "log_array_attr: ["value1", "value2"],
       "log_json_attr": {"json_key": "json_value"}
       "log-string-attr": "string",
   } 
   

2. If an attribute contains an array or JSON elements, then the system converts the value into a string.

   For example, the following illustrates how the LogEntry represents the previous attributes:

    labels: {
       "log_array_attr": "[\"value1\",\"value2\"]",
       "log_json_attr": "{\"json_key\":\"json_value\"}",
       "log-string-attr": "string",
   } 
   

   The maximum nesting depth for attributes is five. Any content that has deeper nesting is truncated.

Otel field

For fields in OTLP log data that don't have an equivalent field in a LogEntry , the system converts data types and then adds the converted data to the otel field. For example, the otel field stores attributes from the resource , scope , and entity fields.

The system uses the following rules to convert the OpenTelemetry data types to protobuf Value types :

To avoid double precision errors, pass integers as strings.

Payload field

The data type of the OTLP logRecord.body field determines the structure of the LogEntry payload:

• string : The system copies the string into the LogEntry.textPayload field.

• Array : The system creates a string of the array elements while preserving new lines. It then copies the string into the LogEntry.textPayload field.

• KeyValueList : The system converts those pairs into JSON and then populates the LogEntry.jsonPayload field, with the following restrictions:

  • When an OTLP record contains duplicate attribute keys, the system retains the first key and discards attributes with duplicate keys.
  • When the nesting depth for a JSON pair is larger than five, the system truncates the content to a depth of five.

Severity field

This section describes how Google Cloud Observability maps the OpenTelemetry severity fields to Cloud Logging severity levels. OpenTelemetry defines a severity number and severity text. The logs.proto defines the severity numbers.

Google Cloud Observability determines the Logging severity from the OpenTelemetry severity number, if set. Otherwise, it uses the severity text. If neither are set, then the Logging severity is set to DEFAULT .

SourceLocation field

Google Cloud Observability maps the following OTLP Code directly to LogEntry fields. This mapping is possible because these OpenTelemetry attributes are semantically identical to Cloud Logging concepts.

Limitations

This section describes limits. It also describes how Google Cloud Observability handles certain types of data.

Limits

Behavior

• When both the OpenTelemetry severity number and severity text are set, the system uses the severity number to determine the Cloud Logging severity level. If the OTLP record doesn't contain severity information, then the Cloud Logging severity level is set to DEFAULT .

• When an OTLP record contains duplicate attribute keys, the system retains the first key and discards the attributes with duplicate keys.

• The system converts attributes attached to a log record into a string. For an example, see Labels field .

• Log names must be URL-safe or they are URL-encoded during ingestion. For information about how log names are set, see How LogEntry fields are set .