Microservices observability reference information

Configuration data

The configuration data found in the environment variables is defined in the following table.

export GRPC_GCP_OBSERVABILITY_CONFIG='{
    "project_id": "your-project-here",
    "cloud_monitoring": {
    }
}'

export GRPC_GCP_OBSERVABILITY_CONFIG='{
    "project_id": "your-project-here",
    "cloud_trace": {
    }
}'

Trace definitions

This section provides information on tracing.

Trace context propagation

For cross-service tracing to work, the service owner must support the propagation of trace context received from upstream (or started by itself) to downstream. Trace context is propagated among services through gRPC metadata. Make sure that you enable the Cloud Monitoring, Cloud Logging, Cloud Trace APIs, and Microservices APIs, which let services in this configuration to report their telemetry data to the appropriate service.

Without propagation support, downstream services can't generate spans for a trace. Existing spans are not affected. The Microservices observability plugins support the OpenCensus Binary Format for encoding and encoding trace context.

Spans

The name of a span is formatted as follows:

Span labels

The integrations provide different span labels.

For attempt spans, two additional retry-related attributes (span labels) are attached:

Metrics definitions

The following metrics are available and are displayed in a dashboard named Microservices (gRPC) Monitoringfor common user journeys.

The following are metrics from the gRPC client-side metrics:

The following gRPC server side metrics are available:

Each distribution in the table above contains a histogram with buckets as follows:

• Size in bytes: 0, 1024, 2048, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864, 268435456, 1073741824, 4294967296

• Latency in ms: 0, 0.01, 0.05, 0.1, 0.3, 0.6, 0.8, 1, 2, 3, 4, 5, 6, 8, 10, 13, 16, 20, 25, 30, 40, 50, 65, 80, 100, 130, 160, 200, 250, 300, 400, 500, 650, 800, 1000, 2000, 5000, 10000, 20000, 50000, 100000

Tag description:

• grpc_client_method : Full gRPC method name, including package, service and method, for example, google.bigtable.v2.Bigtable/CheckAndMutateRow
• grpc_client_status : gRPC server status code received, for example, OK , CANCELLED , DEADLINE_EXCEEDED
• grpc_server_method : Full gRPC method name, including package, service and method, for example, com.exampleapi.v4.BookshelfService/Checkout
• grpc_server_status : gRPC server status code returned, for example, OK , CANCELLED , DEADLINE_EXCEEDED

Log record definitions

The Microservices observability logs are uploaded to Cloud Logging using the log name ( PROJECT_ID is the placeholder for the string representing your project):

  logName 
 = 
 projects 
 /[ 
 PROJECT_ID 
 ]/ 
 logs 
 / 
 microservices 
 . 
 googleapis 
 . 
 com 
 % 
 2 
 Fobservability 
 % 
 2 
 Fgrpc 
 

The following is the JSON representation of the generated log record:

  { 
  
 "authority" 
 : 
  
 string 
 , 
  
 "callId" 
 : 
  
 string 
 , 
  
 "type" 
 : 
  
 string 
 , 
  
 "logger" 
 : 
  
 string 
 , 
  
 "serviceName" 
 : 
  
 string 
 , 
  
 "methodName" 
 : 
  
 string 
 , 
  
 "peer" 
 : 
  
 { 
  
 "type" 
 : 
  
 string 
 , 
  
 "address" 
 : 
  
 string 
 , 
  
 "ipPort" 
 : 
  
 int 
  
 }, 
  
 "payload" 
 : 
  
 { 
  
 "timeout" 
 : 
  
 string 
 , 
  
 "metadata" 
 : 
  
 { 
  
 string 
 : 
  
 string 
 , 
  
 string 
 : 
  
 string 
  
 }, 
  
 "statusCode" 
 : 
  
 string 
 , 
  
 "statusMessage" 
 : 
  
 string 
 , 
  
 "statusDetails" 
 : 
  
 string 
 , 
  
 "message" 
 : 
  
 string 
 , 
  
 "messageLength" 
 : 
  
 int 
 , 
  
 }, 
  
 "sequenceId" 
 : 
  
 int 
 } 
 

The following table describes the fields in the log entry:

Resource labels

Resource labels identify the source that generates observability data. Each resource label is a key-value pair, where keys are predefined values that are specific to the source environment (for example, GKE or Compute Engine).

For metrics and tracing on GKE deployments, resource labels are populated by default except for the container name and namespace name. The missing values can be populated using the Downward API .

The following are the environment variable keys:

• CONTAINER_NAME
• NAMESPACE

For example, the env section in the following includes two resource labels:

  apiVersion 
 : 
  
 apps 
 / 
 v1 
 kind 
 : 
  
 Deployment 
 metadata 
 : 
  
 labels 
 : 
  
 run 
 : 
  
 app1 
  
 name 
 : 
  
 app1 
 spec 
 : 
  
 replicas 
 : 
  
 2 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 run 
 : 
  
 app1 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 run 
 : 
  
 app1 
  
 spec 
 : 
  
 containers 
 : 
  
 - 
  
 image 
 : 
  
 'o11y-examples:1.00' 
  
 name 
 : 
  
 container1 
  
 ports 
 : 
  
 - 
  
 protocol 
 : 
  
 TCP 
  
 containerPort 
 : 
  
 50051 
  
 env 
 : 
  
 - 
  
 name 
 : 
  
 CONTAINER_NAME 
  
 value 
 : 
  
 container1 
  
 - 
  
 name 
 : 
  
 NAMESPACE 
  
 valueFrom 
 : 
  
 fieldRef 
 : 
  
 fieldPath 
 : 
  
 metadata 
 . 
 namespace 
 

Custom labels

Custom labels represent additional user-provided information in the observability data. Labels consist of a key and a value. The key-value pair is attached to tracing data as span labels, to metrics data as metrics labels, and to logging data as log entry labels. All custom labels are of the STRING type.

You can provide custom labels in the configuration by specifying a list of key- value pairs for labels . The implementation reads the configuration and creates a separate label for each key-value pair, then attaches the label to the observability data. For example:

 "labels": {
    "DATACENTER": "SAN_JOSE_DC",
    "APP_ID": "24512"
  } 

Each log entry has the following additional labels in them:

 {
   "DATACENTER": "SAN_JOSE_DC"
   "APP_ID": "24512"
} 

What's next

• For information on metrics kinds and types, see Value types and metric kinds .
• For information on distributions, see the Distribution reference page .
• For information about charting Distribution metrics, see Distribution metrics .
• For information on units (such as 1 , ms and By ), see the unit field in the MetricDescriptor reference.