The dashboards that Application Monitoring creates display log and metric data, including the following golden signals, when that data includes application-specific labels:
- Traffic : Incoming request rates on the service or workload over the selected time period. For some configurations, this chart contains tabs. The Request tab displays the networking L7 metric and the Bytes tab displays the networking L4 metric.
- Server error rate : Ratio of incoming requests whose response code is equivalent to an 5xx HTTP response code to all incoming requests over the selected time period.
- P95 latency : 95th percentile of latency for a request served over the selected time period, in milliseconds.
- Saturation : Measures how full your service or workload is. For example, for managed instance groups (MIGs), Cloud Run, and Google Kubernetes Engine deployments, this field shows the CPU utilization.
If you explore a dashboard, you might notice that a charted metric for a golden signal differs from the metric you expect. For some golden signals, Google Cloud Observability has a prioritized list of source metrics. If a preferred metric isn't available, then Google Cloud Observability searches the list until it finds an available metric.
Supported infrastructure resources
The Notes and limitationscolumn lists details about which golden signals are supported. This column also lists limitations.
- Traffic : instance/postgres/transaction_count
- Server error rate : Ratio of the instance/postgres/abort_count to the total instance/postgres/transaction_count .
- Saturation : instance/cpu/average_utilization
- Saturation : cluster/cpu_load
- Saturation : container/cpu/utilizations
- Traffic : request_count
- Server error rate
: The ratio of the number
of requests with a response status of
5xxto the total number of requests. - P95 latency : request_latencies
- Saturation : container/cpu/utilizations
Only trace spans generated by instrumented workloads contain application labels.
Only customer-instrumented workloads running within Cloud Run generate golden signals.
- Saturation : database/cpu/utilization
- Traffic : api/request_count
- Server error rate
: Ratio of requests with a response
status that indicates an error to the total number of requests.
The following status values indicate an error:
-
DATA_LOSS -
DEADLINE_EXCEEDED -
INTERNAL -
UNAVAILABLE -
UNIMPLEMENTED -
UNKNOWN
-
- Traffic : instance/network/received_bytes_count
- Saturation : instance/cpu/utilization
- Traffic
: The metric depends on whether the service is
single- or multi-region:
- Single: service/request_count .
- Multi: service/multi_region/request_count .
- Server error rate
: The ratio of the number of service/grpc/finished_requests_count
with a response status that indicates an error to the total number
of finished requests.
The following status values indicate an error:
-
DEADLINE_EXCEEDED -
INTERNAL -
UNAVAILABLE -
UNIMPLEMENTED
-
- P95 latency
: The metric depends on whether the service is
single- or multi-region:
- Single: service/request_latencies
- Multi: service/multi_region/request_latencies .
Application labels aren't attached to spans.
For general information, see Dataproc Metastore overview .
- Traffic : api/request_count
- Server error rate
: Ratio of requests with a response
status that indicates an error to the total number of requests.
The following status values indicate an error:
-
DEADLINE_EXCEEDED -
INTERNAL -
UNAVAILABLE -
UNIMPLEMENTED
-
- P95 latency : api/request_latencies
- Traffic : application_lb/request_count
- Server error rate
: The ratio of the number
of requests with a response status of
5xxto the total number of requests. - P95 latency : application_lb/total_latencies
- Traffic : application_lb/request_count
- Server error rate
: The ratio of the number
of requests with a response status of
5xxto the total number of requests. - P95 latency : application_lb/total_latencies
(Global, regional, and cross region)
- Traffic : Based on a Cloud Load Balancing metric type that records the request count, like https/request_count . The configuration of the Cloud Load Balancing determines the actual metric.
- Server error rate
: The ratio of the number
of requests with a response status of
5xxto the total number of requests. - P95 latency : Based on a Cloud Load Balancing metric type that records total latencies, like https/total_latencies . The configuration of the Cloud Load Balancing determines the actual metric.
- Saturation : cluster/cpu/average_utilization
- Traffic : topic/send_request_count
- Server error rate
: The ratio of the number of
requests with a response code of
internalto the total number of requests. - P95 latency : topic/send_request_latencies
- Traffic : subscription/pull_request_count
- Server error rate
: The ratio of the number of
requests with a response code of
internalto the total number of requests. - P95 latency : subscription/push_request_latencies
- Traffic : api/api_request_count
- Server error rate
: Ratio of requests with a response
status that indicates an error to the total number of requests.
The following status values indicate an error:
-
data_loss -
deadline_exceeded -
internal -
unavailable -
unimplemented -
unknown
-
- P95 latency : api/request_latencies
- Saturation : instance/cpu/utilization
- Traffic : reasoning_engine/request_count
- Server error rate : Ratio of requests whose response status is at least 500 and no more than 599 to the total number of requests.
- P95 latency : reasoning_engine/request_latencies
- Traffic : mcp/request_count
- Server error rate
: The ratio of the number
of requests with a response status of
5xxto the total number of requests. - P95 latency : mcp/request_durations
Golden signals are shown only in the detailed dashboard.
Application labels aren't attached to metric data or to spans.
Golden signals for Kubernetes
For Kubernetes DaemonSets, Deployments, and StatefulSets, Google Cloud Observability uses the following metric data to derive values for golden signals, when that metric data is available:
- Traffic : service/server/request_count . The units for this metric are requests per second.
- Server error rate
: Ratio of
service/server/request_countrequests whose response status is at least 500 and no more than 599 to the total number of requests. - P95 latency : The 95th percentile from the service/server/response_latencies .
- Saturation : The ratio of the container/cpu/core_usage_time to the container/cpu/request_cores .
On a dashboard, you might see that the chart that displays traffic information contains two tabs:
- The tab labeled Traffic (Requests)displays networking L7 traffic data and the units are requests per second.
- The tab labeled Traffic (Bytes)displays the pod/network/received_bytes_count metric, which has units KiB per second and is a networking L4 metric.
When both the L7 and L4 traffic metrics are available, the L7 metrics appear on the summary pages but the traffic chart contains tabs, which let you view both metrics. If only one metric is available, then that metric is shown on the summary page and on dashboards.
For workloads that run on GKE, Google Cloud Observability might
derive golden signals from the Prometheus metric http_server_request_duration_seconds
, which is only available
when you instrument your application by using OpenTelemetry. To learn more, see Instrument your application
.
