This page describes how to emit gRPC client-side metrics to Cloud Monitoring when you use gRPC to interact with Cloud Storage using one of the following supported interfaces:
-
Cloud Storage Java client library , version 2.41.0 and later
- Apache Beam connector on Dataflow , version 2.2.26 and later
Client-side metrics can be used to monitor the performance of the client application that interacts with Cloud Storage using gRPC. The client-side metric differs from server-side metrics, which provide insight on Cloud Storage performance from the server-side perspective.
How it works
You can opt in to emit client-side metrics to Cloud Monitoring when using gRPC to interact with Cloud Storage using one of the supported interfaces. You can view client-side metrics using the Metrics Explorer to help you monitor and optimize interactions between Cloud Storage and the gRPC client, manage usage, and troubleshoot performance bottlenecks and technical issues.
Pricing
Cloud Storage client-side metrics are non-chargeable, meaning you can emit, store, and access Cloud Storage client-side metrics without incurring Cloud Monitoring charges . For more information about pricing, refer to Google Cloud Observability pricing .
Before you begin
To use client-side metrics, you must first complete the following steps:
-
Verify that the Cloud Storage client library or connector you want to use supports gRPC. The following Cloud Storage client libraries and connectors support gRPC:
-
Enable the Cloud Monitoring API .
-
Enable the Cloud Storage API.
-
Set the required roles and permissions needed to emit client-side metrics.
Required roles
To set the permissions that you need to emit gRPC client-side metrics to
Cloud Monitoring, grant the Monitoring Metric Writer
( roles/monitoring.metricWriter
) IAM role on the service
account used by the gRPC client.
This predefined role contains the permissions required to emit gRPC client-side metrics to Cloud Monitoring. To see the exact permissions that are required, refer to the Required permissions section:
Required permissions
-
monitoring.timeSeries.create
This IAM permission is included in the following Cloud Storage predefined roles:
- Storage Admin
(
roles/storage.admin
) - Storage Object Admin
(
roles/storage.objectAdmin
) - Storage Object User
(
roles/storage.objectUser
)
You might also be able to get these permissions with other custom roles
or predefined roles
. For more information about the Monitoring
Metric Writer role, see IAM documentation about roles/monitoring.metricWriter
.
Considerations
-
If you're using Cloud Storage connector on Dataproc, you need to grant the Monitoring Metric Writer (
roles/monitoring.metricWriter
) IAM role to the Dataproc virtual machine (VM) service account . -
If you're using Apache Beam connector on Dataflow, you need to grant the Monitoring Metric Writer (
roles/monitoring.metricWriter
) IAM role to the Dataflow worker service account .
View metrics in Metrics Explorer
Use the following instructions to view Cloud Storage gRPC client-side metrics in the Metrics Explorer.
-
In the Google Cloud console, go to the Metrics Explorerpage.
-
Select the project you want to view metrics for.
-
From the Metricdrop-down menu, click Select a metric.
-
In the Filter by resource or metric namesearch bar, enter
storage.googleapis.com/Client
or search for the metric you want to apply by metric name and click Apply. To add more than one metric, click Add query.Cloud Storage applies the metrics to your project. You can filter or aggregate your metrics using the following drop-down menus:
-
To select and view a subset of your data based on specified criteria, use the Filterdrop-down menu.
-
To combine multiple data points into a single value and see a summarized view of your metrics, use the Aggregationdrop-down menu.
Let your application run for at least a minute before you check for any published metrics.
-
To view the metrics you've added to your project using a dashboard, see Dashboards overview .
Metrics descriptions
The following sections outline Cloud Storage client-side metrics that can be used to monitor the performance of the gRPC client.
Client per-attempt metrics
The following metrics gather performance data about individual attempts made by a client to communicate with a server. Client per-attempt metrics can help you measure retry behavior, bottlenecks, and optimize communication between a client and server.
storage.googleapis.com/client/grpc/client/attempt/started
Preview
. The total number of RPC attempts started,
including those that haven't completed.{attempt}
-
grpc.method
: the full gRPC method name, including package, service, and method. -
grpc.target
: canonicalized target URI used when created gRPC channel.
storage.googleapis.com/client/grpc/client/attempt/duration
Preview
. The end-to-end time it takes to complete an RPC
attempt, including the time it takes to pick a subchannel.s
-
grpc.method
: full gRPC method name, including package, service, and method. -
grpc.target
: canonicalized target URI used when creating gRPC channel. -
grpc.status
: gRPC server status code received, such asOK
,CANCELLED
, orDEADLINE_EXCEEDED
. -
grpc.lb.locality
: the locality the traffic is being sent to. This will be set to the resolver attribute passed down from theweighted_target
policy, or the empty string if the resolver attribute is unset.
storage.googleapis.com/client/grpc/client/attempt/sent_total_compressed_message_size
Preview
. The total bytes, compressed but not encrypted,
that are sent across all request messages except metadata per RPC
attempt. This doesn't include gRPC or transport framing bytes.By
-
grpc.method
: full gRPC method name, including package, service, and method. -
grpc.target
: canonicalized target URI used when creating gRPC channel. -
grpc.status
: gRPC server status code received, such asOK
,CANCELLED
, orDEADLINE_EXCEEDED
. -
grpc.lb.locality
: the locality the traffic is being sent to. This will be set to the resolver attribute passed down from theweighted_target
policy, or the empty string if the resolver attribute is unset.
storage.googleapis.com/client/grpc/client/attempt/rcvd_total_compressed_message_size
Preview
. The total bytes, compressed but not encrypted,
that are received across all response messages except metadata per RPC
attempt. This doesn't include gRPC or transport framing bytes.By
-
grpc.method
: full gRPC method name, including package, service, and method. -
grpc.target
: canonicalized target URI used when creating gRPC channel. -
grpc.status
: gRPC server status code received, such asOK
,CANCELLED
, orDEADLINE_EXCEEDED
. -
grpc.lb.locality
: the locality the traffic is being sent to. This will be set to the resolver attribute passed down from theweighted_target
policy, or the empty string if the resolver attribute is unset.
For more information about client per-attempt instruments, see OpenTelemetry metrics documentation in GitHub .
Client per-call metrics
The following metrics provide an aggregated view of the entire lifecycle of a client call to a server. Client per-call metrics provide high-level data about client calls, provide tracking metrics to understand call patterns, and help you identify frequencies in errors.
storage.googleapis.com/client/grpc/client/call/duration
Preview
. Measures the end-to-end time the gRPC library
takes to complete an RPC from the application's perspective.s
-
grpc.method
: full gRPC method name, including package, service, and method. -
grpc.target
: canonicalized target URI used when creating gRPC channel. -
grpc.status
: gRPC server status code received such asOK
,CANCELLED
, orDEADLINE_EXCEEDED
.
For more information about client per-call instruments, see OpenTelemetry metrics documentation in GitHub .
Request load sensing metrics
The following metrics provide insights on the effectiveness of your client application's use of request load sensing. Request load sensing metrics can help you balance server loads, optimize resource utilization, and improve client response times. The following metrics are only available with direct connectivity.
storage.googleapis.com/client/grpc/lb/rls/cache_entries
Preview
. The number of entries in the request load
sensing cache.{entry}
-
grpc.target
: indicates the target of the gRPC channel the WRR is used in. -
grpc.lb.rls.server_target
: the target URI of the request load sensing server talks to. -
grpc.lb.rls.instance_uuid
: a universally unique identifier (UUID) for an individual request load sensing client instance. The value isn't meaningful by itself, but it's useful for differentiating between request load sensing client instances in cases where there are either multiple instances in the same gRPC channel or where there are multiple channels to the same target.
storage.googleapis.com/client/grpc/lb/rls/cache_size
Preview
. The current size of the request load sensing
cache.By
-
grpc.target
: the target of the gRPC channel the WRR is used in. -
grpc.lb.rls.server_target
: the target URI of the request load sensing server talks to. -
grpc.lb.rls.instance_uuid
: a UUID for an individual request load sensing client instance. The value isn't meaningful by itself, but it's useful for differentiating between request load sensing client instances in cases where there are either multiple instances in the same gRPC channel or where there are multiple channels to the same target.
storage.googleapis.com/client/grpc/lb/rls/default_target_picks
Preview
. The number of load balancer (LB) picks sent to
the default target.{pick}
-
grpc.target
: indicates the target of the gRPC channel in which request load sensing is used. -
grpc.lb.rls.server_target
: the target URI of the request load sensing server to talk to. -
grpc.lb.rls.data_plane_target
: a target string used by request load sensing for routing data plane traffic. The value is either returned by the request load sensing server for a particular key or configured as the default target in the request load sensing config. -
grpc.lb.pick_result
:the result of an LB pick, such as"complete"
,"fail"
, or"drop"
.
storage.googleapis.com/client/grpc/lb/rls/target_picks
Preview
. The number of LB picks sent to each request load
sensing target. If the default target is also returned by the request load
sensing server, RPCs sent to that target from the cache are counted in
this metric, not in grpc.rls.default_target_picks
.{pick}
-
grpc.target
: the target of the gRPC channel in which request load sensing is used. -
grpc.lb.rls.server_target
: the target URI of the request load sensing server to talk to. -
grpc.lb.rls.data_plane_target
: a target string used by request load sensing for routing data plane traffic. The value is either returned by the request load sensing server for a particular key or configured as the default target in the request load sensing config. -
grpc.lb.pick_result
: the result of an LB pick such as"complete"
,"fail"
, or"drop"
.
storage.googleapis.com/client/grpc/lb/rls/failed_picks
Preview
. The number of LB picks that failed due to either
a failed request load sensing request or the request load sensing
channel being throttled.{pick}
-
grpc.target
: the target of the gRPC channel in which request load sensing is used. -
grpc.lb.rls.server_target
: the target URI of the request load sensing server to talk to.
xDiscovery Service client metrics
The following metrics provide insights on how your client application interacts with the xDiscovery Service (xDS) control plane to discover and configure connections to backend services. xDS metrics can help you track service request latency, monitor configuration updates, and optimize overall xDS performance.
The following metrics are only available with direct connectivity.
storage.googleapis.com/client/grpc/xds_client/connected
Preview
. Measures whether or not the xDS client has a
working ADS stream to the xDS server. For a given server, this metric is
set to 1
when the stream is initially created. If there is a
connectivity failure or when the ADS stream fails without seeing a
response message as per A57
, the metric is set to 0
. Once set to 0
, the metric
will be reset to 1
when the first response
is received on an ADS stream. This metric is only available for
Cloud Client Libraries for C++.{bool}
-
grpc.target
: for clients, indicates the target of the gRPC channel theXdsClient
is used in. For servers, will be the string"#server"
. -
grpc.xds.server
: the target URI of the xDS server theXdsClient
is communicating with.
storage.googleapis.com/client/grpc/xds_client/resource_updates_invalid
Preview
. The number of resources received that were
considered invalid. This metric is only available for
Cloud Client Libraries for C++.{resource}
-
grpc.target
: for clients, indicates the target of the gRPC channel in which theXdsClient
is used. For servers, will be the string"#server"
. -
grpc.xds.server
: the target URI of the xDS server theXdsClient
is communicating with. -
grpc.xds.resource_type
: indicates an xDS resource type, such as"envoy.config.listener.v3.Listener"
.
storage.googleapis.com/client/grpc/xds_client/resource_updates_valid
Preview
. The number of resources received that were
considered valid, even if unchanged.
This metric is only available for Cloud Client Libraries for C++.{resource}
-
grpc.target
: for clients, indicates the target of the gRPC channel in which theXdsClient
is used. For servers, will be the string"#server"
. -
grpc.xds.server
: the target URI of the xDS server theXdsClient
is communicating with. -
grpc.xds.resource_type
: indicates an xDS resource type, such as"envoy.config.listener.v3.Listener"
.
storage.googleapis.com/client/grpc/xds_client/resources
Preview
. The number of xDS resources. This metric is only
available for Cloud Client Libraries for C++.{resource}
-
grpc.target
: for clients, indicates the target of the gRPC channel in which theXdsClient
is used. For servers, will be the string"#server"
. -
grpc.xds.authority
: the xDS authority. The value will be"#old"
for non-xdstp resource names which were identified in the xDS API, before the introduction of thexdstp://
URI representation. -
grpc.xds.cache_state
: indicates the cache state of an xDS resource. -
grpc.xds.resource_type
indicates an xDS resource type, such as"envoy.config.listener.v3.Listener"
.
storage.googleapis.com/client/grpc/xds_client/server_failure
Preview
. The number of xDS servers that are no longer
functioning correctly and have either become unavailable, overloaded, or
are providing incorrect or invalid configuration data. This metric is only
available for Cloud Client Libraries for C++.{failure}
-
grpc.target
: the target URI of the xDS server with theXdsClient
is communicating with. -
grpc.xds.server
: for clients, this indicates the target of the gRPC channel in which theXdsClient
is used. For servers, this is the string"#server"
.
For more information about xDS client metrics, see the xDS-Based global load balancing documentation in GitHub.
Opt out of client-side metrics
You can opt out of client-side metrics if needed.
Java
public GrpcStorageOptions.Builder setEnableGrpcClientMetrics(false enableGrpcClientMetrics)
For more information, see Cloud Client Libraries for Java class GrpcStorageOptions.Builder
method for gRPC client metrics
.
C++
To opt out of client-side metrics for the gRPC API using
Cloud Client Libraries for C++, see Struct EnableGrpcMetricsOption
.
If you're using Bazel
to build your application and want to disable
client-side metrics, set the enable_grpc_metrics
option to false
in
your application's build file.
What's next
-
Learn more about observability in Google Cloud .
-
Learn how to add custom traces and metrics to your application .