Label metrics

Labels let you differentiate the characteristics of a metric. In addition to collecting metrics , the MonitoringTarget custom resource lets you label metrics for optional observability features:

• Send metrics to another project .
• Add new labels to metrics .
• Rename a metric .

For more information on metrics relabeling, see https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/ .

Before you begin

To manage MonitoringTarget custom resources, you must have the necessary permissions.

Request IAM organization roles

Contact your Organization IAM Admin to request the following roles:

• MonitoringTarget PA Creator( monitoringtarget-pa-creator ): create MonitoringTarget custom resources.
• MonitoringTarget PA Editor( monitoringtarget-pa-editor ): edit or modify MonitoringTarget custom resources.
• MonitoringTarget PA Viewer( monitoringtarget-pa-viewer~): view MonitoringTarget` custom resources.

Request IAM project roles

Contact your Project IAM Admin to request the following roles:

• MonitoringTarget Editor( monitoringtarget-editor ): edit or modify MonitoringTarget custom resources in the project namespace.
• MonitoringTarget Viewer( monitoringtarget-viewer ): view MonitoringTarget custom resources in the project namespace.

Send metrics to another project

Container authors can code specific metrics to be sent to a different project, even to projects they don't own. To send a metric to a specific project, add the _gdch_project label to the metric in your code, setting its value to the destination project name.

Alternatively, use the MonitoringTarget custom resource to send all collected metric data to another project. Set the _gdch_project label as a target label in the metricsRelabelings field, with the new project name as the replacement value.

The following code sample shows how to set the _gdch_project target label in the MonitoringTarget custom resource to send metrics to the project named another-project-name :

  apiVersion 
 : 
  
 monitoring.gdc.goog/v1 
 kind 
 : 
  
 MonitoringTarget 
 metadata 
 : 
  
 # Choose the same namespace as the workload pods. 
  
 namespace 
 : 
  
  PROJECT_NAMESPACE 
 
  
 name 
 : 
  
  MONITORING_TARGET_NAME 
 
 spec 
 : 
  
 [ 
 ... 
 ] 
  
 podMetricsEndpoints 
 : 
  
 [ 
 ... 
 ] 
  
 metricsRelabelings 
 : 
  
 - 
  
 action 
 : 
  
 replace 
  
 targetLabel 
 : 
  
 _gdch_project 
  
 replacement 
 : 
  
 another-project-name 
  
 [ 
 ... 
 ] 
 

Replace the following:

• PROJECT_NAMESPACE : your project namespace.
• MONITORING_TARGET_NAME : the name of the MonitoringTarget definition file.

Add new labels to metrics

Labels help with observability and monitoring by identifying metric characteristics. You can add labels that the scraped containers don't initially expose. In the MonitoringTarget custom resource, set the new label as a target and provide a replacement value. This key-value pair is added to all metrics the resource collects.

The following code sample shows how to set the my_new_label target label and configure its value to my_label_value in the MonitoringTarget custom resource:

  apiVersion 
 : 
  
 monitoring.gdc.goog/v1 
 kind 
 : 
  
 MonitoringTarget 
 metadata 
 : 
  
 # Choose the same namespace as the workload pods. 
  
 namespace 
 : 
  
  PROJECT_NAMESPACE 
 
  
 name 
 : 
  
  MONITORING_TARGET_NAME 
 
 spec 
 : 
  
 [ 
 ... 
 ] 
  
 podMetricsEndpoints 
 : 
  
 [ 
 ... 
 ] 
  
 metricsRelabelings 
 : 
  
 - 
  
 action 
 : 
  
 replace 
  
 targetLabel 
 : 
  
 my_new_label 
  
 replacement 
 : 
  
 my_label_value 
  
 [ 
 ... 
 ] 
 

Replace the following:

• PROJECT_NAMESPACE : your project namespace.
• MONITORING_TARGET_NAME : the name of the MonitoringTarget definition file.

Rename a metric

You can rename a container's metric using the MonitoringTarget resource. Set the original metric name and its replacement in the metricsRelabelings field. The original name is treated as a regular expression (regex) and is replaced by the new name.

The following code sample shows how to replace the cpu_usage metric name (regex) with cpu_usage_new_name in the MonitoringTarget custom resource:

  apiVersion 
 : 
  
 monitoring.gdc.goog/v1 
 kind 
 : 
  
 MonitoringTarget 
 metadata 
 : 
  
 # Choose the same namespace as the workload pods. 
  
 namespace 
 : 
  
  PROJECT_NAMESPACE 
 
  
 name 
 : 
  
  MONITORING_TARGET_NAME 
 
 spec 
 : 
  
 [ 
 ... 
 ] 
  
 podMetricsEndpoints 
 : 
  
 [ 
 ... 
 ] 
  
 metricsRelabelings 
 : 
  
 - 
  
 action 
 : 
  
 replace 
  
 regex 
 : 
  
 cpu_usage 
  
 replacement 
 : 
  
 cpu_usage_new_name 
  
 sourceLabels 
 : 
  
 [ 
 __name__ 
 ] 
  
 targetLabel 
 : 
  
 __name__ 
  
 [ 
 ... 
 ] 
 

Replace the following:

• PROJECT_NAMESPACE : your project namespace.
• MONITORING_TARGET_NAME : the name of the MonitoringTarget definition file.