Collect metrics from your workloads

This page outlines the process for scraping metrics from workloads in Google Distributed Cloud (GDC) air-gapped environments to facilitate monitoring and data observability.

You can scrape and collect metrics your components produce over time. The monitoring platform offers a custom API to scrape metrics from running workloads within your Distributed Cloud project namespace. To scrape metrics, you deploy a MonitoringTarget custom resource to your project namespace in the Management API server. Upon deployment of this resource, the monitoring platform initiates data collection.

The MonitoringTarget custom resource directs the monitoring pipeline to scrape designated pods within your project. These pods must expose an HTTP endpoint that delivers metrics in a Prometheus exposition format, such as OpenMetrics. The scraped metrics are then displayed in your project's Grafana instance, providing insights into your application's operational state.

To configure the MonitoringTarget custom resource, you must specify the pods within your project namespace for metric collection. You can customize various settings, including the scraping frequency, the pods' metrics endpoint, labels, and annotations.

Before you begin

To get the permissions that you need to manage MonitoringTarget custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated MonitoringTarget roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a project. For more information, see Prepare IAM permissions .

Configure the MonitoringTarget custom resource

The MonitoringTarget custom resource tells the monitoring platform where to collect metrics from. You can specify the pods for which you are collecting metrics, the metrics endpoint of those pods, the scrapping frequency, and any additional settings.

This resource defines the following configurations:

• Targets: The pods and their endpoints within your project that expose metrics.
• Scrape interval: How frequently you want to pull metrics from the selected endpoints.
• Label customizations: Optional rules with any label modifications for metrics.

Choose one of the following methods for specifying metrics endpoints in the MonitoringTarget custom resource:

• Static endpoints: You explicitly declare the endpoint (port, path, scheme) in the MonitoringTarget configuration.

• Annotations: The pod metric endpoint information is retrieved from annotations within the container's Deployment file. This method offers more flexibility if each pod has different endpoints.

Refer to the complete MonitoringTarget specification and the API reference documentation for additional fields and options.

Complete MonitoringTarget specification

The following YAML file shows an example for the complete specification of the MonitoringTarget custom resource. For more information and a complete description of fields, see the API reference documentation .

  apiVersion 
 : 
  
 monitoring.gdc.goog/v1 
 kind 
 : 
  
 MonitoringTarget 
 metadata 
 : 
  
 # Choose the same namespace as the workload pods. 
  
 namespace 
 : 
  
  PROJECT_NAMESPACE 
 
  
 name 
 : 
  
  MONITORING_TARGET_NAME 
 
 spec 
 : 
  
 # Choose matching pattern that identifies pods for this job. 
  
 # Optional 
  
 # Relationship between different selectors: AND 
  
 selector 
 : 
  
 # Choose clusters to consider for this job. 
  
 # Optional: List 
  
 # Default: All clusters applicable to this project. 
  
 # Relationship between different list elements: OR 
  
 matchClusters 
 : 
  
 - 
  
 string 
  
 # Choose pod labels to consider for this job. 
  
 # Optional: Map of key-value pairs. 
  
 # Default: No filtering by label. 
  
 # Relationship between different pairs: AND 
  
 matchLabels 
 : 
  
 key1 
 : 
  
 value1 
  
 # Choose annotations to consider for this job. 
  
 # Optional: Map of key-value pairs 
  
 # Default: No filtering by annotation 
  
 # Relationship between different pairs: AND 
  
 matchAnnotations 
 : 
  
 key1 
 : 
  
 value1 
  
 # Configure the endpoint exposed for this job. 
  
 podMetricsEndpoints 
 : 
  
 # Choose a port either through static value or annotation. 
  
 # Optional 
  
 # Annotation takes priority. 
  
 # Default: static port 80 
  
 port 
 : 
  
 value 
 : 
  
 integer 
  
 annotation 
 : 
  
 string 
  
 # Choose a path either through static value or annotation. 
  
 # Optional 
  
 # Annotation takes priority 
  
 # Default: static path /metrics 
  
 path 
 : 
  
 value 
 : 
  
 string 
  
 annotation 
 : 
  
 string 
  
 # Choose a scheme either through a static value (http or https) or annotation. 
  
 # Optional 
  
 # Annotation takes priority 
  
 # Default: static scheme http 
  
 scheme 
 : 
  
 value 
 : 
  
 string 
  
 annotation 
 : 
  
 string 
  
 # Choose the frequency to scrape the metrics endpoint defined in podMetricsEndpoints 
  
 # Optional 
  
 # Default: 60s 
  
 scrapeInterval 
 : 
  
 string 
  
 # Dynamically rewrite the label set of a target before it gets scraped. 
  
 # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config 
  
 # Optional 
  
 # Default: No filtering by label 
  
 metricsRelabelings 
 : 
  
 - 
  
 sourceLabels 
 : 
  
 - 
  
 string 
  
 separator 
 : 
  
 string 
  
 regex 
 : 
  
 string 
  
 action 
 : 
  
 string 
  
 targetLabel 
 : 
  
 string 
  
 replacement 
 : 
  
 string 
 

Replace the following:

• PROJECT_NAMESPACE : your project namespace.
• MONITORING_TARGET_NAME : the name of the MonitoringTarget definition file.