Enable application logging and monitoring

This page shows how to configure a cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring and Managed Service for Prometheus .

Enable Managed Service for Prometheus

The configuration for Managed Service for Prometheus is specified in a Stackdriver object named stackdriver . For additional information, including best practices and troubleshooting, see the Managed Service for Prometheus documentation .

To configure the stackdriver object to enable Google Cloud Managed Service for Prometheus:

1. Open the stackdriver object for editing:

    kubectl  
   --kubeconfig = 
    CLUSTER_KUBECONFIG 
     
    \ 
     
   --namespace  
   kube-system  
   edit  
   stackdriver  
   stackdriver 
   

   Replace CLUSTER_KUBECONFIG with the path of your cluster kubeconfig file.

2. Under spec , set enableGMPForApplications to true :

     apiVersion 
    : 
     
    addons.gke.io/v1alpha1 
    kind 
    : 
     
    Stackdriver 
    metadata 
    : 
     
    name 
    : 
     
    stackdriver 
     
    namespace 
    : 
     
    kube-system 
    spec 
    : 
     
    projectID 
    : 
     
    ... 
     
    clusterName 
    : 
     
    ... 
     
    clusterLocation 
    : 
     
    ... 
     
    proxyConfigSecretName 
    : 
     
    ... 
      
    enableGMPForApplications 
    : 
     
    true 
     
    enableVPC 
    : 
     
    ... 
     
    optimizedMetrics 
    : 
     
    true 
    
   

3. Save and close the edited file.

   The Google-managed Prometheus components start automatically in the cluster in the gmp-system namespace.

4. Check the Google-managed Prometheus components:

    kubectl  
   --kubeconfig = 
    CLUSTER_KUBECONFIG 
     
   --namespace  
   gmp-system  
   get  
   pods 
   

   The output of this command is similar to the following:

    NAME                              READY   STATUS    RESTARTS        AGE
   collector-abcde                   2/2     Running   1 (5d18h ago)   5d18h
   collector-fghij                   2/2     Running   1 (5d18h ago)   5d18h
   collector-klmno                   2/2     Running   1 (5d18h ago)   5d18h
   gmp-operator-68d49656fc-abcde     1/1     Running   0               5d18h
   rule-evaluator-7c686485fc-fghij   2/2     Running   1 (5d18h ago)   5d18h 
   

Managed Service for Prometheus supports rule evaluation and alerting. To set up rule evaluation, see Rule evaluation .

Run an example application

The managed service provides a manifest for an example application, prom-example , that emits Prometheus metrics on its metrics port. The application uses three replicas.

To deploy the application:

1. Create the gmp-test namespace for resources that you create as part of the example application:

    kubectl  
   --kubeconfig = 
    CLUSTER_KUBECONFIG 
     
   create  
   ns  
   gmp-test 
   

2. Apply the application manifest with the following command:

    kubectl  
   -n  
   gmp-test  
   apply  
    \ 
     
   -f  
   https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.1/examples/example-app.yaml 
   

Configure a PodMonitoring resource

In this section, you configure a PodMonitoring custom resource to capture metrics data emitted by the example application and send it to Managed Service for Prometheus. The PodMonitoring custom resource uses target scraping. In this case, the collector agents scrape the /metrics endpoint to which the sample application emits data.

A PodMonitoring custom resource scrapes targets in the namespace in which it's deployed only. To scrape targets in multiple namespaces, deploy the same PodMonitoring custom resource in each namespace. You can verify the PodMonitoring resource is installed in the intended namespace by running the following command:

 kubectl  
--kubeconfig  
 CLUSTER_KUBECONFIG 
  
get  
podmonitoring  
-A 

For reference documentation about all the Managed Service for Prometheus custom resources, see the prometheus-engine/doc/api reference .

The following manifest defines a PodMonitoring resource, prom-example , in the gmp-test namespace. The resource finds all Pods in the namespace that have the label app with the value prom-example . The matching Pods are scraped on a port named metrics , every 30 seconds, on the /metrics HTTP path.

  apiVersion 
 : 
  
 monitoring.googleapis.com/v1 
 kind 
 : 
  
 PodMonitoring 
 metadata 
 : 
  
 name 
 : 
  
 prom-example 
 spec 
 : 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 app 
 : 
  
 prom-example 
  
 endpoints 
 : 
  
 - 
  
 port 
 : 
  
 metrics 
  
 interval 
 : 
  
 30s 
 

To apply this resource, run the following command:

 kubectl  
--kubeconfig  
 CLUSTER_KUBECONFIG 
  
-n  
gmp-test  
apply  
 \ 
  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.1/examples/pod-monitoring.yaml 

Managed Service for Prometheus is now scraping the matching Pods.

Query metrics data

The simplest way to verify that your Prometheus data is being exported is to use PromQL queries in the Metrics Explorerin the Google Cloud console.

To run a PromQL query, do the following:

1. In the Google Cloud console, go to the Monitoringpage or click the following button:

   Go to Monitoring

2. In the navigation pane, select Metrics Explorer.

3. Use Prometheus Query Language (PromQL) to specify the data to display on the chart:

   1. In the toolbar of the Select a metricpane, select Code Editor.

   2. Select PromQLin the Languagetoggle. The language toggle is at the bottom of the Code Editorpane.

   3. Enter your query into the query editor. For example, to chart the average number of seconds CPUs spent in each mode over the past hour, use the following query:

        avg 
       ( 
       rate 
       ( 
       kubernetes_io 
       : 
       anthos_container_cpu_usage_seconds_total 
       { 
       monitored_resource 
       = 
       " 
       k8s_node 
       "}[ 
       1h 
       ] 
       )) 
       
      

   For more information about using PromQL, see PromQL in Cloud Monitoring .

The following screenshot shows a chart that displays the anthos_container_cpu_usage_seconds_total metric:

If you collect large amounts of data, you might want to filter exported metrics to keep down costs.

Enable Cloud Logging for user applications

The configuration for Cloud Logging and Cloud Monitoring is held in a Stackdriver object named stackdriver .

1. Open the stackdriver object for editing:

    kubectl  
   --kubeconfig = 
    CLUSTER_KUBECONFIG 
     
    \ 
     
   --namespace  
   kube-system  
   edit  
   stackdriver  
   stackdriver 
   

   Replace CLUSTER_KUBECONFIG with the path of your user cluster kubeconfig file.

2. In the spec section, set enableCloudLoggingForApplications to true :

     apiVersion 
    : 
     
    addons.gke.io/v1alpha1 
     
    kind 
    : 
     
    Stackdriver 
     
    metadata 
    : 
     
    name 
    : 
     
    stackdriver 
     
    namespace 
    : 
     
    kube-system 
     
    spec 
    : 
     
    projectID 
    : 
     
    ... 
     
    clusterName 
    : 
     
    ... 
     
    clusterLocation 
    : 
     
    ... 
     
    proxyConfigSecretName 
    : 
     
    ... 
      
    enableCloudLoggingForApplications 
    : 
     
    true 
     
    enableVPC 
    : 
     
    ... 
     
    optimizedMetrics 
    : 
     
    true 
    
   

3. Save and close the edited file.

Run an example application

In this section, you create an application that writes custom logs.

1. Save the following Deployment manifests to a file named my-app.yaml .

     apiVersion 
    : 
     
    apps/v1 
    kind 
    : 
     
    Deployment 
    metadata 
    : 
     
    name 
    : 
     
    "monitoring-example" 
     
    namespace 
    : 
     
    "default" 
     
    labels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
    spec 
    : 
     
    replicas 
    : 
     
    1 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
     
    template 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
     
    spec 
    : 
     
    containers 
    : 
     
    - 
     
    image 
    : 
     
    gcr.io/google-samples/prometheus-dummy-exporter:latest 
     
    name 
    : 
     
    prometheus-example-exporter 
     
    imagePullPolicy 
    : 
     
    Always 
     
    command 
    : 
     
    - 
     
    /bin/sh 
     
    - 
     
    -c 
     
    - 
     
    ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    100m 
    
   

2. Create the Deployment

    kubectl  
   --kubeconfig  
    CLUSTER_KUBECONFIG 
     
   apply  
   -f  
   my-app.yaml 
   

View application logs

Enable Logging and Monitoring for user applications (Legacy)

Note, it's strongly recommended to use the above enableGMPForApplications and enableCloudLoggingForApplications to enable monitoring and logging for user applications.

The following steps still work but aren't recommended. Please read this known issue before using the following steps.

To enable Logging and Monitoring for your applications, use the spec.clusterOperations.enableApplication field in the cluster configuration file.

1. Update the cluster configuration file to set enableApplication to true :

     apiVersion 
    : 
     
    v1 
    kind 
    : 
     
    Namespace 
    metadata 
    : 
     
    name 
    : 
     
    cluster-user-basic 
    --- 
    apiVersion 
    : 
     
    baremetal.cluster.gke.io/v1 
    kind 
    : 
     
    Cluster 
    metadata 
    : 
     
    name 
    : 
     
    user-basic 
     
    namespace 
    : 
     
    cluster-user-basic 
    spec 
    : 
     
    type 
    : 
     
    user 
     
    ... 
     
    clusterOperations 
    : 
     
    projectID 
    : 
     
    project-fleet 
     
    location 
    : 
     
    us-central1 
      
    enableApplication 
    : 
     
    true 
     
    ... 
    
   

2. Use bmctl update to apply your changes:

    bmctl  
   update  
   cluster  
   -c  
    CLUSTER_NAME 
     
   --admin-kubeconfig = 
    ADMIN_KUBECONFIG 
    
   

   Replace the following:

   • CLUSTER_NAME : the name of the cluster to upgrade.
   • ADMIN_KUBECONFIG : the path to the admin cluster kubeconfig file.

Annotate workloads

To enable the collection of custom metrics from an application, add the prometheus.io/scrape: "true" annotation to the application's Service or Pod manifest, or add the same annotation to the spec.template section in Deployment or DaemonSet manifest in order to be passed to their Pods.

Run an example application

In this section, you create an application that writes custom logs and exposes a custom metric.

1. Save the following Service and Deployment manifests to a file named my-app.yaml . Notice that the Service has the annotation prometheus.io/scrape: "true" :

     kind 
    : 
     
    Service 
    apiVersion 
    : 
     
    v1 
    metadata 
    : 
     
    name 
    : 
     
    "monitoring-example" 
     
    namespace 
    : 
     
    "default" 
     
    annotations 
    : 
     
    prometheus.io/scrape 
    : 
     
    "true" 
    spec 
    : 
     
    selector 
    : 
     
    app 
    : 
     
    "monitoring-example" 
     
    ports 
    : 
     
    - 
     
    name 
    : 
     
    http 
     
    port 
    : 
     
    9090 
    --- 
    apiVersion 
    : 
     
    apps/v1 
    kind 
    : 
     
    Deployment 
    metadata 
    : 
     
    name 
    : 
     
    "monitoring-example" 
     
    namespace 
    : 
     
    "default" 
     
    labels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
    spec 
    : 
     
    replicas 
    : 
     
    1 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
     
    template 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    app 
    : 
     
    "monitoring-example" 
     
    spec 
    : 
     
    containers 
    : 
     
    - 
     
    image 
    : 
     
    gcr.io/google-samples/prometheus-dummy-exporter:latest 
     
    name 
    : 
     
    prometheus-example-exporter 
     
    imagePullPolicy 
    : 
     
    Always 
     
    command 
    : 
     
    - 
     
    /bin/sh 
     
    - 
     
    -c 
     
    - 
     
    ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    100m 
    
   

2. Create the Deployment and the Service:

    kubectl  
   --kubeconfig  
    CLUSTER_KUBECONFIG 
     
   apply  
   -f  
   my-app.yaml 
   

View application logs

View application metrics in the Google Cloud console

Your example application exposes a custom metric named example_monitoring_up . You can view the values of that metric in the Google Cloud console.

1. Go to the Metrics explorer in the Google Cloud console.

   Go to the Metrics explorer

2. For Resource type, select Kubernetes Pod or Kubernetes Container .

3. For metric, select external.googleapis.com/prometheus/example_monitoring_up .

4. In the chart, you can see that example_monitoring_up has a repeated value of 1.