Creating alerting policies

This page shows how to create alerting policies for Google Distributed Cloud clusters.

Before you begin

You must have the following permissions to create alerting policies:

• monitoring.alertPolicies.create
• monitoring.alertPolicies.delete
• monitoring.alertPolicies.update

You'll have these permissions if you have any one of the following roles :

• monitoring.alertPolicyEditor
• monitoring.editor
• Project editor
• Project owner

To check your roles, go to the IAM page in the Google Cloud console.

Creating a policy: admin cluster API server unavailable

In this exercise, you create an alerting policy for Kubernetes API servers of admin clusters. With this policy in place, you can arrange to be notified whenever the API server of an admin cluster is unavailable.

1. Download the policy configuration file: admin-cluster-apiserver-unavailable.json .

2. Create the policy:

   gcloud alpha monitoring policies create --policy-from-file= POLICY_CONFIG 
   

   Replace POLICY_CONFIG with the path of the configuration file you just downloaded.

3. View your alerting policies:

   Console

   1. In the Google Cloud console, go to the Monitoringpage.

      Go to Monitoring

   2. On the left, select Alerting.

   3. Under Policies, you can see a list of your alerting policies.

      In the list, select GKE on-prem admin cluster API server unavailable (critical)to see details about your new policy. Under Conditions, you can see a description of the policy. For example:

      Policy violates when ANY condition is met
      
      GKE on-prem admin cluster API server uptime is absent
      
      GKE on-prem admin cluster API server uptime is less than 99.99% per minute

   gcloud

   gcloud alpha monitoring policies list

   The output shows detailed information about the policy. For example:

     combiner 
    : 
     
    OR 
    conditions 
    : 
    - 
     
    conditionAbsent 
    : 
     
    aggregations 
    : 
     
    - 
     
    alignmentPeriod 
    : 
     
    60 
    s 
     
    crossSeriesReducer 
    : 
     
    REDUCE_MEAN 
     
    groupByFields 
    : 
     
    - 
     
    resource 
    . 
    label 
    . 
    project_id 
     
    - 
     
    resource 
    . 
    label 
    . 
    location 
     
    - 
     
    resource 
    . 
    label 
    . 
    cluster_name 
     
    - 
     
    resource 
    . 
    label 
    . 
    namespace_name 
     
    - 
     
    resource 
    . 
    label 
    . 
    container_name 
     
    perSeriesAligner 
    : 
     
    ALIGN_MAX 
     
    duration 
    : 
     
    300 
    s 
     
    filter 
    : 
     
    resource 
    . 
    type 
     
    = 
     
    "k8s_container" 
     
    AND 
     
    resource 
    . 
    labels 
    . 
    namespace_name 
     
    = 
     
    "kube-system" 
     
    AND 
     
    metric 
    . 
    type 
     
    = 
     
    "kubernetes.io/anthos/container/uptime" 
     
    AND 
     
    resource 
    . 
    label 
    . 
    "container_name" 
    = 
    monitoring 
    . 
    regex 
    . 
    full_match 
    ( 
    "kube-apiserver" 
    ) 
     
    trigger 
    : 
     
    count 
    : 
     
    1 
     
    displayName 
    : 
     
    GKE 
     
    on 
    - 
    prem 
     
    admin 
     
    cluster 
     
    API 
     
    server 
     
    uptime 
     
    is 
     
    absent 
     
    name 
    : 
     
    projects 
    /…/alertPolicies/17065318077071152828/conditions/ 
    17065318077071154437 
    - 
     
    conditionThreshold 
    : 
     
    aggregations 
    : 
     
    - 
     
    alignmentPeriod 
    : 
     
    120 
    s 
     
    crossSeriesReducer 
    : 
     
    REDUCE_MEAN 
     
    groupByFields 
    : 
     
    - 
     
    resource 
    . 
    label 
    . 
    project_id 
     
    - 
     
    resource 
    . 
    label 
    . 
    location 
     
    - 
     
    resource 
    . 
    label 
    . 
    cluster_name 
     
    - 
     
    resource 
    . 
    label 
    . 
    namespace_name 
     
    - 
     
    resource 
    . 
    label 
    . 
    container_name 
     
    perSeriesAligner 
    : 
     
    ALIGN_MAX 
     
    comparison 
    : 
     
    COMPARISON_LT 
     
    duration 
    : 
     
    300 
    s 
     
    filter 
    : 
     
    resource 
    . 
    type 
     
    = 
     
    "k8s_container" 
     
    AND 
     
    resource 
    . 
    labels 
    . 
    namespace_name 
     
    = 
     
    "kube-system" 
     
    AND 
     
    metric 
    . 
    type 
     
    = 
     
    "kubernetes.io/anthos/container/uptime" 
     
    AND 
     
    resource 
    . 
    label 
    . 
    "container_name" 
    = 
    monitoring 
    . 
    regex 
    . 
    full_match 
    ( 
    "kube-apiserver" 
    ) 
     
    thresholdValue 
    : 
     
    119.0 
     
    trigger 
    : 
     
    count 
    : 
     
    1 
     
    displayName 
    : 
     
    GKE 
     
    on 
    - 
    prem 
     
    admin 
     
    cluster 
     
    API 
     
    server 
     
    uptime 
     
    is 
     
    less 
     
    than 
     
    99.99 
    % 
     
    per 
     
    minute 
     
    name 
    : 
     
    projects 
    /…/alertPolicies/17065318077071152828/conditions/ 
    17065318077071151950 
    creationRecord 
    : 
     
    mutateTime 
    : 
     
    … 
     
    mutatedBy 
    : 
     
    … 
    displayName 
    : 
     
    GKE 
     
    on 
    - 
    prem 
     
    admin 
     
    cluster 
     
    API 
     
    server 
     
    unavailable 
     
    ( 
    critical 
    ) 
    enabled 
    : 
     
    true 
    mutationRecord 
    : 
     
    mutateTime 
    : 
     
    … 
     
    mutatedBy 
    : 
     
    … 
    name 
    : 
     
    projects 
    /xxxxxx/alertPolicies/ 
    17065318077071152828 
    
   

Creating additional alerting policies

This section provides descriptions and configuration files for a set of recommended alerting policies.

To create a policy, follow the same steps that you used in the preceding exercise:

1. Click the link in the right column to download the configuration file.

2. Run gcloud alpha monitoring policies create to create the policy.

Admin cluster control plane components availability

User cluster control plane components availability

The user cluster control plane alerts are based on metrics. For most cluster metrics, the cluster_name field is the name of the cluster itself. But for user cluster control plane metrics, the cluster_name field is the name of the admin cluster, and the namespace_name field is the name of the user cluster.

You can see this in a screenshot under Create a control plane uptime dashboard .

Kubernetes system

Kubernetes performance

Getting notified

After you create an alerting policy, you can define one or more notification channels for the policy. There are several kinds of notification channels. For example, you could be notified by email, a Slack channel, or a mobile app. You can choose the channels that suit your needs.

For instructions about how to configure notification channels, see Managing notification channels .