This page shows how to create alerting policies for Google Distributed Cloud clusters.
Before you begin
You must have the following permissions to create alerting policies:
- monitoring.alertPolicies.create
- monitoring.alertPolicies.delete
- monitoring.alertPolicies.update
You'll have these permissions if you have any one of the following roles :
- monitoring.alertPolicyEditor
- monitoring.editor
- Project editor
- Project owner
To check your roles, go to the IAM page in the Google Cloud console.
Creating a policy: cluster API server down
In this exercise, you create an alerting policy for Kubernetes API servers of clusters. With this policy in place, you can arrange to be notified whenever the API server of a cluster goes down.
-
Download the policy configuration file: apiserver-down.json .
-
Create the policy:
gcloud alpha monitoring policies create --policy-from-file= POLICY_CONFIGReplace POLICY_CONFIG with the path of the configuration file you just downloaded.
-
View your alerting policies:
Console
-
In the Google Cloud console, go to the Monitoringpage.
-
On the left, select Alerting.
-
Under Policies, you can see a list of your alerting policies.
In the list, select Anthos on baremetal API server down (critical)to see details about your new policy. Under Conditions, you can see a description of the policy. For example:
Policy violates when ANY condition is met Anthos on baremetal API server is up
gcloud
gcloud alpha monitoring policies list
The output shows detailed information about the policy. For example:
--- combi ner : OR co n di t io ns : - co n di t io n Mo n i t ori n gQueryLa n guage : dura t io n : 0 s query : | - { t _ 0 : fet ch k 8 s_co nta i ner | me tr ic 'kuber netes .io/a nt hos/up' | f il ter (resource.co nta i ner _ na me =~ 'kube - apiserver') | alig n mea n _alig ner () | group_by 1 m , [ value_up_mea n : mea n (value.up) ] | every 1 m | group_by [ resource.projec t _id , resource.loca t io n , resource.clus ter _ na me ], [ value_up_mea n _aggrega te : aggrega te (value_up_mea n ) ] ; t _ 1 : fet ch k 8 s_co nta i ner :: kuber netes .io/a nt hos/a nt hos_clus ter _i nf o | f il ter (me tr ic.a nt hos_dis tr ibu t io n = 'bareme tal ') | alig n mea n _alig ner () | group_by [ resource.projec t _id , resource.loca t io n , resource.clus ter _ na me ], [ value_a nt hos_clus ter _i nf o_aggrega te : aggrega te (value.a nt hos_clus ter _i nf o) ] | every 1 m } | joi n | value [ t _ 0. value_up_mea n _aggrega te ] | wi n dow 1 m | abse nt _ f or 300 s tr igger : cou nt : 1 displayName : A nt hos o n bareme tal API server is up na me : projec ts /xxxxxx/aler t Policies/ 8497323605386949154 /co n di t io ns / 8497323605386950375 crea t io n Record : mu tate Time : ' 2021-03-17 T 23 : 07 : 18.618778106 Z' mu tate dBy : sharo n @example.com displayName : A nt hos o n bareme tal API server dow n (cri t ical) e na bled : true mu tat io n Record : mu tate Time : ' 2021-03-17 T 23 : 07 : 18.618778106 Z' mu tate dBy : sharo n @example.com na me : projec ts /xxxxxx/aler t Policies/ 8497323605386949154 -
Creating additional alerting policies
This section provides descriptions and configuration files for a set of recommended alerting policies.
To create a policy, follow the same steps that you used in the preceding exercise:
-
Click the link in the right column to download the configuration file.
-
Run
gcloud alpha monitoring policies createto create the policy.
Control plane components availability
| Alert name | Description | Alerting policy definition in Cloud Monitoring |
|---|---|---|
|
Anthos on baremetal API server down (critical)
|
API server has disappeared from metrics target discovery | apiserver-down.json |
|
Anthos on baremetal scheduler down (critical)
|
Scheduler has disappeared from metrics target discovery | scheduler-down.json |
|
Anthos on baremetal controller manager down (critical)
|
Controller manager has disappeared from metrics target discovery | controller-manager-down.json |
Kubernetes system
| Alert name | Description | Alerting policy definition in Cloud Monitoring |
|---|---|---|
|
Anthos on baremetal pod crash looping (critical)
|
Pod is in a crash loop status | pod-crash-looping.json |
|
Anthos on baremetal pod not ready for more than one hour (critical)
|
Pod is in a non-ready state for more than one hour | pod-not-ready-1h.json |
|
Anthos on baremetal persistent volume high usage (critical)
|
Claimed persistent volume is expected to fill up | persistent-volume-usage-high.json |
|
Anthos on baremetal node not ready for more than one hour (critical)
|
Node is in a non-ready state for more than one hour | node-not-ready-1h.json |
|
Anthos on baremetal node cpu usage exceeds 80 percent (critical)
|
Node cpu usage is over 80% | node-cpu-usage-high.json |
|
Anthos on baremetal node memory usage exceeds 80 percent (critical)
|
Node memory usage is over 80% | node-memory-usage-high.json |
|
Anthos on baremetal node disk usage exceeds 80 percent (critical)
|
Node disk usage is over 80% | node-disk-usage-high.json |
Kubernetes performance
| Alert name | Description | Alerting policy definition in Cloud Monitoring |
|---|---|---|
|
Anthos on baremetal API server error count ratio exceeds 10 percent (critical)
|
API server is returning errors for more than 10% of requests | api-server-error-ratio-10-percent.json |
|
Anthos on baremetal API server error count ratio exceeds 5 percent (warning)
|
API server is returning errors for more than 5% of requests | api-server-error-ratio-5-percent.json |
|
Anthos on baremetal etcd leader change too frequent (critical)
|
The etcd leader changes too frequently | etcd-leader-changes-too-frequent.json |
|
Anthos on baremetal etcd proposals failed too frequent (critical)
|
The etcd proposals are failing too frequently | etcd-proposals-failed-too-frequent.json |
|
Anthos on baremetal etcd server is not in quorum (critical)
|
The etcd server is not in quorum | etcd-server-not-in-quorum.json |
Getting notified
After you create an alerting policy, you can define one or more notification channels for the policy. There are several kinds of notification channels. For example, you could be notified by email, a Slack channel, or a mobile app. You can choose the channels that suit your needs.
For instructions about how to configure notification channels, see Managing notification channels .
