Annotate notifications with user-defined documentation

This page describes how you can configure your alerting policy documentation so that notifications provide incident responders with resources and additional information for incident resolution.

Documentation structure

The documentation of an alerting policy consists of a subject, content, and links. You can configure documentation in the Google Cloud console, the Cloud Monitoring API, and the Google Cloud CLI.

Subjects

The subjectof your documentation appears in the subject of notifications for incidents related to your alerting policy. Notification recipients can manage and sort their notifications by subject.

Subject lines are limited to 255 characters. If you don't define a subject in your documentation, then Cloud Monitoring determines the subject line. Subject lines support plain text and variables .

Content

The contentof your documentation appears in the following notification types:

• Email, in the Policy Documentationsection
• PagerDuty
• Pub/Sub
• Slack
• Webhooks

We recommend configuring your content so that incident responders can view remediation steps and incident information in notifications related to your alerting policy. For example, you might configure the documentation to include a summary of the incident and information about relevant resources.

Documentation content supports the following:

• Plain text
• Variables
• Channel-specific controls

• Markdown

Links

You can add links to your documentation so that incident responders can access resources such as playbooks, repositories, and Google Cloud dashboards from a notification.

The Cloud Monitoring API lets you define an object that contains the most relevant links for responders. While the Google Cloud console doesn't have a field specifically for links, you can add a section for links in your documentation body.

  "links" 
  
 [ 
  
 { 
  
 "displayName" 
 : 
  
 "Playbook" 
 , 
  
 "url" 
 : 
  
 "https://myownpersonaldomain.com/playbook?name=${resource.type}" 
  
 } 
 ] 
 

### Troubleshooting and Debug References

Playbook: https://myownpersonaldomain.com/playbook?name=${resource.type}

Markdown in documentation content

You can use Markdown to format your documentation content. Documentation content supports the following subset of Markdown tagging:

• Headers, indicated by initial hash characters.
• Unordered lists, indicated by initial plus, minus, or asterisk characters.
• Ordered lists, indicated by an initial number followed by a period.
• Italic text, indicated by single underscores or asterisks around a phrase.
• Bold text, indicated by double underscores or asterisks around a phrase.
• Links, indicated by [link text](url) syntax. To add links to your notification, we recommend that you use the Cloud Monitoring API and configure the Link object.

For more information about this tagging, see any Markdown reference, for example, Markdown guide .

Variables in documentation

To customize the text in your documentation, you can use variables of the form ${varname} . When the documentation is sent with a notification, the string ${varname} is replaced with a value drawn from the corresponding Google Cloud resource, as described in the following table.

¹ For example, ${resource.label.zone} is replaced with the value of the zone label. The values of these variables are subject to grouping; see null values for more information.
² To retrieve the value of the project_id label on a monitored resource in the alerting policy, use ${resource.project} .
³ You can't access user-defined resource metadata labels by using resource.label. KEY . Use metadata.user_label. KEY instead.

Usage notes

• Only the variables in the table are supported. You can't combine them into more complex expressions, like ${varname1 + varname2} .
• To include the literal string ${ in your documentation, escape the $ symbol with a second $ symbol, and $${ renders as ${ in your documentation.
• These variables are replaced by their values only in notifications sent through notification channels. In the Google Cloud console, when the documentation is shown, you see the variables, not the values. Examples in the console include the descriptions of incidents and the preview of the documentation when creating an alerting policy.
• Verify that the aggregation settings of the condition don't eliminate the label. If the label is eliminated, then the value of the label in the notification is null . For more information, see Variable for a metric label is null .

null values

Values for the metric.* , resource.* and metadata.* variables are derived from time series. Their values can be null if no values are returned from the time series query.

• The resource.label. KEY and metric.label. KEY variables can have null values if your alerting policy uses cross-series aggregation (reduction), for example, calculating the SUM across each of the time-series that match a filter. When using cross-series aggregation, any labels not used in grouping are dropped and as a result they render as null when the variable is replaced with its value. All labels are retained when there is no cross-series aggregation. For more information, see Variable for a metric label is null .

• Values for metadata.* variables are available only if the labels are explicitly included in a condition's filter or grouping for cross-series aggregation. That is, you must refer to the metadata label in either filter or grouping for it to have a value for the template.

Variable resolution

Variables in documentation templates are resolved only in the notifications sent by using the following notification channels:

• Email
• Google Chat
• Slack
• Pub/Sub, JSON schema version 1.2
• Webhooks, JSON schema version 1.2
• PagerDuty, JSON schema version 1.2

Channel controls

The text in the documentation field can also include special characters used by the notification channel itself to control formatting and notifications.

For example, Slack uses @ for mentions. You can use @ to link the notification to a specific user ID. Mentions can't include names. Suppose you include a string like this in the documentation field:

 <@backendoncall>  
Incident  
created  
based  
on  
policy  
 ${ 
 policy 
 . 
 display_name 
 } 
 

When the documentation field is received by the relevant Slack channel as part of the notification, the previous string causes Slack to send an additional message to the user ID backendoncall . The message sent by Slack to the user could contain relevant information from the notification; for example, "Incident created based on policy High CPU rate of change".

These additional options are specific to the channels; for more information on what may be available, consult the documentation provided by the channel vendor.

Example

The following example shows Google Cloud console and Cloud Monitoring API versions of template documentation for a CPU utilization alerting policy. These examples use an email for the notification channel type . The documentation templates include several variables to summarize the incident and to reference the alerting policy and condition REST resources.

   
 "documentation" 
 : 
  
 { 
  
 "content" 
 : 
  
 "### CPU utilization exceeded\n\n### Summary\n\nThe ${metric.display_name} of the ${resource.type} ${resource.label.instance_id} in the project ${resource.project} has exceeded 5% for over 60 seconds.\n\n#### Additional resource information\n\nCondition resource name: ${condition.name}  \nAlerting policy resource name: ${policy.name}" 
 , 
  
 "mimeType" 
 : 
  
 "text/markdown" 
 , 
  
 "subject" 
 : 
  
 "Alert: ${metric.display_name} exceeded" 
 , 
  
 "links" 
 : 
  
 [ 
  
 { 
  
 "displayName" 
 : 
  
 "Playbook" 
 , 
  
 "url" 
 : 
  
 "https://myownpersonaldomain.com/playbook?name=${resource.type}" 
  
 }, 
  
 { 
  
 "displayName" 
 : 
  
 "Repository with debug scripts" 
 , 
  
 "url" 
 : 
  
 "https://altostrat.com" 
  
 }, 
  
 { 
  
 "displayName" 
 : 
  
 "Google Cloud dashboard" 
 , 
  
 "url" 
 : 
  
 "https://example.com" 
  
 } 
  
 ] 
  
 } 
 

### CPU utilization exceeded

#### Summary

The ${metric.display_name} of the ${resource.type}
${resource.label.instance_id} in the project ${resource.project} has
exceeded 5% for over 60 seconds.

#### Additional resource information

Condition resource name: ${condition.name}  
Alerting policy resource name: ${policy.name}  

#### Troubleshooting and Debug References

Playbook: https://myownpersonaldomain.com/playbook?name=${resource.type}  
Repository with debug scripts: https://altostrat.com  
${resource.type} dashboard: https://example.com