Configure cluster notifications for third-party services

Slack notifications

To set up Slack notifications, you must create a Slack application, activate Incoming Webhooks for the application, and install the application to a Slack workspace.

Create the Slack application

1. Join a Slack workspace , either by registering with your email or by using an invitation sent by a Workspace Admin.

2. Sign in to Slack using your workspace name and your Slack account credentials.

3. Create a new Slack app

   1. In the Create an appdialog, click From scratch.
   2. Specify an App Nameand choose your Slack workspace.
   3. Click Create App.
   4. Under Add features and functionality, click Incoming Webhooks.
   5. Click the Activate Incoming Webhookstoggle.
   6. In the Webhook URLs for Your Workspacesection, click Add New Webhook to Workspace.
   7. On the authorization page that opens, select a channel to receive notifications.
   8. Click Allow.
   9. A webhook for your Slack application is displayed in the Webhook URLs for Your Workspacesection. Save the URL for later.

Write the Cloud Run function

When GKE publishes a cluster notification to a Pub/Sub topic, the event triggers Cloud Run functions to send a Slack notification.

1. Create a new directory named gke_slack and change directory into it:

    mkdir  
   ~/gke_slack && 
    cd 
     
    $_ 
    
   

2. Create the following files in the gke_slack directory:

   index.js

     const 
     
    functions 
     
    = 
     
    require 
    ( 
    '@google-cloud/functions-framework' 
    ); 
    const 
     
    { 
     
    IncomingWebhook 
     
    } 
     
    = 
     
    require 
    ( 
    '@slack/webhook' 
    ); 
    const 
     
    url 
     
    = 
     
    process 
    . 
    env 
    . 
    SLACK_WEBHOOK 
    ; 
    const 
     
    webhook 
     
    = 
     
    new 
     
    IncomingWebhook 
    ( 
    url 
    ); 
    // Optionally filter what notification types to forward to Slack. 
    // If empty, all types will be allowed. 
    const 
     
    allowedTypeURLs 
     
    = 
     
    []; 
    // Register a CloudEvent callback with the Functions Framework that will 
    // be executed when the Pub/Sub trigger topic receives a message. 
    functions 
    . 
    cloudEvent 
    ( 
    'slackNotifier' 
    , 
     
    pubSubEvent 
     
    = 
   >  
    { 
     
    const 
     
    data 
     
    = 
     
    decode 
    ( 
    pubSubEvent 
    . 
    data 
    . 
    message 
    . 
    data 
    ); 
     
    // Send message to Slack. 
     
    if 
     
    ( 
    isAllowedType 
    ( 
    pubSubEvent 
    . 
    data 
    . 
    message 
    . 
    attributes 
    )) 
     
    { 
     
    const 
     
    message 
     
    = 
     
    createSlackMessage 
    ( 
    data 
    , 
     
    pubSubEvent 
    . 
    data 
    . 
    message 
    . 
    attributes 
    ); 
     
    webhook 
    . 
    send 
    ( 
    message 
    ); 
     
    } 
    }); 
    // decode decodes a pubsub event message from base64. 
    const 
     
    decode 
     
    = 
     
    ( 
    data 
    ) 
     
    = 
   >  
    { 
     
    return 
     
    Buffer 
    . 
    from 
    ( 
    data 
    , 
     
    'base64' 
    ). 
    toString 
    (); 
    } 
    // isAllowedType can be used to filter out messages that don't match the 
    // allowed type URLs. If allowedTypeURLs is empty, it allows all types. 
    const 
     
    isAllowedType 
     
    = 
     
    ( 
    attributes 
    ) 
     
    = 
   >  
    { 
     
    if 
     
    ( 
    allowedTypeURLs 
    . 
    length 
     
    == 
     
    0 
    ) 
     
    { 
     
    return 
     
    true 
    ; 
     
    } 
     
    for 
     
    ( 
    var 
     
    x 
     
    in 
     
    allowedTypeURLs 
    ) 
     
    { 
     
    if 
     
    ( 
    attributes 
    [ 
    'type_url' 
    ] 
     
    == 
     
    allowedTypeURLs 
    [ 
    x 
    ]) 
     
    { 
     
    return 
     
    true 
    ; 
     
    } 
     
    } 
     
    return 
     
    false 
    ; 
    } 
    // createSlackMessage creates a message from a data object. 
    const 
     
    createSlackMessage 
     
    = 
     
    ( 
    data 
    , 
     
    attributes 
    ) 
     
    = 
   >  
    { 
     
    // Write the message data and attributes. 
     
    text 
     
    = 
     
    ` 
    ${ 
    data 
    } 
    ` 
     
    for 
     
    ( 
    var 
     
    key 
     
    in 
     
    attributes 
    ) 
     
    { 
     
    if 
     
    ( 
    attributes 
    . 
    hasOwnProperty 
    ( 
    key 
    )) 
     
    { 
     
    text 
     
    = 
     
    text 
     
    + 
     
    `\n\t\` 
    ${ 
    key 
    } 
    : 
    ${ 
    attributes 
    [ 
    key 
    ] 
    } 
    \`` 
     
    } 
     
    } 
     
    const 
     
    message 
     
    = 
     
    { 
     
    text 
    : 
     
    text 
    , 
     
    mrkdwn 
    : 
     
    true 
    , 
     
    }; 
     
    return 
     
    message 
    ; 
    } 
    
   

   SLACK_WEBHOOK is a Cloud Run functions environment variable specifying the webhook URL created for your Slack application. You define the environment variable when you deploy the function.

   The webhook listens for and receives messages from Cloud Run functions. When GKE sends a cluster notification to Pub/Sub (the event), the function sends a message (the trigger) to the webhook URL, which in turn sends the message to the configured Slack workspace.

   You can expand the message in the createSlackMessage function to include much more, including text formatting and images. The isAllowedType function is provided to enable basic filtering of notifications by the type URL. You can specify what type URLs to allow in allowedTypeURLs . This function isn't necessary if you already filtered notifications , either in GKE or in your Pub/Sub subscription.

   package.json

     { 
     
    "name" 
    : 
     
    "gke-slack" 
    , 
     
    "version" 
    : 
     
    "0.0.1" 
    , 
     
    "description" 
    : 
     
    "Slack integration for GKE, using Cloud Run functions" 
    , 
     
    "main" 
    : 
     
    "index.js" 
    , 
     
    "dependencies" 
    : 
     
    { 
     
    "@slack/webhook" 
    : 
     
    "6.1.0" 
    , 
     
    "@google-cloud/functions-framework" 
    : 
     
    "^3.0.0" 
     
    } 
    } 
    
   

   package.json describes the following attributes of the program:

   • Name, version, and description
   • Primary runtime file(s)
   • Dependencies

   You can add more dependencies, requirements, and other information as needed.

You should now have the index.js and package.json files in the gke_slack directory.

Deploy the Cloud Run function

You can deploy the Cloud Run function using either the Google Cloud CLI or the Google Cloud console.

 gcloud  
functions  
deploy  
slackNotifier  
 \ 
  
--gen2  
 \ 
  
--trigger-topic = 
 TOPIC_NAME 
  
 \ 
  
--runtime = 
nodejs14  
 \ 
  
--entry-point = 
slackNotifier  
 \ 
  
--region = 
 REGION 
  
 \ 
  
--source = 
.  
 \ 
  
--set-env-vars = 
 "SLACK_WEBHOOK= WEBHOOK_URL 
" 
 

 Deploying function…
availableMemoryMb: 256
entryPoint: slackNotifier
environmentVariables:
  SLACK_WEBHOOK: https://hooks.slack.com/services/…
eventTrigger:
  eventType: google.pubsub.topic.publish
  failurePolicy: {}
  resource: projects/ PROJECT_ID 
/topics/ TOPIC_NAME 
service: pubsub.googleapis.com
labels:
  deployment-tool: cli-gcloud
name: projects/ PROJECT_ID 
/locations/us-central1/functions/slackNotifier
runtime: nodejs10
serviceAccountEmail: PROJECT_ID 
@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/…
status: ACTIVE
timeout: 60s
updateTime: 'YYYY-MM-DDThh:mm:ssZ'
versionId: '1' 

After you've completed deployment of the Cloud Run function, you will receive a Slack notification whenever GKE sends a cluster notification.

Verify Slack notifications

If you used an Autopilot cluster, verify notifications as follows:

1. Start a control plane upgrade .
2. Wait for GKE to automatically upgrade your nodes to the new version. The time taken might vary based on configured maintenance windows and exclusions.
3. After GKE upgrades your nodes, check Slack for a message.

If you used a Standard cluster, verify notifications as follows:

1. Upgrade a specific node pool to a new version . If you don't want to change the GKE version on your nodes, you can upgrade to the same version currently on the nodes.

2. After GKE upgrades your nodes, check Slack for a message.

The Slack notification is similar to the following:

 Master is upgrading to version 1.20.10-gke.301.
    cluster_location: us-central1
    cluster_name: pubsub-cluster
    payload: {"resourceType":"MASTER", "operation":"operation-1632775054313-45128f4f", "operationStartTime":"2021-09-27T20:37:34.313742491Z", "currentVersion":"1.20.9-gke.1001", "targetVersion":"1.20.10-gke.301"}
    project_id: 729788050015
    type_url: type.googleapis.com/google.container.v1beta1.UpgradeEvent