Manage host events across VMs

This document explains how to use the host maintenance features that are available from the Cluster Director suite. It explains how to monitor, plan for, and perform scheduled maintenance on virtual machine (VM) instances. To manage maintenance on your reserved blocks of capacity, whether or not VMs are running on them, see instead Manage host events across reservations .

When you proactively manage upcoming maintenance host events on your VMs, you can minimize disruptions and maintain optimal performance.

Before you begin

Select the tab for how you plan to use the samples on this page:

Required roles

To get the permissions that you need to manage host maintenance events across VMs, ask your administrator to grant you the following IAM roles:

• Compute Admin ( roles/compute.admin ) on the project
• For read-only access to System Event audit logs: Logs Viewer ( roles/logging.viewer ) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations .

These predefined roles contain the permissions required to manage host maintenance events across VMs. To see the exact permissions that are required, expand the Required permissionssection:

You might also be able to get these permissions with custom roles or other predefined roles .

Overview

To optimize the maintenance of your VMs, complete the following steps:

1. Understand host maintenance. Learn about the frequency and maintenance behavior of your VMs based on their machine series. This information helps you minimize disruptions to your workloads.

2. Set up notification alerts. Create log-based alerts to receive notifications when maintenance for your VMs is scheduled, started, or completed. This approach helps you proactively plan your activities and avoid unexpected downtime.

3. Manage maintenance across VMs. View if maintenance is scheduled for your VMs. If needed, you can manually start maintenance across your VMs. This process helps you increase the resilience of your workloads to host events, prevent downtime, and maximize the availability of your applications.

Understand host maintenance

During the lifecycle of a Compute Engine instance , the host machine that your instance runs on undergoes multiple host events . A host event can include the regular maintenance of Compute Engine infrastructure, or in rare cases, a host error. Compute Engine also applies some non-disruptive lightweight upgrades for the hypervisor and network in the background.

The following table describes the host maintenance features for accelerator-optimized machine types:

Compute Engine might perform maintenance more frequently.

Set up notification alerts for VMs

You can get notified about scheduled, started, or completed maintenance events for your VMs by creating log-based alerting policies .

To create an alert for the maintenance events of your VMs, complete the following procedure. Repeat this procedure for each alert that you want to create.

1. In the Google Cloud console, go to the Logs Explorer page:

   Go to Logs Explorer

   If you use the search bar to find this page, then select the result whose subheading is Logging .

2. Click the Show querytoggle to the on position.

3. In the Querypane, build one of the following queries. These queries filter log entries to identify specific maintenance events. If you want to use multiple queries, repeat this procedure to create an unique alert for each query.

   • To receive alerts when maintenance for a VM is scheduled:

      protoPayload.methodName="compute.instances.upcomingMaintenance" severity>=DEFAULT
     protoPayload.status.message =~ "scheduled" 
     

   • To receive alerts when the maintenance window for a VM has opened:

      protoPayload.methodName="compute.instances.upcomingMaintenance" severity>=DEFAULT
     protoPayload.status.message =~ "ongoing" 
     

   • To receive alerts when maintenance for a VM has started:

      protoPayload.methodName="compute.instance.terminateOnHostMaintenance" severity>=DEFAULT 
     

   • To receive alerts when maintenance for a VM has completed:

      protoPayload.methodName="compute.instances.upcomingMaintenance" severity>=DEFAULT
     protoPayload.status.message =~ "completed" 
     

4. To validate the query, click Run query. If the query is valid, then the Query resultspane displays log entries that match the query.

5. In the Queryresults toolbar, click the Actionslist, and then select add_alert Create log alert. The Create logs-based alert policypane appears.

6. In the Alert detailssection, do the following:

   1. In the Alert Policy Namefield, enter a name for the policy.

   2. In the Policy severity levellist, select Warning(or a higher severity).

   3. Click Next.

7. In the Choose logs to include in the alertsection, click Next.

8. In the Set notification frequency and autoclose durationsection, specify the following:

   1. In the Time between notificationslist, select how often you want to be notified.

   2. In the Incident autoclose durationlist, select after how long Cloud Logging stops sending notifications and automatically closes the incident.

   3. Click Next.

9. In the Who should be notified?section, specify a notification channel for Logging to send notifications to.

10. Click Save.

To view examples of maintenance event notifications in the Logs Explorer, see Examples of maintenance notifications in the Compute Engine documentation.

Manage maintenance across VMs

You can view and control maintenance for your VMs by doing one or more of the following:

• To check the state and scheduled time of upcoming maintenance for your VMs, view the maintenance state of VMs .

• To immediately start maintenance on your VMs, rather than waiting for their scheduled maintenance time, manually start maintenance on VMs .

View the maintenance state of VMs

You can view the state and scheduled time of upcoming maintenance for your VMs by checking the value of the upcomingMaintenance field in the instance's metadata. If a VM doesn't contain the upcomingMaintenance field, then no host maintenance event is scheduled for the VM. For more information about the fields in upcomingMaintenance , see Maintenance status definitions in the Compute Engine documentation.

You can view the maintenance state for multiple VMs simultaneously or for individual VMs. For multiple VMs, use the Google Cloud console or REST API. For individual VMs, select any of the following options:

 gcloud compute instances describe VM_NAME 
\
    --flatten=resourceStatus.upcomingMaintenance \
    --zone= ZONE 
 

 {
  "items": [
    {
      "name": "vm-01",
      "machineType": "https://www.googleapis.com/compute/v1/projects/example-project/zones/europe-west1-b/machineTypes/a3-ultragpu-8g",
      "resourceStatus": { "upcomingMaintenance": {
          "canReschedule": true,
          "latestWindowStartTime": "2024-12-01T19:00:00Z",
          "machineType": "a3-ultragpu-8g",
          "maintenanceStatus": "PENDING",
          "type": "SCHEDULED",
          "windowEndTime": "2024-12-01T22:00:00Z",
          "windowStartTime": "2024-12-01T19:00:00Z"
        }
      }
    },
    ...
  ]
} 

Manually start maintenance on VMs

You can manually start maintenance for your VMs instead of waiting for the scheduled time.

Depending on the maintenance state of a VM, the following occurs:

You can manually start maintenance for multiple VMs simultaneously or for individual VMs. For multiple VMs, use the Google Cloud console or, for VMs located in the same zone, the gcloud CLI. For individual VMs, select any of the following options:

 gcloud compute instances perform-maintenance VM_NAMES 
\
    --zone= ZONE 
 

 POST https://compute.googleapis.com/compute/v1/projects/ PROJECT_ID 
/zones/ ZONE 
/instances/ VM_NAME 
/performMaintenance 

What's next

• Query the metadata server for maintenance event notices