Report faulty host

If you notice an issue on an A4X Max, A4X, A4, or A3 Ultra instance that you can't resolve otherwise—such as slower performance within a cluster or consistently high GPU temperatures—then you can report its host as faulty. When you report a host as faulty, Compute Engine automatically repairs the compute instance by running host maintenance. For A4 and A3 Ultra instances, Compute Engine attempts to migrate the instance to a different host when maintenance starts, if you have unused reserved capacity or capacity is available in the instance's zone. Reporting a host as faulty helps you minimize downtime for your workload.

This document explains how to report and repair faulty host instances that are part of a Slurm cluster or other instance-based clusters. To report faulty hosts in a Google Kubernetes Engine (GKE) cluster, see Report faulty hosts through GKE .

Limitations

When you report a faulty host, the following limitations apply:

• You can only report a faulty host if the compute instance that runs on the host meets all of the following conditions:

  • The compute instance is running.

  • The compute instance uses an A4X Max, A4X, A4, or A3 Ultra machine type.

  • The compute instance uses the reservation-bound provisioning model .

• Google Cloud makes best-effort attempts to fulfill all your report faulty host requests. However, due to capacity constraints or rate limits, a request might not always be fulfilled.

Before you begin

Select the tab for how you plan to use the samples on this page:

Required roles

To get the permissions that you need to report a faulty host, ask your administrator to grant you the following IAM roles:

• Compute Instance Admin (v1) ( roles/compute.instanceAdmin.v1 ) on the compute instance or the project
• To view the state of a faulty host report operation by using Cloud Logging: Logs Viewer ( roles/logging.viewer ) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations .

These predefined roles contain the permissions required to report a faulty host. To see the exact permissions that are required, expand the Required permissionssection:

You might also be able to get these permissions with custom roles or other predefined roles .

Understand the faulty host report process

Report a faulty host

To report a faulty host, complete the following steps:

1. Review the host on which your compute instance runs.

   For instructions, see View topology of a compute instance .

2. Optional: Back up Local SSD data. When the instance stops, Compute Engine automatically discards the data of any Local SSD disks that are attached to the instance. You can't recover Local SSD data after Compute Engine discards it.

   For instructions on how to preserve Local SSD data, see Local SSD data backup .

3. Report the faulty host. To report a faulty host, select one of the following options. The host repair operation starts immediately, within a minute after the report faulty host operation completes. If the instance becomes unresponsive after you start the faulty host report operation, then, after you wait for at least 15 minutes, we recommend that you restart the instance.

   gcloud

   To report a faulty host, use the following gcloud compute instances report-host-as-faulty command :

    gcloud compute instances report-host-as-faulty INSTANCE_NAME 
   \
       --async \
       --disruption-schedule=IMMEDIATE \
       --fault-reasons=behavior= FAULT_REASON 
   ,description= DESCRIPTION 
   \
       --zone= ZONE 
    
   

   Replace the following:

   • INSTANCE_NAME : the name of the compute instance.

   • FAULT_REASON : a list of host issues that your instance encountered, separated by commas—for example, ISSUE_1,ISSUE_2 . You can specify the following values:

     • PERFORMANCE : that GPUs that are attached to the instance have performance issues compared to other GPUs in the cluster, you see no XID errors in the logs, and the Compute Engine detects no other usual failure patterns such as silent data corruption.

     • SILENT_DATA_CORRUPTION : you see data corruption in your instance, but the instance keeps running. Silent data corruption can be due to issues like vCPUs defects, software bugs, or kernel issues.

     • UNRECOVERABLE_GPU_ERROR : you identified an unrecoverable GPU error with an XID.

     • BEHAVIOR_UNSPECIFIED : you aren't sure about what the issue to your instance is.

   • DESCRIPTION : a description of the issue that is affecting your instance, such as XID information or suspected performance problems.

   • ZONE : the zone where the instance exists.

   REST

   To report a faulty host, make the following POST request to the instances.reportHostAsFaulty method .

   When you report a faulty host, you can specify multiple fault reasons at once. For example, to specify two fault reasons, make a request as follows:

    POST https://compute.googleapis.com/compute/v1/projects/ PROJECT_ID 
   /zones/ ZONE 
   /instances/ INSTANCE_NAME 
   /reportHostAsFaulty
   
   {
     "disruptionSchedule": "IMMEDIATE",
     "faultReasons": [
       {
         "behavior": " FAULT_REASON_1 
   ",
         "description": " DESCRIPTION_1 
   "
       },
       {
         "behavior": " FAULT_REASON_2 
   ",
         "description": " DESCRIPTION_2 
   "
       }
     ]
   } 
   

   Replace the following:

   • PROJECT_ID : the ID of the project where the instance exists.

   • ZONE : the zone where the instance exists.

   • INSTANCE_NAME : the name of the compute instance.

   • FAULT_REASON_1 and FAULT_REASON_2 : each host issue that your instance encountered. You can specify the following values:

     • PERFORMANCE : that GPUs that are attached to the instance have performance issues compared to other GPUs in the cluster, you see no XID errors in the logs, and the Compute Engine detects no other usual failure patterns such as silent data corruption.

     • SILENT_DATA_CORRUPTION : you see data corruption in your instance, but the instance keeps running. Silent data corruption can be due to issues like vCPUs defects, software bugs, or kernel issues.

     • UNRECOVERABLE_GPU_ERROR : you identified an unrecoverable GPU error with an XID.

     • BEHAVIOR_UNSPECIFIED : you aren't sure about what the issue to your instance is.

   • DESCRIPTION_1 and DESCRIPTION_2 : a description for each host issue that you specified, such as XID information or suspected performance problems.

Review report faulty host operations

After you report a faulty host, Compute Engine starts a series of operations to mark the host as faulty and prepares the host for repair. Specifically, during a report faulty host operation, the following process happens:

1. Mark the host as faulty. Compute Engine creates the report faulty host operation. The report faulty host operation then creates a sequence of sub-operations. These sub-operations mark the underlying host as faulty.

2. Prepare the host for repairs. After all sub-operations complete, the report faulty host operation starts. Compute Engine stops the instance and starts the repair faulty host operation. Based on the reservation operational mode that is specified in the reservation that the instance uses, and if healthy hosts are available, Compute Engine either keeps the instance stopped or attempts to automatically migrate and restart the instance.

3. Report completion and repair the host. Compute Engine completes the report faulty host operation, and the host repair operation runs.

To track the status of the report faulty host ( compute.instances.reportHostAsFaulty ) operations in your project, select one of the following options. For more information about other operations that you can use to track repairs, migration, and automatic restart, see Maintenance and restart behaviors and Monitor and plan for a host maintenance event in the Compute Engine documentation.

 GET https://compute.googleapis.com/compute/v1/projects/ PROJECT_ID 
/zones/ ZONE 
/operations&filter=items.operationType:reportHostAsFaulty 

What's next?

• If you encounter issues when reporting a faulty host, then see Troubleshoot faulty host API .