Put nodes into maintenance mode

When you need to repair or maintain nodes, you should first put the nodes into maintenance mode. Putting nodes into maintenance mode safely drains their pods/workloads and excludes the nodes from pod scheduling. In maintenance mode, you can work on your nodes without a risk of disrupting pod traffic.

Google Distributed Cloud provides a way to place nodes into maintenance mode. This approach lets other cluster components correctly know that the node is in maintenance mode. When you place a node in maintenance mode, no additional pods can be scheduled on the node, and existing pods are stopped.

Instead of using maintenance mode, you can manually use Kubernetes commands such as kubectl cordon and kubectl drain on a specific node. If you run Google Distributed Cloud version 1.12.0 ( anthosBareMetalVersion: 1.12.0 ) or lower, see the known issue on Nodes uncordoned if you don't use the maintenance mode procedure .

Choose the nodes you want to put into maintenance mode by specifying IP ranges for the selected nodes under maintenanceBlocks in your cluster configuration file. The nodes you choose must be in a ready state, and functioning in the cluster.

To put nodes into maintenance mode:

1. Edit the cluster configuration file to select the nodes you want to put into maintenance mode.

   You can edit the configuration file with an editor of your choice, or you can edit the cluster custom resource directly by running the following command:

    kubectl  
   -n  
    CLUSTER_NAMESPACE 
     
   edit  
   cluster  
    CLUSTER_NAME 
    
   

   Replace the following:

   • CLUSTER_NAMESPACE : the namespace of the cluster.
   • CLUSTER_NAME : the name of the cluster.

2. Add the maintenanceBlocks section to the cluster configuration file to specify either a single IP address, or an address range, for nodes you want to put into maintenance mode.

   The following sample shows how to select multiple nodes by specifying a range of IP addresses:

     metadata 
    : 
     
    name 
    : 
     
    my-cluster 
     
    namespace 
    : 
     
    cluster-my-cluster 
    spec 
    : 
     
    maintenanceBlocks 
    : 
     
    cidrBlocks 
    : 
     
    - 
     
    172.16.128.1-172.16.128.64 
    
   

3. Save and apply the updated cluster configuration.

   Google Distributed Cloud starts putting the nodes into maintenance mode.

4. Run the following command to get the status of the nodes in your cluster:

    kubectl  
   get  
   nodes  
   -n  
    CLUSTER_NAME 
    
   

   The response is something like the following:

    NAME                              STATUS                     ROLES    AGE     VERSION
   user-anthos-baremetal-01          Ready                      master   2d22h   v1.17.8-gke.16
   user-anthos-baremetal-04          Ready                      <none>   2d22h   v1.17.8-gke.16
   user-anthos-baremetal-05          Ready,SchedulingDisabled   <none>   2d22h   v1.17.8-gke.16
   user-anthos-baremetal-06          Ready                      <none>   2d22h   v1.17.8-gke.16 
   

   A status of SchedulingDisabled indicates that a node is in maintenance mode.

5. Run the following command to get the number of nodes in maintenance mode:

    kubectl  
   get  
   nodepools 
   

   The response should look something like the following output:

    NAME   READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWN
   np1    3       0             0         1                  0 
   

   This UNDERMAINTENANCE column in this sample shows that one node is in maintenance mode.

   Google Distributed Cloud also adds the following taints to nodes when they are put into maintenance mode:

   • baremetal.cluster.gke.io/maintenance:NoExecute
   • baremetal.cluster.gke.io/maintenance:NoSchedule

To remove nodes from maintenance mode:

1. Edit the cluster configuration file to clear the nodes you want to remove from maintenance mode.

   You can edit the configuration file with an editor of your choice, or you can edit the cluster custom resource directly by running the following command:

    kubectl  
   -n  
    CLUSTER_NAMESPACE 
     
   edit  
   cluster  
    CLUSTER_NAME 
    
   

   Replace the following:

   • CLUSTER_NAMESPACE : the namespace of the cluster.
   • CLUSTER_NAME : the name of the cluster.

2. Either edit the IP addresses to remove specific nodes from maintenance mode or remove the maintenanceBlocks section remove all does from maintenance mode.

3. Save and apply the updated cluster configuration.

4. Use kubectl commands to check the status of your nodes.