Use maintenance mode on a database cluster in Kubernetes

If your database runs in a Kubernetes cluster, then you can put the database in maintenance mode by disabling liveness probes and startup probes , which run on your containerized application, to ensure that the database pod is in a healthy state. Maintenance mode helps you place a pod in a running state when you need to make a maintenance update or repair the pod.

Enable maintenance mode

1. Verify that probes exist on a database pod:

   1. Liveness probe:

       kubectl  
      get  
      po  
      -n  
       NAMESPACE 
        
       DATABASE_POD_NAME 
        
      -o  
       jsonpath 
       = 
       '{.spec.containers[?(@.name=="database")].livenessProbe}' 
       
      

      Replace the following:

      • NAMESPACE : the name of a namespace used to run your database—for example, db .
      • DATABASE_POD_NAME : the name of the database pod—for example, al-4017-dbcluster-sample-0 .

      The following shows sample output with a liveness probe:

       {"failureThreshold":3,"httpGet":{"path":"/healthz?","port":8090,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":3} 
      

   2. Startup probe:

       kubectl  
      get  
      po  
      -n  
       NAMESPACE 
        
       DATABASE_POD_NAME 
        
      -o  
       jsonpath 
       = 
       '{.spec.containers[?(@.name=="database")].startupProbe}' 
       
      

      The following shows sample output with a startup probe:

       {"exec":{"command":["/usr/lib/postgresql/15/bin/psql","-hlocalhost","-Ualloydbadmin","-c","SELECT 1"]},"failureThreshold":180,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1} 
      

2. Set mode to maintenance in the spec section of your DBCluster manifest:

     apiVersion 
    : 
     
    alloydbomni.dbadmin.goog/v1 
    kind 
    : 
     
    DBCluster 
    metadata 
    : 
     
    name 
    : 
     
     DB_CLUSTER_NAME 
    
     
    namespace 
    : 
     
     NAMESPACE 
    
    spec 
    : 
     
    allowExternalIncomingTraffic 
    : 
     
    false 
     
    isDeleted 
    : 
     
    false 
     
     mode 
    : 
     
    maintenance 
     
    primarySpec 
    : 
     
    ... 
    
   

   Replace DB_CLUSTER_NAME with the name of this database cluster—for example, my-db-cluster .

3. Apply the manifest:

    kubectl  
   apply  
   -f  
    DB_CLUSTER_YAML 
    
   

   Replace DB_CLUSTER_YAML with the name of this database cluster manifest file—for example, alloydb-omni-db-cluster.yaml .

4. To verify that maintenance mode is enabled, run the following commands:

   • Liveness probe:

      kubectl  
     get  
     po  
     -n  
      NAMESPACE 
       
      DATABASE_POD_NAME 
       
     -o  
      jsonpath 
      = 
      '{.spec.containers[?(@.name=="database")].livenessProbe}' 
      
     

   • Startup probe:

      kubectl  
     get  
     po  
     -n  
      NAMESPACE 
       
      DATABASE_POD_NAME 
       
     -o  
      jsonpath 
      = 
      '{.spec.containers[?(@.name=="database")].startupProbe}' 
      
     

   The output doesn't contain the Liveness and Startup fields in the spec section of your POD_NAME pod.

Disable maintenance mode

To return to the standard database cluster behavior with running liveliness and startup probes, use the following command to remove maintenance from the spec section of your DBCluster manifest:

 kubectl  
patch  
dbclusters.alloydbomni.dbadmin.goog  
 DB_CLUSTER_NAME 
  
-p  
 '{"spec":{"mode":""}}' 
  
--type = 
merge  
-n  
 DB_CLUSTER_NAMESPACE 
  
--type = 
merge 

Sample outputs:

• With a liveness probe:

   {"failureThreshold":3,"httpGet":{"path":"/healthz?","port":8090,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":3} 
  

• With a startup probe:

   {"exec":{"command":["/usr/lib/postgresql/15/bin/psql","-hlocalhost","-Ualloydbadmin","-c","SELECT 1"]},"failureThreshold":180,  "periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}