Back up and restore clusters with bmctl

This page describes how to use bmctl to back up and restore clusters created with Google Distributed Cloud. These instructions apply to all cluster types supported by Google Distributed Cloud.

The bmctl backup and restore process does not include persistent volumes. Any volumes created by the local volume provisioner (LVP) are left unaltered.

Back up a cluster

The bmctl backup cluster command adds the cluster information from the etcd store and the PKI certificates for the specified cluster the cluster to a tar file. The etcd store is the Kubernetes backing store for all cluster data and contains all the Kubernetes objects and custom objects required to manage cluster state. The PKI certificates are used for authentication over TLS. This data is backed up from the cluster's control plane or from one of the control planes for a high-availability (HA) deployment.

The backup tar file contains sensitive credentials, including your service account keys and the SSH key. Store backup files in a secure location. To prevent unintended file exposure, the Google Distributed Cloud backup process uses in-memory files only.

Back up your clusters regularly to ensure your snapshot data is relatively current. Adjust the rate of backups to reflect the frequency of significant changes to your clusters.

The bmctl version you use to back up a cluster must match the version of the managing cluster.

To back up a cluster:

1. Ensure your cluster is operating properly, with working credentials and SSH connectivity to all nodes.

   The intent of the backup process is to capture your cluster in a known good state, so that you can restore operation if a catastrophic failure occurs.

   Use the following command to check your cluster:

    bmctl  
   check  
   cluster  
   -c  
    CLUSTER_NAME 
    
   

   Replace CLUSTER_NAME with the name of the cluster you plan to back up.

2. Run the following command to ensure the target cluster is not in a reconciliation state:

    kubectl  
   describe  
   cluster  
    CLUSTER_NAME 
     
   -n  
    CLUSTER_NAMESPACE 
    
   

   Replace the following:

   • CLUSTER_NAME : the name of the cluster to back up.
   • CLUSTER_NAMESPACE : the namespace for the cluster. By default, the cluster namespaces for Google Distributed Cloud are the name of the cluster prefaced with cluster- . For example, if you name your cluster test , the namespace has a name like cluster-test .

3. Check the command output for status.conditions of type "Reconciling".

   A status of "False" for these status.conditions means the cluster is stable and ready to be backed up.

4. Run the following command to back up the cluster:

    bmctl  
   backup  
   cluster  
   -c  
    CLUSTER_NAME 
     
   --kubeconfig  
    ADMIN_KUBECONFIG 
    
   

   Replace the following:

   • CLUSTER_NAME : the name of the cluster to back up.
   • ADMIN_KUBECONFIG : the path to the admin cluster kubeconfig file.

   By default, the backup tar file saved to the workspace directory ( bmctl-workspace , by default) on your admin workstation. The tar file is named CLUSTER_NAME _backup_ TIMESTAMP .tar.gz , where CLUSTER_NAME is the name of the cluster being backed up and TIMESTAMP is the date and time the backup was made. For example, if the cluster name is testuser , the backup file has a name like testuser_backup_2006-01-02T150405Z0700.tar.gz .

   To specify a different name and location for your backup file, use the --backup-file flag.

The backup file expires after a year and the cluster restore process doesn't work with expired backup files.

Restore a cluster

Restoring a cluster from a backup is a last resort and should be used when a cluster has failed catastrophically and cannot be returned to service any other way. For example, the etcd data is corrupted or the etcd Pod is in a crash loop.

The backup tar file contains sensitive credentials, including your service account keys and the SSH key. To prevent unintended file exposure, the Google Distributed Cloud restore process uses in-memory files only.

The bmctl version you use to restore a cluster must match the version of the managing cluster.

To restore a cluster:

1. Ensure all node machines that were available for the cluster at the time of the backup are operating properly and reachable.

2. Ensure that SSH connectivity between nodes works with the SSH keys that were used at the time of the backup.

   These SSH keys are reinstated as part of the restore process.

3. Ensure that the service account keys that were used at the time of the backup are still active.

   These service account keys are reinstated for the restored cluster.

4. To restore a standalone cluster or a self-managed admin or hybrid cluster, run the following command:

    bmctl  
   restore  
   cluster  
   -c  
    CLUSTER_NAME 
     
   --backup-file  
    BACKUP_FILE 
    
   

   Replace the following:

   • CLUSTER_NAME : the name of the cluster you are restoring.
   • BACKUP_FILE : the path and name of the backup file you are using.

5. To restore a user cluster or admin or hybrid clusters that aren't self-managed, run the following command:

    bmctl  
   restore  
   cluster  
   -c  
    CLUSTER_NAME 
     
   --backup-file  
    BACKUP_FILE 
     
    \ 
     
   --kubeconfig  
    ADMIN_KUBECONFIG 
    
   

   Replace the following:

   • CLUSTER_NAME : the name of the cluster you are restoring.
   • BACKUP_FILE : the path and name of the backup file you are using.
   • ADMIN_KUBECONFIG : the path to the admin cluster kubeconfig file.

At the end of the restore process, a new kubeconfig file is generated for the restored cluster.