Google Distributed Cloud

Binary Authorization webook blocks CNI plugin to start causing one of nodepool failed to come up

Under rare race conditions, an incorrect installation sequence of the Binary Authorization webhook and the gke-connect pod may cause user cluster creation to stall due to a node failing to reach a ready state. In affected scenarios, user cluster creation may stall due to a node failing to reach a ready state. If this occurs, the following message will be displayed:

Node pool is not ready: ready condition is not true: CreateOrUpdateNodePool: 2/3 replicas are ready

Workaround:

1. Remove the Binary Authorization configuration from your config file. For setup instructions, please refer to the Binary Authorization day 2 installation guide for GKE on VMware .
2. To unblock an unhealthy node during the current cluster creation process, temporarily remove the Binary Authorization webhook configuration in user cluster using the following command.

     
   kubectl  
   --kubeconfig  
    USER_KUBECONFIG 
     
   delete  
   ValidatingWebhookConfiguration  
   binauthz-validating-webhook-configuration  
   

   Once the bootstrap process is complete, you can re-add the following webhook configuration.

    apiVersion 
    : 
     
    admissionregistration.k8s.io/v1 
    kind 
    : 
     
    ValidatingWebhookConfiguration 
    metadata 
    : 
     
    name 
    : 
     
    binauthz-validating-webhook-configuration 
    webhooks 
    : 
    - 
     
    name 
    : 
     
    "binaryauthorization.googleapis.com" 
     
    namespaceSelector 
    : 
     
    matchExpressions 
    : 
     
    - 
     
    key 
    : 
     
    control-plane 
     
    operator 
    : 
     
    DoesNotExist 
     
    objectSelector 
    : 
     
    matchExpressions 
    : 
     
    - 
     
    key 
    : 
     
    "image-policy.k8s.io/break-glass" 
     
    operator 
    : 
     
    NotIn 
     
    values 
    : 
     
    [ 
    "true" 
    ] 
     
    rules 
    : 
     
    - 
     
    apiGroups 
    : 
     
    - 
     
    "" 
     
    apiVersions 
    : 
     
    - 
     
    v1 
     
    operations 
    : 
     
    - 
     
    CREATE 
     
    - 
     
    UPDATE 
     
    resources 
    : 
     
    - 
     
    pods 
     
    - 
     
    pods/ephemeralcontainers 
     
    admissionReviewVersions 
    : 
     
    - 
     
    "v1beta1" 
     
    clientConfig 
    : 
     
    service 
    : 
     
    name 
    : 
     
    binauthz 
     
    namespace 
    : 
     
    binauthz-system 
     
    path 
    : 
     
    /binauthz 
     
    # CA Bundle will be updated by the cert rotator. 
     
    caBundle 
    : 
     
    Cg== 
     
    timeoutSeconds 
    : 
     
    10 
     
    # Fail Open 
     
    failurePolicy 
    : 
     
    "Ignore" 
     
    sideEffects 
    : 
     
    None 
     
   

CPV2 user cluster upgrade stuck due to mirrored machine with deletionTimestamp

During a user cluster upgrade, the upgrade operation might get stuck if the mirrored machine object in the user cluster contains a deletionTimestamp . The following error message is displayed if the upgrade is stuck:

machine is still in the process of being drained and subsequently removed

This issue can occur if you previously attempted to repair the user control plane node by running gkectl delete machine against the mirrored machine in the user cluster.

Workaround:

1. Get the mirrored machine object and save it to a local file for backup purposes.
2. Run the following command to delete the finalizer from the mirrored machine and wait for it to be deleted from the user cluster.

     
   kubectl  
   --kubeconfig  
    ADMIN_KUBECONFIG 
     
   patch  
   machine/ MACHINE_OBJECT_NAME 
     
   -n  
    USER_CLUSTER_NAME 
   -gke-onprem-mgmt  
   -p  
    '{"metadata":{"finalizers":[]}}' 
     
   --type = 
   merge

3. Follow the steps in Controlplane V2 user cluster control plane node to trigger node repair on the control plane nodes, so that the correct source machine spec will be re-synced into the user cluster.
4. Rerun the gkectl upgrade cluster to resume the upgrade

Cluster creation failure due to control plane VIP in different subnet

For HA admin cluster or ControlPlane V2 user cluster, the control plane VIP needs to be in the same subnet as other cluster nodes. Otherwise, cluster creation fails because kubelet can't communicate with the API server using the control plane VIP.

Workaround:

Before cluster creation, ensure that the control plane VIP is configured in the same subnet as the other cluster nodes.

Cluster Creation/Upgrade Failure due to non-FQDN vCenter Username

Cluster creation/upgrade fails with an error in vsphere CSI pods indicating that the vCenter username is invalid. This occurs because the username used is not a fully qualified domain name. Error message in the vsphere-csi-controller pod as below:

GetCnsconfig failed with err: username is invalid, make sure it is a fully qualified domain username

This issue only occurs in version 1.29 and later, as a validation was added to the vSphere CSI driver to enforce the use of fully qualified domain usernames.

Workaround:

Use a fully qualified domain name for the vCenter username in the credentials configuration file. For example, instead of using "username1", use "username1@example.com".

Admin cluster upgrade fails for clusters created on versions 1.10 or earlier

When upgrading an admin cluster from 1.16 to 1.28, the bootstrap of the new admin master machine might fail to generate the control-plane certificate. The issue is caused by changes in how certificates are generated for the Kubernetes API server in version 1.28 and later. The issue reproduces for clusters created on versions 1.10 and earlier that have been upgraded all the way to 1.16 and the leaf certificate was not rotated before the upgrade.

To determine if the admin cluster upgrade failure is caused by this issue, do the following steps:

1. Connect to the failed admin master machine by using SSH.
2. Open /var/log/startup.log and search for an error like the following:

Error adding extensions from section apiserver_ext
801B3213B57F0000:error:1100007B:X509 V3 routines:v2i_AUTHORITY_KEYID:unable to get issuer keyid:../crypto/x509/v3_akid.c:177:
801B3213B57F0000:error:11000080:X509 V3 routines:X509V3_EXT_nconf_int:error in extension:../crypto/x509/v3_conf.c:48:section=apiserver_ext, name=authorityKeyIdentifier, value=keyid>

Workaround:

1. Connect to the admin master machine by using SSH. For details, see Using SSH to connect to an admin cluster node .
2. Edit /etc/startup/pki-yaml.sh . Find authorityKeyIdentifier=keyidset and change it to authorityKeyIdentifier=keyid,issuer in the sections for the following extensions: apiserver_ext , client_ext , etcd_server_ext , and kubelet_server_ext . For example:

   [ apiserver_ext ]
         keyUsage = critical, digitalSignature, keyEncipherment
         extendedKeyUsage=serverAuth
         basicConstraints = critical,CA:false
         authorityKeyIdentifier = keyid,issuer
         subjectAltName = @apiserver_alt_names

3. Save the changes to /etc/startup/pki-yaml.sh .
4. Run /opt/bin/master.sh to generate the certificate and complete the machine startup.
5. Run the gkectl upgrade admin again to upgrade the admin cluster.
6. After the upgrade completes, rotate the leaf certificate for both admin and user clusters, as described in Start the rotation .
7. After the certificate rotation completes, make the same edits to /etc/startup/pki-yaml.sh as you did previously, and run /opt/bin/master.sh .

Incorrect warning message for clusters with Dataplane V2 enabled

The following incorrect warning message is output when you run gkectl to create, update, or upgrade a cluster that already has Dataplane V2 enabled:

WARNING: Your user cluster is currently running our original architecture with 
[DataPlaneV1(calico)]. To enable new and advanced features we strongly recommend
to update it to the newer architecture with [DataPlaneV2] once our migration 
tool is available.

There's a bug in gkectl which causes it to always show this warning as long as the dataplaneV2.forwardMode is not being used, even if you already have set enableDataplaneV2: true in your cluster configuration file.

Workaround:

You can safely ignore this warning.

HA admin cluster installation preflight check reports wrong number of required static IPs

When you create an HA admin cluster, the preflight check displays the following incorrect error message:

- Validation Category: Network Configuration
    - [FAILURE] CIDR, VIP and static IP (availability and overlapping): needed
    at least X+1 IP addresses for admin cluster with X nodes

The requirement is incorrect for 1.28 and higher HA admin clusters because they no longer have add-on nodes. Additionally, because the 3 admin cluster control plane node IPs are specified in the network.controlPlaneIPBlock section in the admin cluster configuration file, the IPs in IP block file are only needed for kubeception user cluster control plane nodes.

Workaround:

To skip the incorrect preflight check in a non-fixed release, add --skip-validation-net-config to the gkectl command.

Connect Agent loses connection to Google Cloud after non-HA to HA admin cluster migration

If you migrated from a non-HA admin cluster to an HA admin cluster , the Connect Agent in the admin cluster loses the connection to gkeconnect.googleapis.com with the error "Failed to verify JWT signature". This is because during the migration, the KSA signing key is changed, thus a re-registration is needed to refresh the OIDC JWKs.

Workaround:

To reconnect the admin cluster to Google Cloud, do the following steps to trigger a re-registration:

First get the gke-connect deployment name:

kubectl  
--kubeconfig  
 KUBECONFIG 
  
get  
deployment  
-n  
gke-connect

Delete the gke-connect deployment:

kubectl  
--kubeconfig  
 KUBECONFIG 
  
delete  
deployment  
 GKE_CONNECT_DEPLOYMENT 
  
-n  
gke-connect

Trigger a force reconcile for the onprem-admin-cluster-controller by adding a "force-reconcile" annotation to your onpremadmincluster CR:

kubectl  
--kubeconfig  
 KUBECONFIG 
  
patch  
onpremadmincluster  
 ADMIN_CLUSTER_NAME 
  
-n  
kube-system  
--type  
merge  
-p  
 ' 
 metadata: 
 annotations: 
 onprem.cluster.gke.io/force-reconcile: "true" 
 ' 

The idea is that the onprem-admin-cluster-controller will always redeploy the gke-connect deployment and re-register the cluster if it finds no existing gke-connect deployment available.

After the workaround (it may take a few minutes for the controller to finish the reconcile), you can verify that the "Failed to verify JWT signature" 400 error is gone from the gke-connect-agent logs:

kubectl  
--kubeconfig  
 KUBECONFIG 
  
logs  
 GKE_CONNECT_POD_NAME 
  
-n  
gke-connect

Docker bridge IP uses 172.17.0.1/16 for COS cluster control plane nodes

Google Distributed Cloud specifies a dedicated subnet, --bip=169.254.123.1/24 , for the Docker bridge IP in the Docker configuration to prevent reserving the default 172.17.0.1/16 subnet. However, in version 1.28.0-1.28.500 and 1.29.0, the Docker service wasn't restarted after Google Distributed Cloud customized the Docker configuration because of a regression in the COS OS image. As a result, Docker picks the default 172.17.0.1/16 as its bridge IP address subnet. This might cause an IP address conflict if you already have a workload running within that IP address range.

Workaround:

To work around this issue, you must restart the docker service:

sudo  
systemctl  
restart  
docker

Verify that Docker picks the correct bridge IP address:

ip  
a  
 | 
  
grep  
docker0

This solution does not persist across VM re-creations. You must reapply this workaround whenever VMs are re-created.

Using multiple network interfaces with standard CNI does not work

The standard CNI binaries bridge, ipvlan, vlan, macvlan, dhcp, tuning, host-local, ptp, portmap are not included in the OS images in the affected versions. These CNI binaries are not used by data plane v2, but can be used for additional network interfaces in the multiple network interface feature.

Multiple network interface with these CNI plugins won't work.

Workaround:

Upgrade to the version with the fix if you are using this feature.

Netapp trident dependencies interfere with vSphere CSI driver

Installing multipathd on cluster nodes interferes with the vSphere CSI driver resulting in user workloads being unable to start.

Workaround:

• Disable multipathd

The admin cluster webhook might block updates when you add required configurations

If some required configurations are empty in the admin cluster because validations were skipped, adding them might be blocked by the admin cluster webhook. For example, if the gkeConnect field wasn't set in an existing admin cluster, adding it with the gkectl update admin command might get the following error message:

admission webhook "vonpremadmincluster.onprem.cluster.gke.io" denied the request: connect: Required value: GKE connect is required for user clusters

Workaround:

• For 1.15 admin clusters, run gkectl update admin command with --disable-admin-cluster-webhook flag. For example:

    
  gkectl  
  update  
  admin  
  --config  
   ADMIN_CLUSTER_CONFIG_FILE 
    
  --kubeconfig  
   ADMIN_CLUSTER_KUBECONFIG 
    
  --disable-admin-cluster-webhook  
  

• For 1.16 admin clusters, run gkectl update admin commands with --force flag. For example:

    
  gkectl  
  update  
  admin  
  --config  
   ADMIN_CLUSTER_CONFIG_FILE 
    
  --kubeconfig  
   ADMIN_CLUSTER_KUBECONFIG 
    
  --force  
  

controlPlaneNodePort field defaults to 30968 when manualLB spec is empty

If you will be using a manual load balancer ( loadBalancer.kind is set to "ManualLB" ), you shouldn't need to configre the loadBalancer.manualLB section in the configuration file for a high availability (HA) admin cluster in versions 1.16 and higher. But when this section is empty, Google Distributed Cloud assigns default values to all NodePorts including manualLB.controlPlaneNodePort , which causes cluster creation to fail with the following error message:

-  
Validation  
Category:  
Manual  
LB  
-  
 [ 
FAILURE ] 
  
NodePort  
configuration:  
manualLB.controlPlaneNodePort  
must  
not  
be  
 set 
  
when  
using  
HA  
admin  
cluster,  
got:  
 30968 

Workaround:

Specify manualLB.controlPlaneNodePort: 0 in you admin cluster configuration for the HA admin cluster:

 loadBalancer 
 : 
  
 ... 
  
 kind 
 : 
  
 ManualLB 
  
 manualLB 
 : 
  
 controlPlaneNodePort 
 : 
  
 0 
  
 ... 

nfs-common is missing from Ubuntu OS image

nfs-common is missing from the Ubuntu OS image which may cause issues for customers using NFS-dependent drivers such as NetApp.

Warning  
FailedMount  
63s  
 ( 
x8  
over  
2m28s ) 
  
kubelet  
MountVolume.SetUp  
failed  
 for 
  
volume  
 "pvc-xxx-yyy-zzz" 
  
:  
rpc  
error:  
 code 
  
 = 
  
Internal  
 desc 
  
 = 
  
error  
mounting  
NFS  
volume  
 10 
.0.0.2:/trident_pvc_xxx-yyy-zzz  
on  
mountpoint  
/var/lib/kubelet/pods/aaa-bbb-ccc/volumes/kubernetes.io~csi/pvc-xxx-yyy-zzz/mount:  
 exit 
  
status  
 32 
 ". 
  

Workaround:

Make sure your nodes can download packages from Canonical.

Next, apply the following DaemonSet to your cluster to install nfs-common :

 apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 DaemonSet 
 metadata 
 : 
  
 name 
 : 
  
 install-nfs-common 
  
 labels 
 : 
  
 name 
 : 
  
 install-nfs-common 
  
 namespace 
 : 
  
 kube-system 
 spec 
 : 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 name 
 : 
  
 install-nfs-common 
  
 minReadySeconds 
 : 
  
 0 
  
 updateStrategy 
 : 
  
 type 
 : 
  
 RollingUpdate 
  
 rollingUpdate 
 : 
  
 maxUnavailable 
 : 
  
 100% 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 name 
 : 
  
 install-nfs-common 
  
 spec 
 : 
  
 hostPID 
 : 
  
 true 
  
 hostIPC 
 : 
  
 true 
  
 hostNetwork 
 : 
  
 true 
  
 initContainers 
 : 
  
 - 
  
 name 
 : 
  
 install-nfs-common 
  
 image 
 : 
  
 ubuntu 
  
 imagePullPolicy 
 : 
  
 IfNotPresent 
  
 securityContext 
 : 
  
 privileged 
 : 
  
 true 
  
 command 
 : 
  
 - 
  
 chroot 
  
 - 
  
 /host 
  
 - 
  
 bash 
  
 - 
  
 -c 
  
 args 
 : 
  
 - 
  
 | 
  
 apt install -y nfs-common 
  
 volumeMounts 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 mountPath 
 : 
  
 /host 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 pause 
  
 image 
 : 
  
 gcr.io/gke-on-prem-release/pause-amd64:3.1-gke.5 
  
 imagePullPolicy 
 : 
  
 IfNotPresent 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 hostPath 
 : 
  
 path 
 : 
  
 / 
  

Storage policy field is missing in the admin cluster configuration template

SPBM in admin clusters is supported in 1.28.0 and later versions. But the field vCenter.storagePolicyName is missing in the configuration file template.

Workaround:

Add the `vCenter.storagePolicyName` field in you admin cluster configuration file if you want to configure the storage policy for the admin cluster. Please refer to the instructions .

Kubernetes Metadata API does not support VPC-SC

The recently added API kubernetesmetadata.googleapis.com does not support VPC-SC. This will cause metadata collecting agent to fail to reach this API under VPC-SC. Subsequently, metric metadata labels will be missing.

Workaround:

Set in `kube-system` namespace the CR `stackdriver` set `featureGates.disableExperimentalMetadataAgent` field to `true` by running the command

`kubectl -n kube-system patch stackdriver stackdriver -p '{"spec":{"featureGates":{"disableExperimentalMetadataAgent":true}}}'`

then run

`kubectl -n kube-system patch deployment stackdriver-operator -p '{"spec":{"template":{"spec":{"containers":[{"name":"stackdriver-operator","env":[{"name":"ENABLE_LEGACY_METADATA_AGENT","value":"true"}]}]}}}}'`

The clusterapi-controller may crash when the admin cluster and any user cluster with ControlPlane V2 enabled use different vSphere credentials

When an admin cluster and any user cluster with ControlPlane V2 enabled use different vSphere credentials, e.g., after updating vSphere credentials for the admin cluster, the clusterapi-controller may fail to connect to the vCenter after restart. View the log of the clusterapi-controller running in the admin cluster's `kube-system` namespace,

kubectl  
logs  
-f  
-l  
 component 
 = 
clusterapi-controllers  
-c  
vsphere-controller-manager  
 \ 
  
-n  
kube-system  
--kubeconfig  
 KUBECONFIG 

E1214  
 00 
:02:54.095668  
 1 
  
machine_controller.go:165 ] 
  
Error  
checking  
existence  
of  
machine  
instance  
 for 
  
machine  
object  
gke-admin-node-77f48c4f7f-s8wj2:  
Failed  
to  
check  
 if 
  
machine  
gke-admin-node-77f48c4f7f-s8wj2  
exists:  
failed  
to  
find  
datacenter  
 "VSPHERE_DATACENTER" 
:  
datacenter  
 'VSPHERE_DATACENTER' 
  
not  
found

Workaround:

Update vSphere credentials so that the admin cluster and all the user clusters with Controlplane V2 enabled are using the same vSphere credentials.

etcd high number of failed GRPC requests in Prometheus Alert Manager

Prometheus might report alerts similar to the following example:

Alert  
Name:  
cluster:gke-admin-test1:  
Etcd  
cluster  
kube-system/kube-etcd:  
 100 
%  
of  
requests  
 for 
  
Watch  
failed  
on  
etcd  
instance  
etcd-test-xx-n001.

To check if this alert is a false positive that can be ignored, complete the following steps:

1. Check the raw grpc_server_handled_total metric against the grpc_method given in the alert message. In this example, check the grpc_code label for Watch .
   
   You can check this metric using Cloud Monitoring with the following MQL query:

   fetch  
   k8s_container  
    | 
     
   metric  
    'kubernetes.io/anthos/grpc_server_handled_total' 
     
    | 
     
   align  
   rate ( 
   1m ) 
     
    | 
     
   every  
   1m

2. An alert on all codes other than OK can be safely ignored if the code is not one of the following:

   Unknown | 
   FailedPrecondition | 
   ResourceExhausted | 
   Internal | 
   Unavailable | 
   DataLoss | 
   DeadlineExceeded

Workaround:

To configure Prometheus to ignore these false positive alerts, review the following options:

1. Silence the alert from the Alert Manager UI.
2. If silencing the alert isn't an option, review the following steps to suppress the false positives:
   1. Scale down the monitoring operator to 0 replicas so that the modifications can persist.
   2. Modify the prometheus-config configmap, and add grpc_method!="Watch" to the etcdHighNumberOfFailedGRPCRequests alert config as shown in the following example:
      • Original:

        rate ( 
        grpc_server_handled_total { 
         cluster 
         = 
         " CLUSTER_NAME 
        " 
        ,grpc_code! = 
         "OK" 
        ,job = 
        ~ ".*etcd.*" 
         }[ 
        5m ]) 
        

      • Modified:

        rate ( 
        grpc_server_handled_total { 
         cluster 
         = 
         " CLUSTER_NAME 
        " 
        ,grpc_code = 
        ~ "Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded" 
        ,grpc_method! = 
         "Watch" 
        ,job = 
        ~ ".*etcd.*" 
         }[ 
        5m ]) 
        

        Replace the following CLUSTER_NAME with the name of your cluster.
   3. Restart the Prometheus and Alertmanager Statefulset to pick up the new configuration.
3. If the code falls into one of the problematic cases, then check etcd log and kube-apiserver log to debug more.

Egress NAT long lived connections are dropped

Egress NAT connections might be dropped after 5 to 10 minutes of a connection being established if there's no traffic.

As the conntrack only matters in the inbound direction (external connections to the cluster), this issue only happens if the connection doesn't transmit any information for a while and then the destination side transmits something. If the egress NAT'd Pod always instantiates the messaging, then this issue won't be seen.

This issue occurs because the anetd garbage collection inadvertently removes conntrack entries that the daemon thinks are unused. An upstream fix was recently integrated into anetd to correct the behavior.

Workaround:

There is no easy workaround, and we haven't seen issues in version 1.16 from this behavior. If you notice long lived connections dropped due to this issue, workarounds would be to use a workload on the same node as the egress IP address, or to consistently send messages on the TCP connection.

The CSR signer ignores spec.expirationSeconds when signing certificates

If you create a CertificateSigningRequest (CSR) with expirationSeconds set, the expirationSeconds is ignored.

Workaround:

If you're affected by this issue, you can update your user cluster by adding disableNodeIDVerificationCSRSigning: true in the user cluster configuration file and run the gkectl update cluster command to update the cluster with this configuration.

User cluster load balancer validation fails for disable_bundled_ingress

If you try to disable bundled ingress for an existing cluster , the gkectl update cluster command fails with an error similar to the following example:

[FAILURE] Config: ingress IP is required in user cluster spec

This error happens because gkectl checks for a load balancer ingress IP address during preflight checks. Although this check isn't required when disabling bundled ingress, the gkectl preflight check fails when disableBundledIngress is set to true .

Workaround:

Use the --skip-validation-load-balancer parameter when you update the cluster, as shown in the following example:

gkectl  
update  
cluster  
 \ 
  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 
  
--config  
 USER_CLUSTER_CONFIG 
  
 \ 
  
--skip-validation-load-balancer

For more information, see how to disable bundled ingress for an existing cluster .

Admin cluster updates fail after CA rotation

If you rotate admin cluster certificate authority (CA) certificates, subsequent attempts to run the gkectl update admin command fail. The error returned is similar to the following:

failed to get last CARotationStage: configmaps "ca-rotation-stage" not found

Workaround:

If you're affected by this issue, you can update your admin cluster by using the --disable-update-from-checkpoint flag with the gkectl update admin command:

gkectl  
update  
admin  
--config  
 ADMIN_CONFIG_file 
  
 \ 
  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 
  
 \ 
  
--disable-update-from-checkpoint

When you use the --disable-update-from-checkpoint flag, the update command doesn't use the checkpoint file as the source of truth during the cluster update. The checkpoint file is still updated for future use.

CSI Workload preflight check fails due to Pod startup failure

During preflight checks, the CSI Workload validation check installs a Pod in the default namespace. The CSI Workload Pod validates that the vSphere CSI Driver is installed and can do dynamic volume provisioning. If this Pod doesn't start, the CSI Workload validation check fails.

There are a few known issues that can prevent this Pod from starting:

• If the Pod doesn't have resources limits specified, which is the case for some clusters with admissions webhooks installed, the Pod doesn't start.
• If Cloud Service Mesh is installed in the cluster with automatic Istio sidecar injection enabled in the default namespace, the CSI Workload Pod doesn't start.

If the CSI Workload Pod doesn't start, you see a timeout error like the following during preflight validations:

-  
 [ 
FAILURE ] 
  
CSI  
Workload:  
failure  
 in 
  
CSIWorkload  
validation:  
failed  
to  
create  
writer  
Job  
to  
verify  
the  
write  
functionality  
using  
CSI:  
Job  
default/anthos-csi-workload-writer-<run-id>  
replicas  
are  
not  
 in 
  
Succeeded  
phase:  
timed  
out  
waiting  
 for 
  
the  
condition

To see if the failure is caused by lack of Pod resources set, run the following command to check the anthos-csi-workload-writer-<run-id> job status:

kubectl  
describe  
job  
anthos-csi-workload-writer-<run-id>

If the resources limits aren't set properly for the CSI Workload Pod, the job status contains an error message like the following:

CPU  
and  
memory  
resource  
limits  
is  
invalid,  
as  
it  
are  
not  
defined  
 for 
  
container:  
volume-tester

If the CSI Workload Pod doesn't start because of Istio sidecar injection, you can temporarily disable the automatic Istio sidecar injection in the default namespace. Check the labels of the namespace and use the following command to delete the label that starts with istio.io/rev :

kubectl  
label  
namespace  
default  
istio.io/rev-

If the Pod is misconfigured, manually verify that dynamic volume provisioning with the vSphere CSI Driver works:

1. Create a PVC that uses the standard-rwo StorageClass.
2. Create a Pod that uses the PVC.
3. Verify that the Pod can read/write data to the volume.
4. Remove the Pod and the PVC after you've verified proper operation.

If dynamic volume provisioning with the vSphere CSI Driver works, run gkectl diagnose or gkectl upgrade with the --skip-validation-csi-workload flag to skip the CSI Workload check.

User cluster update timeouts when admin cluster version is 1.15

When you are logged on to a user-managed admin workstation , the gkectl update cluster command might timeout and fail to update the user cluster. This happens if the admin cluster version is 1.15 and you run gkectl update admin before you run the gkectl update cluster . When this failure happens, you see the following error when trying to update the cluster:

Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition
Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition

During the update of a 1.15 admin cluster, the validation-controller that triggers the preflight checks is removed from the cluster. If you then try to update the user cluster, the preflight check hangs until the timeout is reached.

Workaround:

1. Run the following command to redeploy the validation-controller :

   gkectl prepare --kubeconfig ADMIN_KUBECONFIG 
   --bundle-path BUNDLE_PATH 
   --upgrade-platform

2. After the prepare completes, run the gkectl update cluster again to update the user cluster

User cluster create timeouts when admin cluster version is 1.15

When you are logged on to a user-managed admin workstation , the gkectl create cluster command might timeout and fail to create the user cluster. This happens if the admin cluster version is 1.15. When this failure happens, you see the following error when trying to create the cluster:

Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition
Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition

Since the validation-controller was added in 1.16 then when using 1.15 admin cluster the validation-controller that is responsible to trigger the preflight checks is missing. Therefore, when trying to create user cluster the preflight checks hang till timeout is reached.

Workaround:

1. Run the following command to deploy the validation-controller :

   gkectl prepare --kubeconfig ADMIN_KUBECONFIG 
   --bundle-path BUNDLE_PATH 
   --upgrade-platform

2. After the prepare completes, run the gkectl create cluster again to create the user cluster

Admin cluster update or upgrade fails if the projects or locations of add-on services don't match each other

When you upgrade an admin cluster from version 1.15.x to 1.16.x, or add a connect , stackdriver , cloudAuditLogging , or gkeOnPremAPI configuration when you update an admin cluster, the operation might be rejected by admin cluster webhook. One of the following error messages might be displayed:

• "projects for connect, stackdriver and cloudAuditLogging must be the same when specified during cluster creation."
• "locations for connect, gkeOnPremAPI, stackdriver and cloudAuditLogging must be in the same region when specified during cluster creation."
• "locations for stackdriver and cloudAuditLogging must be the same when specified during cluster creation."

An admin cluster update or upgrade requires the onprem-admin-cluster-controller to reconcile the admin cluster in a kind cluster. When the admin cluster state is restored in the kind cluster, the admin cluster webhook can't distinguish if the OnPremAdminCluster object is for an admin cluster creation, or to resume operations for an update or upgrade. Some create-only validations are invoked on updating and upgrading unexpectedly.

Workaround:

Add the onprem.cluster.gke.io/skip-project-location-sameness-validation: true annotation to the OnPremAdminCluster object:

1. Edit the onpremadminclusters cluster resource:

   kubectl  
   edit  
   onpremadminclusters  
    ADMIN_CLUSTER_NAME 
     
   -n  
   kube-system  
   –kubeconfig  
    ADMIN_CLUSTER_KUBECONFIG 
   

   Replace the following:
   • ADMIN_CLUSTER_NAME : the name of the admin cluster.
   • ADMIN_CLUSTER_KUBECONFIG : the path of the admin cluster kubeconfig file.
2. Add the onprem.cluster.gke.io/skip-project-location-sameness-validation: true annotation and save the custom resource.
3. Depending on the type of admin clusters, complete one of the following steps:
   • For non-HA admin clusters with a checkpoint file: add the parameter disable-update-from-checkpoint in the update command, or add the parameter `disable-upgrade-from-checkpoint` in the upgrade command. These parameters are only needed for the next time that you run the update or upgrade command:

     • gkectl  
       update  
       admin  
       --config  
        ADMIN_CONFIG_file 
         
       --kubeconfig  
        ADMIN_CLUSTER_KUBECONFIG 
         
        \ 
         
       --disable-update-from-checkpoint

     • gkectl  
       upgrade  
       admin  
       --config  
        ADMIN_CONFIG_file 
         
       --kubeconfig  
        ADMIN_CLUSTER_KUBECONFIG 
         
        \ 
         
       --disable-upgrade-from-checkpoint

   • For HA admin clusters or checkpoint file is disabled: update or upgrade admin cluster as normal. No additional parameters are needed on the update or upgrade commands.

User cluster deletion fails when using a user-managed admin workstation

When you are logged on to a user-managed admin workstation , the gkectl delete cluster command might timeout and fail to delete the user cluster. This happens if you have first run gkectl on the user-managed workstation to create, update, or upgrade the user cluster. When this failure happens, you see the following error when trying to delete the cluster:

failed to wait for user cluster management namespace " USER_CLUSTER_NAME 
-gke-onprem-mgmt"
      to be deleted: timed out waiting for the condition

During deletion, a cluster first deletes all of its objects. The deletion of the Validation objects (that were created during the create, update, or upgrade) are stuck at the deleting phase. This happens because a finalizer blocks the object's deletion, which causes cluster deletion to fail.

Workaround:

1. Get the names of all the Validation objects:

   kubectl  --kubeconfig ADMIN_KUBECONFIG 
   get validations \
              -n USER_CLUSTER_NAME 
   -gke-onprem-mgmt

2. For each Validation object, run the following command to remove the finalizer from the Validation object:

   kubectl --kubeconfig ADMIN_KUBECONFIG 
   patch validation/ VALIDATION_OBJECT_NAME 
   \
           -n USER_CLUSTER_NAME 
   -gke-onprem-mgmt -p '{"metadata":{"finalizers":[]}}' --type=merge

   After removing the finalizer from all Validation objects, the objects are removed and the user cluster delete operation completes automatically. You don't need to take additional action.

Egress NAT gateway traffic to external server fails

If the source Pod and egress NAT gateway Pod are on two different worker nodes, traffic from the source Pod can't reach any external services. If the Pods are located on the same host, the connection to external service or application is successful.

This issue is caused by vSphere dropping VXLAN packets when tunnel aggregation is enabled. There's a known issue with NSX and VMware that only sends aggregated traffic on known VXLAN ports (4789).

Workaround:

Change the VXLAN port used by Cilium to 4789 :

1. Edit the cilium-config ConfigMap:

   kubectl  
   edit  
   cm  
   -n  
   kube-system  
   cilium-config  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
   

2. Add the following to the cilium-config ConfigMap:

   tunnel-port:  
    4789 
   

3. Restart the anetd DaemonSet:

   kubectl  
   rollout  
   restart  
   ds  
   anetd  
   -n  
   kube-system  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
   

This workaround reverts every time the cluster is upgraded. You must reconfigure after each upgrade. VMware must resolve their issue in vSphere for a permanent fix.

Upgrading an admin cluster with always-on secrets encryption enabled fails

The admin cluster upgrade from 1.14.x to 1.15.x with always-on secrets encryption enabled fails due to a mismatch between the controller-generated encryption key with the key that persists on the admin master data disk. The output of gkectl upgrade admin contains the following error message:

E0926 14:42:21.796444   40110 console.go:93] Exit with error:
      E0926 14:42:21.796491   40110 console.go:93] Failed to upgrade the admin cluster: failed to create admin cluster: failed to wait for OnPremAdminCluster "admin-cluster-name" to become ready: failed to wait for OnPremAdminCluster "admin-cluster-name" to be ready: error: timed out waiting for the condition, message: failed to wait for OnPremAdminCluster "admin-cluster-name" to stay in ready status for duration "2m0s": OnPremAdminCluster "non-prod-admin" is not ready: ready condition is not true: CreateOrUpdateControlPlane: Creating or updating credentials for cluster control plane

Running kubectl get secrets -A --kubeconfig KUBECONFIG ` fails with the following error:

Internal error occurred: unable to transform key "/registry/secrets/anthos-identity-service/ais-secret": rpc error: code = Internal desc = failed to decrypt: unknown jwk

Workaround

If you have a backup of the admin cluster, do the following steps to workaround the upgrade failure:

1. Disable secretsEncryption in the admin cluster configuration file , and update the cluster using the following command:

   gkectl  
   update  
   admin  
   --config  
    ADMIN_CLUSTER_CONFIG_FILE 
     
   --kubeconfig  
    KUBECONFIG 
   

2. When the new admin master VM is created, SSH to the admin master VM, replace the new key on the data disk with the old one from the backup. The key is located at /opt/data/gke-k8s-kms-plugin/generatedkeys on the admin master.
3. Update the kms-plugin.yaml static Pod manifest in /etc/kubernetes/manifests to update the --kek-id to match the kid field in the original encryption key.
4. Restart the kms-plugin static Pod by moving the /etc/kubernetes/manifests/kms-plugin.yaml to another directory then move it back.
5. Resume the admin upgrade by running gkectl upgrade admin again.

Preventing the upgrade failure

If you haven't already upgraded, we recommend that you don't upgrade to 1.15.0-1.15.4. If you must upgrade to an affected version, do the following steps before upgrading the admin cluster:

1. Backup the admin cluster .
2. Disable secretsEncryption in the admin cluster configuration file , and update the cluster using the following command:

   gkectl  
   update  
   admin  
   --config  
    ADMIN_CLUSTER_CONFIG_FILE 
     
   --kubeconfig  
    KUBECONFIG 
   

3. Upgrade the admin cluster.
4. Renable always-on secrets encryption .

Disk errors and attach failures when using Changed Block Tracking (CBT)

Google Distributed Cloud does not support Changed Block Tracking (CBT) on disks. Some backup software uses the CBT feature to track disk state and perform backups, which causes the disk to be unable to connect to a VM that runs Google Distributed Cloud. For more information, see the VMware KB article .

Workaround:

Don't back up the Google Distributed Cloud VMs, as 3rd party backup software might cause CBT to be enabled on their disks. It's not necessary to back up these VMs.

Don't enable CBT on the node, as this change won't persist across updates or upgrades.

If you already have disks with CBT enabled, follow the Resolution steps in the VMware KB article to disable CBT on the First Class Disk.

Data corruption on NFSv3 when parallel appends to a shared file are done from multiple hosts

If you use Nutanix storage arrays to provide NFSv3 shares to your hosts, you might experience data corruption or the inability for Pods to run successfully. This issue is caused by a known compatibility issue between certain versions of VMware and Nutanix versions. For more information, see the associated VMware KB article.

Workaround:

The VMware KB article is out of date in noting that there is no current resolution. To resolve this issue, update to the latest version of ESXi on your hosts and to the latest Nutanix version on your storage arrays.

Version mismatch between the kubelet and the Kubernetes control plane

For certain Google Distributed Cloud releases, the kubelet running on the nodes uses a different version than the Kubernetes control plane. There is a mismatch because the kubelet binary preloaded on the OS image is using a different version.

The following table lists the identified version mismatches:

Workaround:

No action is needed. The inconsistency is only between Kubernetes patch versions and no problems have been caused by this version skew.

Upgrading or updating an admin cluster with a CA version greater than 1 fails

When an admin cluster has a certificate authority (CA) version greater than 1, an update or upgrade fails due to the CA version validation in the webhook. The output of gkectl upgrade/update contains the following error message:

  
CAVersion  
must  
start  
from  
 1 
  

Workaround:

• Scale down the auto-resize-controller deployment in the admin cluster to disable node auto-resizing. This is necessary because a new field introduced to the admin cluster Custom Resource in 1.15 can cause a nil pointer error in the auto-resize-controller .

    
  kubectl  
  scale  
  deployment  
  auto-resize-controller  
  -n  
  kube-system  
  --replicas = 
   0 
    
  --kubeconfig  
   KUBECONFIG 
    
  

• Run gkectl commands with --disable-admin-cluster-webhook flag.For example:

    
  gkectl  
  upgrade  
  admin  
  --config  
   ADMIN_CLUSTER_CONFIG_FILE 
    
  --kubeconfig  
   KUBECONFIG 
    
  --disable-admin-cluster-webhook  
  

Non-HA Controlplane V2 cluster deletion stuck until timeout

When a non-HA Controlplane V2 cluster is deleted, it is stuck at node deletion until it timesout.

Workaround:

If the cluster contains a StatefulSet with critical data, contact contact Cloud Customer Care to resolve this issue.

Otherwise, do the following steps:

• Delete all cluster VMs from vSphere. You can delete the VMs through the vSphere UI, or run the following command:

    
  govc  
  vm.destroy

  .
• Force delete the cluster again:

    
  gkectl  
  delete  
  cluster  
  --cluster  
   USER_CLUSTER_NAME 
    
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  --force  
  

Constant CNS attachvolume tasks appear every minute for in-tree PVC/PV after upgrading to version 1.15+

When a cluster contains in-tree vSphere persistent volumes (for example, PVCs created with the standard StorageClass), you will observe com.vmware.cns.tasks.attachvolume tasks triggered every minute from vCenter.

Workaround:

Edit the vSphere CSI feature configMap and set list-volumes to false:

  
kubectl  
edit  
configmap  
internal-feature-states.csi.vsphere.vmware.com  
-n  
kube-system  
--kubeconfig  
 KUBECONFIG 
  

Restart the vSphere CSI controller pods:

  
kubectl  
rollout  
restart  
deployment  
vsphere-csi-controller  
-n  
kube-system  
--kubeconfig  
 KUBECONFIG 
  

False warnings agaisnt PVCs

When a cluster contains intree vSphere persistent volumes, the commands gkectl diagnose and gkectl upgrade might raise false warnings against their persistent volume claims (PVCs) when validating the cluster storage settings. The warning message looks like the following

  
CSIPrerequisites  
pvc/pvc-name:  
PersistentVolumeClaim  
pvc-name  
bounds  
to  
an  
 in 
-tree  
vSphere  
volume  
created  
before  
CSI  
migration  
enabled,  
but  
it  
doesn ' 
t  
have  
the  
annotation  
pv.kubernetes.io/migrated-to  
 set 
  
to  
csi.vsphere.vmware.com  
after  
CSI  
migration  
is  
enabled  

Workaround:

Run the following command to check the annotations of a PVC with the above warning:

  
kubectl  
get  
pvc  
 PVC_NAME 
  
-n  
 PVC_NAMESPACE 
  
-oyaml  
--kubeconfig  
 KUBECONFIG 
  

If the annotations field in the output contains the following, you can safely ignore the warning:

  
pv.kubernetes.io/bind-completed:  
 "yes" 
  
pv.kubernetes.io/bound-by-controller:  
 "yes" 
  
volume.beta.kubernetes.io/storage-provisioner:  
csi.vsphere.vmware.com  

Service account key rotation fails when multiple keys are expired

If your cluster is not using a private registry, and your component access service account key and Logging-monitoring (or Connect-register) service account keys are expired, when you rotate the service account keys , gkectl update credentials fails with an error similar to the following:

Error:  
reconciliation  
failed:  
failed  
to  
update  
platform:  
...

Workaround:

First, rotate the component access service account key. Although the same error message is displayed, you should be able to rotate the other keys after the component access service account key rotation.

If the update is still not successful, contact Cloud Customer Care to resolve this issue.

1.15 User master machine encounters an unexpected recreation when the user cluster controller is upgraded to 1.16

During a user cluster upgrade, after the user cluster controller is upgraded to 1.16, if you have other 1.15 user clusters managed by the same admin cluster, their user master machine might be unexpectedly recreated.

There is a bug in the 1.16 user cluster controller which can trigger the 1.15 user master machine recreation.

The workaround that you do depends on how you encounter this issue.

Workaround when upgrading the user cluster using the Google Cloud console:

Option 1: Use a 1.16.6+ version of GKE on VMware with the fix.

Option 2: Do the following steps:

1. Manually add the rerun annotation by the following command:

   kubectl  
   edit  
   onpremuserclusters  
    USER_CLUSTER_NAME 
     
   -n  
    USER_CLUSTER_NAME 
   -gke-onprem-mgmt  
   --kubeconfig  
    ADMIN_KUBECONFIG 
   

   The rerun annotation is:

   onprem.cluster.gke.io/server-side-preflight-rerun:  
    true 
   

2. Monitor the upgrade progress by checking the status field of the OnPremUserCluster.

Workaround when upgrading the user cluster using your own admin workstation:

Option 1: Use a 1.16.6+ version of GKE on VMware with the fix.

Option 2: Do the following steps:

1. Add the build info file /etc/cloud/build.info with the following content. This causes the preflight checks to run locally on your admin workstation rather than on the server.

   gke_on_prem_version:  
    GKE_ON_PREM_VERSION 
   

   For example:

   gke_on_prem_version:  
    1 
   .16.0-gke.669

2. Rerun the upgrade command.
3. After the upgrade completes, delete the build.info file.

Preflight check fails when the hostname isn't in the IP block file.

During cluster creation, if you don't specify a hostname for every IP address in the IP block file, the preflight check fails with the following error message:

multiple  
VMs  
found  
by  
DNS  
name  
 in 
  
xxx  
datacenter.  
Anthos  
Onprem  
doesn ' 
t  
support  
duplicate  
hostname  
 in 
  
the  
same  
vCenter  
and  
you  
may  
want  
to  
rename/delete  
the  
existing  
VM. ` 
  

There is a bug in the preflight check which assumes empty hostname as duplicate.

Workaround:

Option 1: Use a version with the fix.

Option 2: Bypass this preflight check by adding --skip-validation-net-config flag.

Option 3: Specify a unique hostname for each IP address in IP block file.

Volume mount failure when upgrade/update the admin cluster if using non-HA admin cluster and control plane v1 user cluster

For a non-HA admin cluster and a control plane v1 user cluster, when you upgrade or update the admin cluster, the admin cluster master machine recreation might happen at the same time as the user cluster master machine reboot, which can surface a race condition. This causes the user cluster control plane Pods to be unable to communicate to the admin cluster control plane, which causes volume attach issues for kube-etcd and kube-apiserver on the user cluster control plane.

To verify the issue, run the following commands for the impacted pod:

kubectl  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 
  
--namespace  
 USER_CLUSTER_NAME 
  
describe  
pod  
 IMPACTED_POD_NAME 

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Warning  FailedMount  101s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[kube-audit], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  86s (x2 over 3m28s)  kubelet            MountVolume.SetUp failed for volume "pvc-77cd0635-57c2-4392-b191-463a45c503cb" : rpc error: code = FailedPrecondition desc = volume ID: "bd313c62-d29b-4ecd-aeda-216648b0f7dc" does not appear staged to "/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/92435c96eca83817e70ceb8ab994707257059734826fedf0c0228db6a1929024/globalmount"

Workaround:

1. SSH into user control plane node , since it is control plane v1 user cluster, the user control plane node is in admin cluster.
2. Restart the kubelet using the following command:

     
   sudo  
   systemctl  
   restart  
   kubelet  
   

   After restart, the kubelet can reconstruct stage global mount properly.

Control plane node fails to be created

During an upgrade or update of an admin cluster, a race condition might cause the vSphere cloud controller manager to unexpectedly delete a new control plane node. This causes the clusterapi-controller to be stuck waiting for the node to be created, and evenutally the upgrade/update times out. In this case, the output of the gkectl upgrade/update command is similar to the following:

  
controlplane  
 'default/gke-admin-hfzdg' 
  
is  
not  
ready:  
condition  
 "Ready" 
:  
condition  
is  
not  
ready  
with  
reason  
 "MachineInitializing" 
,  
message  
 "Wait for the control plane machine " 
gke-admin-hfzdg-6598459f9zb647c8-0 \" 
  
to  
be  
rebooted "... 
  

To identify the symptom, run the command below to get log in vSphere cloud controller manager in the admin cluster:

  
kubectl  
get  
pods  
--kubeconfig  
 ADMIN_KUBECONFIG 
  
-n  
kube-system  
 | 
  
grep  
vsphere-cloud-controller-manager  
kubectl  
logs  
-f  
vsphere-cloud-controller-manager- POD_NAME_SUFFIX 
  
--kubeconfig  
 ADMIN_KUBECONFIG 
  
-n  
kube-system  

Here is a sample error message from the above command:

  
node  
name:  
81ff17e25ec6-qual-335-1500f723  
has  
a  
different  
uuid.  
Skip  
deleting  
this  
node  
from  
cache.  

Workaround:

1. Reboot the failed machine to recreate the deleted node object.
2. SSH into each control plane node and restart the vSphere cloud controller manager static pod:

     
   sudo  
   crictl  
   ps  
    | 
     
   grep  
   vsphere-cloud-controller-manager  
    | 
     
   awk  
    '{print $1}' 
     
   sudo  
   crictl  
   stop  
    PREVIOUS_COMMAND_OUTPUT 
     
   

3. Rerun upgrade/update command.

Duplicate hostname in the same data center causes cluster upgrade or creation failures

Upgrading a 1.15 cluster or creating a 1.16 cluster with static IPs fails if there are duplicate hostnames in the same data center. This failure happens because the vSphere cloud controller manager fails to add an external IP and provider ID in the node object. This causes the cluster upgrade/create to timeout.

To identify the issue, get the vSphere cloud controller manager pod logs for the cluster. The command that you use depends on the cluster type, as follows:

• Admin cluster:

    
  kubectl  
  get  
  pods  
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  -n  
  kube-system  
   | 
    
  grep  
  vsphere-cloud-controller-manager  
  kubectl  
  logs  
  -f  
  vsphere-cloud-controller-manager- POD_NAME_SUFFIX 
    
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  -n  
  kube-system  
  

• User cluster (kubeception) :

    
  kubectl  
  get  
  pods  
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  -n  
   USER_CLUSTER_NAME 
    
   | 
    
  grep  
  vsphere-cloud-controller-manager  
  kubectl  
  logs  
  -f  
  vsphere-cloud-controller-manager- POD_NAME_SUFFIX 
    
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  -n  
   USER_CLUSTER_NAME 
    
  

• User cluster: (Controlplane V2) :

    
  kubectl  
  get  
  pods  
  --kubeconfig  
   USER_KUBECONFIG 
    
  -n  
  kube-system  
   | 
    
  grep  
  vsphere-cloud-controller-manager  
  kubectl  
  logs  
  -f  
  vsphere-cloud-controller-manager- POD_NAME_SUFFIX 
    
  --kubeconfig  
   USER_KUBECONFIG 
    
  -n  
  kube-system  
  

Here is a sample error message:

  
I1003  
 17 
:17:46.769676  
 1 
  
search.go:152 ] 
  
Finding  
node  
admin-vm-2  
 in 
  
 vc 
 = 
vcsa-53598.e5c235a1.asia-northeast1.gve.goog  
and  
 datacenter 
 = 
Datacenter  
E1003  
 17 
:17:46.771717  
 1 
  
datacenter.go:111 ] 
  
Multiple  
vms  
found  
VM  
by  
DNS  
Name.  
DNS  
Name:  
admin-vm-2  

Check if the hostname is duplicated in the data center:

  
 export 
  
 GOVC_DATACENTER 
 = 
 GOVC_DATACENTER 
  
 export 
  
 GOVC_URL 
 = 
 GOVC_URL 
  
 export 
  
 GOVC_USERNAME 
 = 
 GOVC_USERNAME 
  
 export 
  
 GOVC_PASSWORD 
 = 
 GOVC_PASSWORD 
  
 export 
  
 GOVC_INSECURE 
 = 
 true 
  
govc  
find  
.  
-type  
m  
-guest.hostName  
 HOSTNAME 
  

  
 export 
  
 GOVC_DATACENTER 
 = 
mtv-lifecycle-vc01  
 export 
  
 GOVC_URL 
 = 
https://mtv-lifecycle-vc01.anthos/sdk  
 export 
  
 GOVC_USERNAME 
 = 
xxx  
 export 
  
 GOVC_PASSWORD 
 = 
yyy  
 export 
  
 GOVC_INSECURE 
 = 
 true 
  
govc  
find  
.  
-type  
m  
-guest.hostName  
f8c3cd333432-lifecycle-337-xxxxxxxz  
./vm/gke-admin-node-6b7788cd76-wkt8g  
./vm/gke-admin-node-6b7788cd76-99sg2  
./vm/gke-admin-master-5m2jb  

The workaround that you do depends on the operation that failed.

Workaround for upgrades:

Do the workaround for the applicable cluster type.

• User cluster:
  1. Update the hostname of the affected machine in user-ip-block.yaml to a unique name and trigger a forced update:
     

       
     gkectl  
     update  
     cluster  
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     --config  
      NEW_USER_CLUSTER_CONFIG 
       
     --force  
     

  2. Rerun gkectl upgrade cluster
• Admin cluster:
  1. Update the hostname of the affected machine in admin-ip-block.yaml to a unique name and trigger a forced update:
     

       
     gkectl  
     update  
     admin  
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     --config  
      NEW_ADMIN_CLUSTER_CONFIG 
       
     --force  
     --skip-cluster-ready-check  
     

  2. If it is a non-HA admin cluster, and you find admin master vm is using duplicate hostname, you also need to:
     Get admin master machine name

       
     kubectl  
     get  
     machine  
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     -owide  
     -A  
     

     Update admin master machine object
     Note: The NEW_ADMIN_MASTER_HOSTNAME should be same to what you set in admin-ip-block.yaml in step 1.
     

       
     kubectl  
     patch  
     machine  
      ADMIN_MASTER_MACHINE_NAME 
       
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     --type = 
      'json' 
       
     -p  
      '[{"op": "replace", "path": "/spec/providerSpec/value/networkSpec/address/hostname", "value":" NEW_ADMIN_MASTER_HOSTNAME 
     "}]' 
       
     

     Verify hostname is updated in admin master machine object:
     

       
     kubectl  
     get  
     machine  
      ADMIN_MASTER_MACHINE_NAME 
       
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     -oyaml  
     kubectl  
     get  
     machine  
      ADMIN_MASTER_MACHINE_NAME 
       
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     -o  
      jsonpath 
      = 
      '{.spec.providerSpec.value.networkSpec.address.hostname}' 
       
     

  3. Rerun admin cluster upgrade with checkpoint disabled:
     

       
     gkectl  
     upgrade  
     admin  
     --kubeconfig  
      ADMIN_KUBECONFIG 
       
     --config  
      ADMIN_CLUSTER_CONFIG 
       
     --disable-upgrade-from-checkpoint  
     

Workaround for installations:

Do the workaround for the applicable cluster type.

• Admin cluster:
  1. Delete the admin node machine .
  2. Delete the data disk .
  3. Delete the admin cluster checkpoint file .
  4. Update the hostname of the affected machine in admin-ip-block.yaml to a unique name.
  5. Rerun gkectl create admin .
• User cluster:
  1. Clean up resources .
  2. Update the hostname of the affected machine in user-ip-block.yaml to a unique name.
  3. Rerun gkectl create cluster .

$ and ` are not supported in vSphere username or password

The following operations fail when the vSphere username or password contains $ or ` :

• Upgrading a 1.15 user cluster with Controlplane V2 enabled to 1.16
• Upgrading a 1.15 high-availability (HA) admin cluster to 1.16
• Creating a 1.16 user cluster with Controlplane V2 enabled
• Creating a 1.16 HA admin cluster

Use a 1.16.4+ version of Google Distributed Cloud with the fix or perform the below workaround. The workaround that you do depends on the operation that failed.

Workaround for upgrades:

1. Change the vCenter username or password on the vCenter side to remove the $ and ` .
2. Update the vCenter username or password in your credentials configuration file .
3. Trigger a forced update of the cluster.
• User cluster:

    
  gkectl  
  update  
  cluster  
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  --config  
   USER_CLUSTER_CONFIG 
    
  --force  
  

• Admin cluster:

    
  gkectl  
  update  
  admin  
  --kubeconfig  
   ADMIN_KUBECONFIG 
    
  --config  
   ADMIN_CLUSTER_CONFIG 
    
  --force  
  --skip-cluster-ready-check  
  

Workaround for installations:

1. Change the vCenter username or password on the vCenter side to remove the $ and ` .
2. Update the vCenter username or password in your credentials configuration file .
3. Do the workaround for the applicable cluster type.
• Admin cluster:
  1. Delete the admin node machine .
  2. Delete the data disk .
  3. Delete the admin cluster checkpoint file .
  4. Rerun gkectl create admin .
• User cluster:
  1. Clean up resources .
  2. Rerun gkectl create cluster .

PVC creation failure after node is recreated with the same name

After a node is deleted and then recreated with the same node name, there is a slight chance that a subsequent PersistentVolumeClaim (PVC) creation fails with an error like the following:

  
The  
object  
 'vim.VirtualMachine:vm-988369' 
  
has  
already  
been  
deleted  
or  
has  
not  
been  
completely  
created

This is caused by race condition where vSphere CSI controller does not delete a removed machine from its cache.

Workaround:

Restart the vSphere CSI controller pods:

  
kubectl  
rollout  
restart  
deployment  
vsphere-csi-controller  
-n  
kube-system  
--kubeconfig  
 KUBECONFIG 
  

gkectl repair admin-master returns kubeconfig unmarshall error

When you run the gkectl repair admin-master command on an HA admin cluster, gkectl returns the following error message:

  
Exit  
with  
error:  
Failed  
to  
repair:  
failed  
to  
 select 
  
the  
template:  
failed  
to  
get  
cluster  
name  
from  
kubeconfig,  
please  
contact  
Google  
support.  
failed  
to  
decode  
kubeconfig  
data:  
yaml:  
unmarshal  
errors:  
line  
 3 
:  
cannot  
unmarshal  
!!seq  
into  
map [ 
string ] 
*api.Cluster  
line  
 8 
:  
cannot  
unmarshal  
!!seq  
into  
map [ 
string ] 
*api.Context  

Workaround:

Add the --admin-master-vm-template= flag to the command and provide the VM template of the machine to repair:

  
gkectl  
repair  
admin-master  
--kubeconfig = 
 ADMIN_CLUSTER_KUBECONFIG 
  
 \ 
  
--config  
 ADMIN_CLUSTER_CONFIG_FILE 
  
 \ 
  
--admin-master-vm-template = 
/ DATA_CENTER 
/vm/ VM_TEMPLATE_NAME 
  

To find the VM template of the machine:

1. Go to the Hosts and Clusters page in the vSphere client.
2. Click VM Templates and filter by the admin cluster name.

   You should see the three VM templates for the admin cluster.

3. Copy the name VM template that matches the name of the machine you're repairing and use the template name in the repair command.

  
gkectl  
repair  
admin-master  
 \ 
  
--config = 
/home/ubuntu/admin-cluster.yaml  
 \ 
  
--kubeconfig = 
/home/ubuntu/kubeconfig  
 \ 
  
--admin-master-vm-template = 
/atl-qual-vc07/vm/gke-admin-98g94-zx...7vx-0-tmpl

Seesaw VM broken due to disk space low

If you use Seesaw as the load balancer type for your cluster and you see that a Seesaw VM is down or keeps failing to boot, you might see the following error message in the vSphere console:

  
GRUB_FORCE_PARTUUID  
set,  
initrdless  
boot  
failed.  
Attempting  
with  
initrd  

This error indicates that the disk space is low on the VM because the fluent-bit running on the Seesaw VM is not configured with correct log rotation.

Workaround:

Locate the log files that consume most of the disk space using du -sh -- /var/lib/docker/containers/* | sort -rh . Clean up the log file with largest size and reboot the VM.

Note: If the VM is completely inaccessible, attach the disk to a working VM (e.g. admin workstation), remove the file from the attached disk, then reattach the disk back to the original Seesaw VM.

To prevent the issue from happening again, connect to the VM and modify the /etc/systemd/system/docker.fluent-bit.service file. Add --log-opt max-size=10m --log-opt max-file=5 in the Docker command, then run systemctl restart docker.fluent-bit.service

Admin SSH public key error after admin cluster upgrade or update

When you try to upgrade ( gkectl upgrade admin ) or update ( gkectl update admin ) a non-High-Availability admin cluster with checkpoint enabled, the upgrade or update may fail with errors like the following:

Checking  
admin  
cluster  
certificates...FAILURE  
Reason:  
 20 
  
admin  
cluster  
certificates  
error ( 
s ) 
.
Unhealthy  
Resources:  
AdminMaster  
clusterCA  
bundle:  
failed  
to  
get  
clusterCA  
bundle  
on  
admin  
master,  
 command 
  
 [ 
ssh  
-o  
 IdentitiesOnly 
 = 
yes  
-i  
admin-ssh-key  
-o  
 StrictHostKeyChecking 
 = 
no  
-o  
 ConnectTimeout 
 = 
 30 
  
ubuntu@AdminMasterIP  
--  
sudo  
cat  
/etc/kubernetes/pki/ca-bundle.crt ] 
  
failed  
with  
error:  
 exit 
  
status  
 255 
,  
stderr:  
Authorized  
uses  
only.  
All  
activity  
may  
be  
monitored  
and  
reported.  
ubuntu@AdminMasterIP:  
Permission  
denied  
 ( 
publickey ) 
.

failed  
to  
ssh  
AdminMasterIP,  
failed  
with  
error:  
 exit 
  
status  
 255 
,  
stderr:  
Authorized  
uses  
only.  
All  
activity  
may  
be  
monitored  
and  
reported.  
ubuntu@AdminMasterIP:  
Permission  
denied  
 ( 
publickey ) 

error  
dialing  
ubuntu@AdminMasterIP:  
failed  
to  
establish  
an  
authenticated  
SSH  
connection:  
ssh:  
handshake  
failed:  
ssh:  
unable  
to  
authenticate,  
attempted  
methods  
 [ 
none  
publickey ] 
...

Workaround:

If you're unable to upgrade to a patch version of Google Distributed Cloud with the fix, contact Google Support for assistance.

Upgrading an admin cluster enrolled in the Anthos On-Prem API could fail

When an admin cluster is enrolled in the Anthos On-Prem API, upgrading the admin cluster to the affected versions could fail because the fleet membership couldn't be updated. When this failure happens, you see the following error when trying to upgrade the cluster:

  
failed  
to  
register  
cluster:  
failed  
to  
apply  
Hub  
Membership:  
Membership  
API  
request  
failed:  
rpc  
error:  
 code 
  
 = 
  
InvalidArgument  
 desc 
  
 = 
  
InvalidFieldError  
 for 
  
field  
endpoint.on_prem_cluster.resource_link:  
field  
cannot  
be  
updated  

An admin cluster is enrolled in the API when you explicitly enroll the cluster, or when you upgrade a user cluster using a Anthos On-Prem API client .

Workaround:

  
gcloud  
alpha  
container  
vmware  
admin-clusters  
unenroll  
 ADMIN_CLUSTER_NAME 
  
--project  
 CLUSTER_PROJECT 
  
--location = 
 CLUSTER_LOCATION 
  
--allow-missing  

Enrolled admin cluster's resource link annotation is not preserved

When an admin cluster is enrolled in the Anthos On-Prem API, its resource link annotation is applied to the OnPremAdminCluster custom resource, which is not preserved during later admin cluster updates due to the wrong annotation key being used. This can cause the admin cluster to be enrolled in the Anthos On-Prem API again by mistake.

An admin cluster is enrolled in the API when you explicitly enroll the cluster, or when you upgrade a user cluster using a Anthos On-Prem API client .

Workaround:

  
gcloud  
alpha  
container  
vmware  
admin-clusters  
unenroll  
 ADMIN_CLUSTER_NAME 
  
--project  
 CLUSTER_PROJECT 
  
--location = 
 CLUSTER_LOCATION 
  
--allow-missing  

CoreDNS orderPolicy not recognized

OrderPolicy doesn't get recognized as a parameter and isn't used. Instead, Google Distributed Cloud always uses Random .

This issue occurs because the CoreDNS template was not updated, which causes orderPolicy to be ignored.

Workaround:

Update the CoreDNS template and apply the fix. This fix persists until an upgrade.

1. Edit the existing template:

   kubectl  
   edit  
   cm  
   -n  
   kube-system  
   coredns-template

   Replace the contents of the template with the following:

   coredns-template:  
    | 
   -  
   .:53  
    { 
     
   errors  
   health  
    { 
     
   lameduck  
   5s  
    } 
     
   ready  
   kubernetes  
   cluster.local  
    in 
   -addr.arpa  
   ip6.arpa  
    { 
     
   pods  
   insecure  
   fallthrough  
    in 
   -addr.arpa  
   ip6.arpa  
    } 
   { { 
   -  
    if 
     
   .PrivateGoogleAccess  
    }} 
     
   import  
   zones/private.Corefile
   { { 
   -  
   end  
    }} 
   { { 
   -  
    if 
     
   .RestrictedGoogleAccess  
    }} 
     
   import  
   zones/restricted.Corefile
   { { 
   -  
   end  
    }} 
     
   prometheus  
   :9153  
   forward  
   .  
   { { 
     
   .UpstreamNameservers  
    }} 
     
    { 
     
   max_concurrent  
    1000 
     
   { { 
   -  
    if 
     
   ne  
   .OrderPolicy  
    "" 
     
    }} 
     
   policy  
   { { 
     
   .OrderPolicy  
    }} 
     
   { { 
   -  
   end  
    }} 
     
    } 
     
   cache  
    30 
   { { 
   -  
    if 
     
   .DefaultDomainQueryLogging  
    }} 
     
   log
   { { 
   -  
   end  
    }} 
     
   loop  
   reload  
   loadbalance }{{ 
     
   range  
    $i 
   ,  
    $stubdomain 
     
   : = 
     
   .StubDomains  
    }} 
   { { 
     
    $stubdomain 
   .Domain  
    }} 
   :53  
    { 
     
   errors
   { { 
   -  
    if 
     
    $stubdomain 
   .QueryLogging  
    }} 
     
   log
   { { 
   -  
   end  
    }} 
     
   cache  
    30 
     
   forward  
   .  
   { { 
     
    $stubdomain 
   .Nameservers  
    }} 
     
    { 
     
   max_concurrent  
    1000 
     
   { { 
   -  
    if 
     
   ne  
   $.OrderPolicy  
    "" 
     
    }} 
     
   policy  
   { { 
     
   $.OrderPolicy  
    }} 
     
   { { 
   -  
   end  
    }} 
     
    } 
    } 
   { { 
   -  
   end  
    }} 
   

OnPremAdminCluster status inconsistent between checkpoint and actual CR

Certain race conditions could cause the OnPremAdminCluster status to be inconsistent between checkpoint and actual CR. When the issue happens, you could encounter the following error when update the admin cluster after you upgraded it:

Exit  
with  
error:
E0321  
 10 
:20:53.515562  
 961695 
  
console.go:93 ] 
  
Failed  
to  
update  
the  
admin  
cluster:  
OnPremAdminCluster  
 "gke-admin-rj8jr" 
  
is  
 in 
  
the  
middle  
of  
a  
create/upgrade  
 ( 
 "" 
  
->  
 "1.15.0-gke.123" 
 ) 
,  
which  
must  
be  
completed  
before  
it  
can  
be  
updated
Failed  
to  
update  
the  
admin  
cluster:  
OnPremAdminCluster  
 "gke-admin-rj8jr" 
  
is  
 in 
  
the  
middle  
of  
a  
create/upgrade  
 ( 
 "" 
  
->  
 "1.15.0-gke.123" 
 ) 
,  
which  
must  
be  
completed  
before  
it  
can  
be  
updated

Reconciliation process changes admin certificates on admin clusters

Google Distributed Cloud changes the admin certificates on admin cluster control planes with every reconciliation process, such as during a cluster upgrade. This behavior increases the possibility of getting invalid certificates for your admin cluster, especially for version 1.15 clusters.

If you're affected by this issue, you may encounter problems like the following:

• Invalid certificates may cause the following commands to time out and return errors:
  • gkectl create admin
  • gkectl upgrade amdin
  • gkectl update admin

  These commands may return authorization errors like the following:

  Failed  
  to  
  reconcile  
  admin  
  cluster:  
  unable  
  to  
  populate  
  admin  
  clients:  
  failed  
  to  
  get  
  admin  
  controller  
  runtime  
  client:  
  Unauthorized

• The kube-apiserver logs for your admin cluster may contain errors like the following:

  Unable  
  to  
  authenticate  
  the  
  request " err=" 
   [ 
  x509:  
  certificate  
  has  
  expired  
  or  
  is  
  not  
  yet  
  valid...

Workaround:

Upgrade to a version of Google Distributed Cloud with the fix: 1.13.10+, 1.14.6+, 1.15.2+. If upgrading isn't feasible for you, contact Cloud Customer Care to resolve this issue.

Anthos Network Gateway components evicted or pending due to missing priority class

Network gateway Pods in kube-system might show a status of Pending or Evicted , as shown in the following condensed example output:

$  
kubectl  
-n  
kube-system  
get  
pods  
 | 
  
grep  
ang-node
ang-node-bjkkc  
 2 
/2  
Running  
 0 
  
5d2h
ang-node-mw8cq  
 0 
/2  
Evicted  
 0 
  
6m5s
ang-node-zsmq7  
 0 
/2  
Pending  
 0 
  
7h

These errors indicate eviction events or an inability to schedule Pods due to node resources. As Anthos Network Gateway Pods have no PriorityClass, they have the same default priority as other workloads. When nodes are resource-constrained, the network gateway Pods might be evicted. This behavior is particularly bad for the ang-node DaemonSet, as those Pods must be scheduled on a specific node and can't migrate.

Workaround:

Upgrade to 1.15 or later.

As a short-term fix, you can manually assign a PriorityClass to the Anthos Network Gateway components. The Google Distributed Cloud controller overwrites these manual changes during a reconciliation process, such as during a cluster upgrade.

• Assign the system-cluster-critical PriorityClass to the ang-controller-manager and autoscaler cluster controller Deployments.
• Assign the system-node-critical PriorityClass to the ang-daemon node DaemonSet.

admin cluster upgrade fails after registering the cluster with gcloud

After you use gcloud to register an admin cluster with non-empty gkeConnect section, you might see the following error when trying to upgrade the cluster:

failed  
to  
register  
cluster:  
failed  
to  
apply  
Hub  
Mem \ 
bership:  
Membership  
API  
request  
failed:  
rpc  
error:  
 code 
  
 = 
  
InvalidArgument  
 desc 
  
 = 
  
InvalidFieldError  
 for 
  
field  
endpoint.o \ 
n_prem_cluster.admin_cluster:  
field  
cannot  
be  
updated

Delete the gke-connect namespace:

kubectl  
delete  
ns  
gke-connect  
--kubeconfig = 
 ADMIN_KUBECONFIG 

kubectl  
get  
onpremadmincluster  
-n  
kube-system  
--kubeconfig = 
 ADMIN_KUBECONFIG 

gcloud  
container  
fleet  
memberships  
delete  
 ADMIN_CLUSTER_NAME 

gkectl diagnose snapshot --log-since fails to limit the time window for journalctl commands running on the cluster nodes

This does not affect the functionality of taking a snapshot of the cluster, as the snapshot still includes all logs that are collected by default by running journalctl on the cluster nodes. Therefore, no debugging information is missed.

gkectl prepare windows fails

gkectl prepare windows fails to install Docker on Google Distributed Cloud versions earlier than 1.13 because MicrosoftDockerProvider is deprecated.

Workaround:

The general idea to workaround this issue is to upgrade to Google Distributed Cloud 1.13 and use the 1.13 gkectl to create a Windows VM template and then create Windows node pools. There are two options to get to Google Distributed Cloud 1.13 from your current version as shown below.

Note: We do have options to workaround this issue in your current version without needing to upgrade all the way to 1.13, but it will need more manual steps, please reach out to our support team if you would like to consider this option.

Option 1: Blue/Green upgrade

You can create a new cluster using Google Distributed Cloud 1.13+ version with windows node pools, and migrate your workloads to the new cluster, then tear down the current cluster. It's recommended to use the latest Google Distributed Cloud minor version.

Note: This will require extra resources to provision the new cluster, but less downtime and disruption for existing workloads.

Option 2: Delete Windows node pools and add them back when upgrading to Google Distributed Cloud 1.13

Note: For this option, the Windows workloads will not be able to run until the cluster is upgraded to 1.13 and Windows node pools are added back.

1. Delete existing Windows node pools by removing the windows node pools config from user-cluster.yaml file, then run the command:

   gkectl  
   update  
   cluster  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   --config  
    USER_CLUSTER_CONFIG_FILE 
   

2. Upgrade the Linux-only admin+user clusters to 1.12 following the upgrade user guide for the corresponding target minor version.
3. (Make sure to perform this step before upgrading to 1.13) Ensure the enableWindowsDataplaneV2: true is configured in OnPremUserCluster CR, otherwise the cluster will keep using Docker for Windows node pools, which will not be compatible with the newly created 1.13 Windows VM template that not have Docker installed. If not configured or setting to false, update your cluster to set it to true in user-cluster.yaml, then run:

   gkectl  
   update  
   cluster  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   --config  
    USER_CLUSTER_CONFIG_FILE 
   

4. Upgrade the Linux-only admin+user clusters to 1.13 following the upgrade user guide .
5. Prepare Windows VM template using 1.13 gkectl:

   gkectl  
   prepare  
   windows  
   --base-vm-template  
    BASE_WINDOWS_VM_TEMPLATE_NAME 
     
   --bundle-path  
     1 
   .13_BUNDLE_PATH 
     
   --kubeconfig = 
    ADMIN_KUBECONFIG 
   

6. Add back the Windows node pool configuration to user-cluster.yaml with the OSImage field set to the newly created Windows VM template.
7. Update the cluster to add Windows node pools

   gkectl  
   update  
   cluster  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   --config  
    USER_CLUSTER_CONFIG_FILE 
   

RootDistanceMaxSec configuration not taking effect for ubuntu nodes

The 5 seconds default value for RootDistanceMaxSec will be used on the nodes, instead of 20 seconds which should be the expected configuration. If you check the node startup log by SSH'ing into the VM, which is located at `/var/log/startup.log`, you can find the following error:

+  
has_systemd_unit  
systemd-timesyncd
/opt/bin/master.sh:  
line  
 635 
:  
has_systemd_unit:  
 command 
  
not  
found

Using a 5 seconds RootDistanceMaxSec might cause the system clock to be out of sync with NTP server when the clock drift is larger than 5 seconds.

Workaround:

Apply the following DaemonSet to your cluster to configure RootDistanceMaxSec :

 apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 DaemonSet 
 metadata 
 : 
  
 name 
 : 
  
 change-root-distance 
  
 namespace 
 : 
  
 kube-system 
 spec 
 : 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 app 
 : 
  
 change-root-distance 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 app 
 : 
  
 change-root-distance 
  
 spec 
 : 
  
 hostIPC 
 : 
  
 true 
  
 hostPID 
 : 
  
 true 
  
 tolerations 
 : 
  
 # Make sure pods gets scheduled on all nodes. 
  
 - 
  
 effect 
 : 
  
 NoSchedule 
  
 operator 
 : 
  
 Exists 
  
 - 
  
 effect 
 : 
  
 NoExecute 
  
 operator 
 : 
  
 Exists 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 change-root-distance 
  
 image 
 : 
  
 ubuntu 
  
 command 
 : 
  
 [ 
 "chroot" 
 , 
  
 "/host" 
 , 
  
 "bash" 
 , 
  
 "-c" 
 ] 
  
 args 
 : 
  
 - 
  
 | 
  
 while true; do 
  
 conf_file="/etc/systemd/timesyncd.conf.d/90-gke.conf" 
  
 if [ -f $conf_file ] && $(grep -q "RootDistanceMaxSec=20" $conf_file); then 
  
 echo "timesyncd has the expected RootDistanceMaxSec, skip update" 
  
 else 
  
 echo "updating timesyncd config to RootDistanceMaxSec=20" 
  
 mkdir -p /etc/systemd/timesyncd.conf.d 
  
 cat > $conf_file << EOF 
  
 [Time] 
  
 RootDistanceMaxSec=20 
  
 EOF 
  
 systemctl restart systemd-timesyncd 
  
 fi 
  
 sleep 600 
  
 done 
  
 volumeMounts 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 mountPath 
 : 
  
 /host 
  
 securityContext 
 : 
  
 privileged 
 : 
  
 true 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 hostPath 
 : 
  
 path 
 : 
  
 / 

gkectl update admin fails because of empty osImageType field

When you use version 1.13 gkectl to update a version 1.12 admin cluster, you might see the following error:

Failed  
to  
update  
the  
admin  
cluster:  
updating  
OS  
image  
 type 
  
 in 
  
admin  
cluster
is  
not  
supported  
 in 
  
 "1.12.x-gke.x" 

When you use gkectl update admin for version 1.13 or 1.14 clusters, you might see the following message in the response:

Exit  
with  
error:
Failed  
to  
update  
the  
cluster:  
the  
update  
contains  
multiple  
changes.  
Please
update  
only  
one  
feature  
at  
a  
 time 

If you check the gkectl log, you might see that the multiple changes include setting osImageType from an empty string to ubuntu_containerd .

These update errors are due to improper backfilling of the osImageType field in the admin cluster config since it was introduced in version 1.9.

Workaround:

Upgrade to a version of Google Distributed Cloud with the fix. If upgrading isn't feasible for you, contact Cloud Customer Care to resolve this issue.

SNI doesn't work on user clusters with Controlplane V2

The ability to provide an additional serving certificate for the Kubernetes API server of a user cluster with authentication.sni doesn't work when the Controlplane V2 is enabled ( enableControlplaneV2: true ).

Workaround:

Until a Google Distributed Cloud patch is available with the fix, if you need to use SNI, disable Controlplane V2 ( enableControlplaneV2: false ).

$ in the private registry username causes admin control plane machine startup failure

The admin control plane machine fails to start up when the private registry username contains $ . When checking the /var/log/startup.log on the admin control plane machine, you see the following error:

++  
 REGISTRY_CA_CERT 
 = 
xxx
++  
 REGISTRY_SERVER 
 = 
xxx
/etc/startup/startup.conf:  
line  
 7 
:  
anthos:  
unbound  
variable

Workaround:

Use a private registry username without $ , or use a version of Google Distributed Cloud with the fix.

False-positive warnings about unsupported changes during admin cluster update

When you update admin clusters , you will see the following false-positive warnings in the log, and you can ignore them.

  
console.go:47 ] 
  
detected  
unsupported  
changes:  
 & 
v1alpha1.OnPremAdminCluster { 
  
...  
-  
CARotation:  
 & 
v1alpha1.CARotationConfig { 
Generated:  
 & 
v1alpha1.CARotationGenerated { 
CAVersion:  
 1 
 }} 
,  
+  
CARotation:  
nil,  
...  
 } 

Update user cluster failed after KSA signing key rotation

After you rotate KSA signing keys and subsequently update a user cluster , gkectl update might fail with the following error message:

Failed  
to  
apply  
OnPremUserCluster  
 'USER_CLUSTER_NAME-gke-onprem-mgmt/USER_CLUSTER_NAME' 
:
admission  
webhook  
 "vonpremusercluster.onprem.cluster.gke.io" 
  
denied  
the  
request:
requests  
must  
not  
decrement  
*v1alpha1.KSASigningKeyRotationConfig  
Version,  
old  
version:  
 2 
,  
new  
version:  
 1 
 " 

Workaround:

1. Check the secret in admin cluster under USER_CLUSTER_NAME namespace, and get the name of ksa-signing-key secret:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
     
   get  
   secrets  
    | 
     
   grep  
   ksa-signing-key

2. Copy the ksa-signing-key secret, and name the copied secret as service-account-cert:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
     
   get  
   secret  
    KSA-KEY-SECRET-NAME 
     
   -oyaml  
    | 
     
    \ 
   sed  
    's/ name: .*/ name: service-account-cert/' 
     
    | 
     
    \ 
   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
     
   apply  
   -f  
   -

3. Delete the previous ksa-signing-key secret:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
     
   delete  
   secret  
    KSA-KEY-SECRET-NAME 
   

4. Update the data.data field in ksa-signing-key-rotation-stage configmap to '{"tokenVersion":1,"privateKeyVersion":1,"publicKeyVersions":[1]}' :

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
     
    \ 
   edit  
   configmap  
   ksa-signing-key-rotation-stage

5. Disable the validation webhook to edit the version information in the OnPremUserCluster custom resource:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   patch  
   validatingwebhookconfiguration  
   onprem-user-cluster-controller  
   -p  
    ' 
    webhooks: 
    - name: vonpremnodepool.onprem.cluster.gke.io 
    rules: 
    - apiGroups: 
    - onprem.cluster.gke.io 
    apiVersions: 
    - v1alpha1 
    operations: 
    - CREATE 
    resources: 
    - onpremnodepools 
    - name: vonpremusercluster.onprem.cluster.gke.io 
    rules: 
    - apiGroups: 
    - onprem.cluster.gke.io 
    apiVersions: 
    - v1alpha1 
    operations: 
    - CREATE 
    resources: 
    - onpremuserclusters 
    ' 
   

6. Update the spec.ksaSigningKeyRotation.generated.ksaSigningKeyRotation field to 1 in your OnPremUserCluster custom resource:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
   -gke-onprem-mgmt  
    \ 
   edit  
   onpremusercluster  
    USER_CLUSTER_NAME 
   

7. Wait until the target user cluster to be ready, you can check the status by:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
    USER_CLUSTER_NAME 
   -gke-onprem-mgmt  
    \ 
   get  
   onpremusercluster

8. Restore the validation webhook for the user cluster:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   patch  
   validatingwebhookconfiguration  
   onprem-user-cluster-controller  
   -p  
    ' 
    webhooks: 
    - name: vonpremnodepool.onprem.cluster.gke.io 
    rules: 
    - apiGroups: 
    - onprem.cluster.gke.io 
    apiVersions: 
    - v1alpha1 
    operations: 
    - CREATE 
    - UPDATE 
    resources: 
    - onpremnodepools 
    - name: vonpremusercluster.onprem.cluster.gke.io 
    rules: 
    - apiGroups: 
    - onprem.cluster.gke.io 
    apiVersions: 
    - v1alpha1 
    operations: 
    - CREATE 
    - UPDATE 
    resources: 
    - onpremuserclusters 
    ' 
   

9. Avoid another KSA signing key rotation until the cluster is upgraded to the version with the fix.

F5 BIG-IP virtual servers aren't cleaned up when Terraform deletes user clusters

When you use Terraform to delete a user cluster with a F5 BIG-IP load balancer, the F5 BIG-IP virtual servers aren't removed after the cluster deletion.

Workaround:

To remove the F5 resources, follow the steps to clean up a user cluster F5 partition

kind cluster pulls container images from docker.io

If you create a version 1.13.8 or version 1.14.4 admin cluster, or upgrade an admin cluster to version 1.13.8 or 1.14.4, the kind cluster pulls the following container images from docker.io :

If docker.io isn't accessible from your admin workstation, the admin cluster creation or upgrade fails to bring up the kind cluster. Running the following command on the admin workstation shows the corresponding containers pending with ErrImagePull :

docker  
 exec 
  
gkectl-control-plane  
kubectl  
get  
pods  
-A

The response contains entries like the following:

...
kube-system  
kindnet-xlhmr  
 0 
/1  
ErrImagePull  
 0 
  
3m12s
...
local-path-storage  
local-path-provisioner-86666ffff6-zzqtp  
 0 
/1  
Pending  
 0 
  
3m12s
...

These container images should be preloaded in the kind cluster container image. However, kind v0.18.0 has an issue with the preloaded container images , which causes them to be pulled from the internet by mistake.

Workaround:

Run the following commands on the admin workstation, while your admin cluster is pending on creation or upgrade:

docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/kindnetd:v20230330-48f316cd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497af  
docker.io/kindest/kindnetd:v20230330-48f316cd
docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/kindnetd:v20230330-48f316cd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497af  
docker.io/kindest/kindnetd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497af

docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/local-path-helper:v20230330-48f316cd@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270  
docker.io/kindest/local-path-helper:v20230330-48f316cd
docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/local-path-helper:v20230330-48f316cd@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270  
docker.io/kindest/local-path-helper@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270

docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/local-path-provisioner:v0.0.23-kind.0@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501  
docker.io/kindest/local-path-provisioner:v0.0.23-kind.0
docker  
 exec 
  
gkectl-control-plane  
ctr  
-n  
k8s.io  
images  
tag  
docker.io/kindest/local-path-provisioner:v0.0.23-kind.0@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501  
docker.io/kindest/local-path-provisioner@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501

Unsuccessful failover on HA Controlplane V2 user cluster and admin cluster when the network filters out duplicate GARP requests

If your cluster VMs are connected with a switch that filters out duplicate GARP (gratuitous ARP) requests, the keepalived leader election might encounter a race condition, which causes some nodes to have incorrect ARP table entries.

The affected nodes can ping the control plane VIP, but a TCP connection to the control plane VIP will time out.

Workaround:

  
iptables  
-I  
FORWARD  
-i  
ens192  
--destination  
 CONTROL_PLANE_VIP 
  
-j  
DROP  

vsphere-csi-controller needs be restarted after the vCenter certificate rotation

vsphere-csi-controller should refresh its vCenter secret after vCenter certificate rotation. However, the current system does not properly restart the pods of vsphere-csi-controller , causing vsphere-csi-controller to crash after the rotation.

Workaround:

For clusters created at 1.13 and later versions, follow the instructions below to restart vsphere-csi-controller

kubectl --kubeconfig= ADMIN_KUBECONFIG 
rollout restart deployment vsphere-csi-controller -n kube-system

Admin cluster creation does not fail on cluster registration errors

Even when cluster registration fails during admin cluster creation, the command gkectl create admin does not fail on the error and might succeed. In other words, the admin cluster creation could "succeed" without being registered to a fleet.

Failed to register admin cluster

You can also check whether you can find the cluster among registered clusters on cloud console.

Workaround:

For clusters created at 1.12 and later versions, follow the instructions for re-attempting the admin cluster registration after cluster creation. For clusters created at earlier versions,

1. Append a fake key-value pair like "foo: bar" to your connect-register SA key file
2. Run gkectl update admin to re-register the admin cluster.

Admin cluster re-registration might be skipped during admin cluster upgrade

During admin cluster upgrade, if upgrading user control plane nodes times out, the admin cluster will not be re-registered with the updated connect agent version.

Workaround:

1. Append a fake key-value pair like "foo: bar" to your connect-register SA key file
2. Run gkectl update admin to re-register the admin cluster.

False error message about vCenter.dataDisk

For a high-availability admin cluster, gkectl prepare shows this false error message:

vCenter.dataDisk must be present in the AdminCluster spec

Workaround:

You can safely ignore this error message.

Node pool creation fails because of redundant VM-Host affinity rules

During creation of a node pool that uses VM-Host affinity , a race condition might result in multiple VM-Host affinity rules being created with the same name. This can cause node pool creation to fail.

Workaround:

Remove the old redundant rules so that node pool creation can proceed. These rules are named [USER_CLUSTER_NAME] - [HASH] .

gkectl repair admin-master may fail due to failed to delete the admin master node object and reboot the admin master VM

The gkectl repair admin-master command may fail due to a race condition with the following error.

Failed  
to  
repair:  
failed  
to  
delete  
the  
admin  
master  
node  
object  
and  
reboot  
the  
admin  
master  
VM

Workaround:

This command is idempotent. It can rerun safely until the command succeeds.

Pods remain in Failed state afer re-creation or update of a control-plane node

After you re-create or update a control-plane node, certain Pods might be left in the Failed state due to NodeAffinity predicate failure. These failed Pods don't affect normal cluster operations or health.

Workaround:

You can safely ignore the failed Pods or manually delete them.

OnPremUserCluster not ready because of private registry credentials

If you use prepared credentials and a private registry, but you haven't configured prepared credentials for your private registry, the OnPremUserCluster might not become ready, and you might see the following error message:

failed to check secret reference for private registry …

Workaround:

Prepare the private registry credentials for the user cluster according to the instructions in Configure prepared credentials .

gkectl upgrade admin fails with StorageClass standard sets the parameter diskformat which is invalid for CSI Migration

During gkectl upgrade admin , the storage preflight check for CSI Migration verifies that the StorageClasses don't have parameters that are ignored after CSI Migration. For example, if there's a StorageClass with the parameter diskformat then gkectl upgrade admin flags the StorageClass and reports a failure in the preflight validation. Admin clusters created in Google Distributed Cloud 1.10 and before have a StorageClass with diskformat: thin which will fail this validation however this StorageClass still works fine after CSI Migration. These failures should be interpreted as warnings instead.

For more information, check the StorageClass parameter section in Migrating In-Tree vSphere Volumes to vSphere Container Storage Plug-in .

Workaround:

After confirming that your cluster has a StorageClass with parameters ignored after CSI Migration run gkectl upgrade admin with the flag --skip-validation-cluster-health .

Migrated in-tree vSphere volumes using the Windows file system can't be used with vSphere CSI driver

Under certain conditions disks can be attached as readonly to Windows nodes. This results in the corresponding volume being readonly inside a Pod. This problem is more likely to occur when a new set of nodes replaces an old set of nodes (for example, cluster upgrade or node pool update). Stateful workloads that previously worked fine might be unable to write to their volumes on the new set of nodes.

Workaround:

1. Get the UID of the Pod that is unable to write to its volume:

   kubectl  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
     
   get  
   pod  
    \ 
     
    POD_NAME 
     
   --namespace  
    POD_NAMESPACE 
     
    \ 
     
   -o = 
    jsonpath 
    = 
    '{.metadata.uid}{"\n"}' 
   

2. Use the PersistentVolumeClaim to get the name of the PersistentVolume:

   kubectl  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
     
   get  
   pvc  
    \ 
     
    PVC_NAME 
     
   --namespace  
    POD_NAMESPACE 
     
    \ 
     
   -o  
    jsonpath 
    = 
    '{.spec.volumeName}{"\n"}' 
   

3. Determine the name of the node where the Pod is running:

   kubectl  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
   get  
   pods  
    \ 
     
   --namespace  
    POD_NAMESPACE 
     
    \ 
     
   -o  
    jsonpath 
    = 
    '{.spec.nodeName}{"\n"}' 
   

4. Obtain powershell access to the node, either through SSH or the vSphere web interface.
5. Set environment variables:

   PS C:\Users\administrator> pvname= PV_NAME 
   PS C:\Users\administrator> podid= POD_UID 
   

6. Identify the disk number for the disk associated with the PersistentVolume:

   PS C:\Users\administrator> disknum=(Get-Partition -Volume (Get-Volume -UniqueId ("\\?\"+(Get-Item (Get-Item
   "C:\var\lib\kubelet\pods\$podid\volumes\kubernetes.io~csi\$pvname\mount").Target).Target))).DiskNumber

7. Verify that the disk is readonly :

   PS C:\Users\administrator> (Get-Disk -Number $disknum).IsReadonly

   The result should be True .
8. Set readonly to false .

   PS C:\Users\administrator> Set-Disk -Number $disknum -IsReadonly $false
   PS C:\Users\administrator> (Get-Disk -Number $disknum).IsReadonly

9. Delete the Pod so that it will get restarted:

   kubectl  
   --kubeconfig  
    USER_CLUSTER_KUBECONFIG 
     
   delete  
   pod  
    POD_NAME 
     
    \ 
     
   --namespace  
    POD_NAMESPACE 
   

10. The Pod should get scheduled to the same node. But in case the Pod gets scheduled to a new node, you might need to repeat the preceding steps on the new node.

vsphere-csi-secret is not updated after gkectl update credentials vsphere --admin-cluster

If you update the vSphere credentials for an admin cluster following updating cluster credentials , you might find vsphere-csi-secret under kube-system namespace in the admin cluster still uses the old credential.

Workaround:

1. Get the vsphere-csi-secret secret name:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
   kube-system  
   get  
   secrets  
    | 
     
   grep  
   vsphere-csi-secret

2. Update the data of the vsphere-csi-secret secret you got from the above step:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
   kube-system  
   patch  
   secret  
    CSI_SECRET_NAME 
     
   -p  
    \ 
     
    "{\"data\":{\"config\":\" 
    $( 
     
    \ 
     
   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
   kube-system  
   get  
   secrets  
    CSI_SECRET_NAME 
     
   -ojsonpath = 
    '{.data.config}' 
     
    \ 
     
    | 
     
   base64  
   -d  
    \ 
     
    | 
     
   sed  
   -e  
    '/user/c user = \" VSPHERE_USERNAME_TO_BE_UPDATED 
   \"' 
     
    \ 
     
    | 
     
   sed  
   -e  
    '/password/c password = \" VSPHERE_PASSWORD_TO_BE_UPDATED 
   \"' 
     
    \ 
     
    | 
     
   base64  
   -w  
    0 
     
    \ 
     
    ) 
    \"}}" 
   

3. Restart vsphere-csi-controller :

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
   kube-system  
   rollout  
   restart  
   deployment  
   vsphere-csi-controller

4. You can track the rollout status with:

   kubectl  
   --kubeconfig = 
    ADMIN_KUBECONFIG 
     
   -n = 
   kube-system  
   rollout  
   status  
   deployment  
   vsphere-csi-controller

   After the deployment is successfully rolled out, the updated vsphere-csi-secret should be used by the controller.

audit-proxy crashloop when enabling Cloud Audit Logs with gkectl update cluster

audit-proxy might crashloop because of empty --cluster-name . This behavior is caused by a bug in the update logic, where the cluster name is not propagated to the audit-proxy pod / container manifest.

Workaround:

For a control plane v2 user cluster with enableControlplaneV2: true , connect to the user control plane machine using SSH, and update /etc/kubernetes/manifests/audit-proxy.yaml with --cluster_name=USER_CLUSTER_NAME .

For a control plane v1 user cluster, edit the audit-proxy container in the kube-apiserver statefulset to add --cluster_name=USER_CLUSTER_NAME :

kubectl  
edit  
statefulset  
kube-apiserver  
-n  
 USER_CLUSTER_NAME 
  
--kubeconfig = 
 ADMIN_CLUSTER_KUBECONFIG 

An additional control plane redeployment right after gkectl upgrade cluster

Right after gkectl upgrade cluster , the control plane pods might be re-deployed again. The cluster state from gkectl list clusters change from RUNNING TO RECONCILING . Requests to the user cluster might timeout.

This behavior is because of the control plane certificate rotation happens automatically after gkectl upgrade cluster .

This issue only happens to user clusters that do NOT use control plane v2.

Workaround:

Wait for the cluster state to change back to RUNNING again in gkectl list clusters , or upgrade to versions with the fix: 1.13.6+, 1.14.2+ or 1.15+.

Bad release 1.12.7-gke.19 has been removed

Google Distributed Cloud 1.12.7-gke.19 is a bad release and you should not use it. The artifacts have been removed from the Cloud Storage bucket.

Workaround:

Use the 1.12.7-gke.20 release instead.

gke-connect-agent continues to use the older image after registry credential updated

If you update the registry credential using one of the following methods:

• gkectl update credentials componentaccess if not using private registry
• gkectl update credentials privateregistry if using private registry

you might find gke-connect-agent continues to use the older image or the gke-connect-agent pods cannot be pulled up due to ImagePullBackOff .

This issue will be fixed in Google Distributed Cloud releases 1.13.8, 1.14.4, and subsequent releases.

Workaround:

Option 1 : Redeploy gke-connect-agent manually:

1. Delete the gke-connect namespace:

   kubectl  
   --kubeconfig = 
    KUBECONFIG 
     
   delete  
   namespace  
   gke-connect

2. Redeploy gke-connect-agent with the original register service account key (no need to update the key):For admin cluster:

   gkectl  
   update  
   credentials  
   register  
   --kubeconfig = 
    ADMIN_CLUSTER_KUBECONFIG 
     
   --config  
    ADMIN_CLUSTER_CONFIG_FILE 
     
   --admin-cluster

   For user cluster:

   gkectl  
   update  
   credentials  
   register  
   --kubeconfig = 
    ADMIN_CLUSTER_KUBECONFIG 
     
   --config  
    USER_CLUSTER_CONFIG_FILE 
   

Option 2 : You can manually change the data of the image pull secret regcred which is used by gke-connect-agent deployment:

kubectl  
--kubeconfig = 
 KUBECONFIG 
  
-n = 
gke-connect  
patch  
secrets  
regcred  
-p  
 "{\"data\":{\".dockerconfigjson\":\" 
 $( 
kubectl  
--kubeconfig = 
 KUBECONFIG 
  
-n = 
kube-system  
get  
secrets  
private-registry-creds  
-ojsonpath = 
 '{.data.\.dockerconfigjson}' 
 ) 
 \"}}" 

Option 3 : You can add the default image pull secret for your cluster in the gke-connect-agent deployment by:

1. Copy the default secret to gke-connect namespace:

   kubectl  
   --kubeconfig = 
    KUBECONFIG 
     
   -n = 
   kube-system  
   get  
   secret  
   private-registry-creds  
   -oyaml  
    | 
     
   sed  
    's/ namespace: .*/ namespace: gke-connect/' 
     
    | 
     
   kubectl  
   --kubeconfig = 
    KUBECONFIG 
     
   -n = 
   gke-connect  
   apply  
   -f  
   -

2. Get the gke-connect-agent deployment name:

   kubectl  
   --kubeconfig = 
    KUBECONFIG 
     
   -n = 
   gke-connect  
   get  
   deployment  
    | 
     
   grep  
   gke-connect-agent

3. Add the default secret to gke-connect-agent deployment:

   kubectl  
   --kubeconfig = 
    KUBECONFIG 
     
   -n = 
   gke-connect  
   patch  
   deployment  
    DEPLOYMENT_NAME 
     
   -p  
    '{"spec":{"template":{"spec":{"imagePullSecrets": [{"name": "private-registry-creds"}, {"name": "regcred"}]}}}}' 
   

Manual LB configuration check failure

When you validate the configuration before creating a cluster with Manual load balancer by running gkectl check-config , then the command will fail with the following error messages.

  
-  
Validation  
Category:  
Manual  
LB  
Running  
validation  
check  
 for 
  
 "Network 
 configuration" 
...panic:  
runtime  
error:  
invalid  
memory  
address  
or  
nil  
pointer  
dereference

Workaround:

Option 1: You can use the patch version 1.13.7 and 1.14.4 that will include the fix.

Option 2: You can also run the same command to validate the configuration but skip the load balancer validation.

gkectl  
check-config  
--skip-validation-load-balancer

etcd watch starvation

Clusters running etcd version 3.4.13 or earlier may experience watch starvation and non-operational resource watches, which can lead to the following problems:

• Pod scheduling is disrupted
• Nodes are unable to register
• kubelet doesn't observe pod changes

These problems can make the cluster non-functional.

This issue is fixed in Google Distributed Cloud releases 1.12.7, 1.13.6, 1.14.3, and subsequent releases. These newer releases use etcd version 3.4.21. All prior versions of Google Distributed Cloud are affected by this issue.

Workaround

If you can't upgrade immediately, you can mitigate the risk of cluster failure by reducing the number of nodes in your cluster. Remove nodes until the etcd_network_client_grpc_sent_bytes_total metric is less than 300 MBps.

To view this metric in Metrics Explorer:

1. Go to the Metrics Explorer in the Google Cloud console:

   Go to Metrics Explorer

2. Select the Configurationtab.
3. Expand the Select a metric, enter Kubernetes Container in the filter bar, and then use the submenus to select the metric:
   1. In the Active resourcesmenu, select Kubernetes Container.
   2. In the Active metric categoriesmenu, select Anthos.
   3. In the Active metricsmenu, select etcd_network_client_grpc_sent_bytes_total .
   4. Click Apply.

GKE Identity Service can cause control plane latencies

At cluster restarts or upgrades, GKE Identity Service can get overwhelmed with traffic consisting of expired JWT tokens forwarded from the kube-apiserver to GKE Identity Service over the authentication webhook. Although GKE Identity Service doesn't crashloop, it becomes unresponsive and ceases to serve further requests. This problem ultimately leads to higher control plane latencies.

This issue is fixed in the following Google Distributed Cloud releases:

• 1.12.6+
• 1.13.6+
• 1.14.2+

To determine if you're affected by this issue, perform the following steps:

1. Check whether the GKE Identity Service endpoint can be reached externally:

   curl  
   -s  
   -o  
   /dev/null  
   -w  
    "%{http_code}" 
     
    \ 
     
   -X  
   POST  
   https:// CLUSTER_ENDPOINT 
   /api/v1/namespaces/anthos-identity-service/services/https:ais:https/proxy/authenticate  
   -d  
    '{}' 
   

   Replace CLUSTER_ENDPOINT with the control plane VIP and control plane load balancer port for your cluster (for example, 172.16.20.50:443 ).

   If you're affected by this issue, the command returns a 400 status code. If the request times out, restart the ais Pod and rerun the curl command to see if that resolves the problem. If you get a status code of 000 , the problem has been resolved and you are done. If you still get a 400 status code, the GKE Identity Service HTTP server isn't starting. In this case, continue.

2. Check the GKE Identity Service and kube-apiserver logs:
   1. Check the GKE Identity Service log:

      kubectl  
      logs  
      -f  
      -l  
      k8s-app = 
      ais  
      -n  
      anthos-identity-service  
       \ 
        
      --kubeconfig  
       KUBECONFIG 
      

      If the log contains an entry like the following, then you are affected by this issue:

      I0811  
       22 
      :32:03.583448  
       32 
        
      authentication_plugin.cc:295 ] 
        
      Stopping  
      OIDC  
      authentication  
       for 
        
      ???.  
      Unable  
      to  
      verify  
      the  
      OIDC  
      ID  
      token:  
      JWT  
      verification  
      failed:  
      The  
      JWT  
      does  
      not  
      appear  
      to  
      be  
      from  
      this  
      identity  
      provider.  
      To  
      match  
      this  
      provider,  
      the  
       'aud' 
        
      claim  
      must  
      contain  
      one  
      of  
      the  
      following  
      audiences:

   2. Check the kube-apiserver logs for your clusters:

      In the following commands, KUBE_APISERVER_POD is the name of the kube-apiserver Pod on the given cluster.

      Admin cluster:

      kubectl  
      --kubeconfig  
       ADMIN_KUBECONFIG 
        
      logs  
       \ 
        
      -n  
      kube-system  
       KUBE_APISERVER_POD 
        
      kube-apiserver

      User cluster:

      kubectl  
      --kubeconfig  
       ADMIN_KUBECONFIG 
        
      logs  
       \ 
        
      -n  
       USER_CLUSTER_NAME 
        
       KUBE_APISERVER_POD 
        
      kube-apiserver

      If the kube-apiserver logs contain entries like the following, then you are affected by this issue:

      E0811  
       22 
      :30:22.656085  
       1 
        
      webhook.go:127 ] 
        
      Failed  
      to  
      make  
      webhook  
      authenticator  
      request:  
      error  
      trying  
      to  
      reach  
      service:  
      net/http:  
      TLS  
      handshake  
      timeout
      E0811  
       22 
      :30:22.656266  
       1 
        
      authentication.go:63 ] 
        
       "Unable to authenticate the request" 
        
       err 
       = 
       "[invalid bearer token, error trying to reach service: net/http: TLS handshake timeout]" 
      

Workaround

If you can't upgrade your clusters immediately to get the fix, you can identify and restart the offending pods as a workaround:

1. Increase the GKE Identity Service verbosity level to 9:

   kubectl  
   patch  
   deployment  
   ais  
   -n  
   anthos-identity-service  
   --type = 
   json  
    \ 
     
   -p = 
    '[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", \ 
    "value":"--vmodule=cloud/identity/hybrid/charon/*=9"}]' 
     
    \ 
     
   --kubeconfig  
    KUBECONFIG 
   

2. Check the GKE Identity Service log for the invalid token context:

   kubectl  
   logs  
   -f  
   -l  
   k8s-app = 
   ais  
   -n  
   anthos-identity-service  
    \ 
     
   --kubeconfig  
    KUBECONFIG 
   

3. To get the token payload associated with each invalid token context, parse each related service account secret with the following command:

   kubectl  
   -n  
   kube-system  
   get  
   secret  
    SA_SECRET 
     
    \ 
     
   --kubeconfig  
    KUBECONFIG 
     
    \ 
     
   -o  
    jsonpath 
    = 
    '{.data.token}' 
     
    | 
     
   base64  
   --decode

4. To decode the token and see the source pod name and namespace, copy the token to the debugger at jwt.io .
5. Restart the pods identified from the tokens.

The memory usage increase issue of etcd-maintenance pods

The etcd maintenance pods that use etcddefrag:gke_master_etcddefrag_20210211.00_p0 image are affected. The `etcddefrag` container opens a new connection to etcd server during each defrag cycle and the old connections are not cleaned up.

Workaround:

Option 1: Upgrade to the latest patch version from 1.8 to 1.11 which contain the fix.

Option 2: If you are using patch version earlier than 1.9.6 and 1.10.3, you need to scale down the etcd-maintenance pod for admin and user cluster:

kubectl  
scale  
--replicas  
 0 
  
deployment/gke-master-etcd-maintenance  
-n  
 USER_CLUSTER_NAME 
  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 
kubectl  
scale  
--replicas  
 0 
  
deployment/gke-master-etcd-maintenance  
-n  
kube-system  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 

Miss the health checks of user cluster control plane pods

Both the cluster health controller and the gkectl diagnose cluster command perform a set of health checks including the pods health checks across namespaces. However, they start to skip the user control plane pods by mistake. If you use the control plane v2 mode, this won't affect your cluster.

Workaround:

This won't affect any workload or cluster management. If you want to check the control plane pods healthiness, you can run the following commands:

kubectl  
get  
pods  
-owide  
-n  
 USER_CLUSTER_NAME 
  
--kubeconfig  
 ADMIN_CLUSTER_KUBECONFIG 

1.6 and 1.7 admin cluster upgrades may be affected by the k8s.gcr.io -> registry.k8s.io redirect

Kubernetes redirected the traffic from k8s.gcr.io to registry.k8s.io on 3/20/2023. In Google Distributed Cloud 1.6.x and 1.7.x, the admin cluster upgrades use the container image k8s.gcr.io/pause:3.2 . If you use a proxy for your admin workstation and the proxy doesn't allow registry.k8s.io and the container image k8s.gcr.io/pause:3.2 is not cached locally, the admin cluster upgrades will fail when pulling the container image.

Workaround:

Add registry.k8s.io to the allowlist of the proxy for your admin workstation.

Seesaw validation failure on load balancer creation

gkectl create loadbalancer fails with the following error message:

-  
Validation  
Category:  
Seesaw  
LB  
-  
 [ 
FAILURE ] 
  
Seesaw  
validation:  
xxx  
cluster  
lb  
health  
check  
failed:  
LB "xxx.xxx.xxx.xxx" 
  
is  
not  
healthy:  
Get  
 "http://xxx.xxx.xxx.xxx:xxx/healthz" 
:  
dial  
tcpxxx.xxx.xxx.xxx:xxx:  
connect:  
no  
route  
to  
host

This is due to the seesaw group file already existing. And the preflight check tries to validate a non-existent seesaw load balancer.

Workaround:

Remove the existing seesaw group file for this cluster. The file name is seesaw-for-gke-admin.yaml for the admin cluster, and seesaw-for-{CLUSTER_NAME}.yaml for a user cluster.

Application timeouts caused by conntrack table insertion failures

Google Distributed Cloud version 1.14 is susceptible to netfilter connection tracking (conntrack) table insertion failures when using Ubuntu or COS operating system images. Insertion failures lead to random application timeouts and can occur even when the conntrack table has room for new entries. The failures are caused by changes in kernel 5.15 and higher that restrict table insertions based on chain length.

To see if you are affected by this issue, you can check the in-kernel connection tracking system statistics on each node with the following command:

sudo  
conntrack  
-S

The response looks like this:

 cpu 
 = 
 0 
  
 found 
 = 
 0 
  
 invalid 
 = 
 4 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 0 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 0 
  
 chaintoolong 
 = 
 0 
  
 cpu 
 = 
 1 
  
 found 
 = 
 0 
  
 invalid 
 = 
 0 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 0 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 0 
  
 chaintoolong 
 = 
 0 
  
 cpu 
 = 
 2 
  
 found 
 = 
 0 
  
 invalid 
 = 
 16 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 0 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 0 
  
 chaintoolong 
 = 
 0 
  
 cpu 
 = 
 3 
  
 found 
 = 
 0 
  
 invalid 
 = 
 13 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 0 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 0 
  
 chaintoolong 
 = 
 0 
  
 cpu 
 = 
 4 
  
 found 
 = 
 0 
  
 invalid 
 = 
 9 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 0 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 0 
  
 chaintoolong 
 = 
 0 
  
 cpu 
 = 
 5 
  
 found 
 = 
 0 
  
 invalid 
 = 
 1 
  
 insert 
 = 
 0 
  
 insert_failed 
 = 
 0 
  
 drop 
 = 
 0 
  
 early_drop 
 = 
 0 
  
 error 
 = 
 519 
  
 search_restart 
 = 
 0 
  
 clash_resolve 
 = 
 126 
  
 chaintoolong 
 = 
 0 
  
...

If a chaintoolong value in the response is a non-zero number, you're affected by this issue.

Workaround

The short term mitigation is to increase the size of both the netfiler hash table ( nf_conntrack_buckets ) and the netfilter connection tracking table ( nf_conntrack_max ). Use the following commands on each cluster node to increase the size of the tables:

sysctl  
-w  
net.netfilter.nf_conntrack_buckets = 
 TABLE_SIZE 
sysctl  
-w  
net.netfilter.nf_conntrack_max = 
 TABLE_SIZE