Known issues

  $ 
  
 gkectl 
  
 upgrade 
  
 cluster 
  
 -- 
 kubeconfig 
  
 kubeconfig 
  
 -- 
 config 
  
 user 
 - 
 cluster 
 . 
 yaml 
 … 
 Upgrading 
  
 to 
  
 bundle 
  
 version 
 : 
  
 "1.7.1-gke.4" 
 … 
 Exit 
  
 with 
  
 error 
 : 
 failed 
  
 to 
  
 register 
  
 to 
  
 GCP 
 , 
  
 gcloud 
  
 output 
 : 
  
 , 
  
 error 
 : 
  
 error 
  
 running 
  
 command 
  
 'gcloud alpha container hub memberships register foo-cluster --kubeconfig kubeconfig --context cluster --version 20210129-01-00  --enable-workload-identity --has-private-issuer --verbosity=error --quiet' 
 : 
  
 error 
 : 
  
 exit 
  
 status 
  
 1 
 , 
  
 stderr 
 : 
  
 'Waiting for membership to be created... 
 

  ● 
  
 systemd 
 - 
 timesyncd 
 . 
 service 
  
 - 
  
 Network 
  
 Time 
  
 Synchronization 
 Loaded 
 : 
  
 loaded 
  
 ( 
 / 
 lib 
 / 
 systemd 
 / 
 system 
 / 
 systemd 
 - 
 timesyncd 
 . 
 service 
 ; 
  
 enabled 
 ; 
  
 vendor 
  
 preset 
 : 
  
 enabled 
 ) 
 Active 
 : 
  
 inactive 
  
 ( 
 dead 
 ) 
 

  apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 DaemonSet 
 metadata 
 : 
  
 name 
 : 
  
 ensure-systemd-timesyncd 
 spec 
 : 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 name 
 : 
  
 ensure-systemd-timesyncd 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 name 
 : 
  
 ensure-systemd-timesyncd 
  
 spec 
 : 
  
 hostIPC 
 : 
  
 true 
  
 hostPID 
 : 
  
 true 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 ensure-systemd-timesyncd 
  
 # Use your preferred image. 
  
 image 
 : 
  
 ubuntu 
  
 command 
 : 
  
 - 
  
 /bin/bash 
  
 - 
  
 -c 
  
 - 
  
 | 
  
 while true; do 
  
 echo $(date -u) 
  
 echo "Checking systemd-timesyncd status..." 
  
 chroot /host systemctl status systemd-timesyncd 
  
 if (( $? != 0 )) ; then 
  
 echo "Restarting systemd-timesyncd..." 
  
 chroot /host systemctl start systemd-timesyncd 
  
 else 
  
 echo "systemd-timesyncd is running." 
  
 fi; 
  
 sleep 60 
  
 done 
  
 volumeMounts 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 mountPath 
 : 
  
 /host 
  
 resources 
 : 
  
 requests 
 : 
  
 memory 
 : 
  
 "10Mi" 
  
 cpu 
 : 
  
 "10m" 
  
 securityContext 
 : 
  
 privileged 
 : 
  
 true 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 host 
  
 hostPath 
 : 
  
 path 
 : 
  
 / 
 ```` 
 ## ClientConfig custom resource 
 ` 
 gkectl update` reverts any manual changes that you have made to the ClientConfig 
 custom resource. We strongly recommend that you back up the ClientConfig 
 resource after every manual change. 
 ## `kubectl describe CSINode` and `gkectl diagnose snapshot` 
 ` 
 kubectl describe CSINode` and `gkectl diagnose snapshot` sometimes fail due to 
 the 
 [ 
 OSS Kubernetes issue 
 ] 
 (https://github.com/kubernetes/kubectl/issues/848){:.external} 
  
 on dereferencing nil pointer fields. 
 ## OIDC and the CA certificate 
 The OIDC provider doesn't use the common CA by default. You must explicitly 
 supply the CA certificate. 
 Upgrading the admin cluster from 1.5 to 1.6.0 breaks 1.5 user clusters that use 
 an OIDC provider and have no value for `authentication.oidc.capath` in the 
 [ 
 user cluster configuration file 
 ] 
 (/anthos/clusters/docs/on-prem/1.7/how-to/user-cluster-configuration-file). 
 To work around this issue, run the following script 
 : 
< section><pre class="devsite-click-to-copy"> 
 USER_CLUSTER_KUBECONFIG=<var class="edit">YOUR_USER_CLUSTER_KUBECONFIG</var> 
 IDENTITY_PROVIDER=<var class="edit">YOUR_OIDC_PROVIDER_ADDRESS</var> 
 openssl s_client -showcerts -verify 5 -connect $IDENTITY_PROVIDER:443 < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){i++}; out="tmpcert"i".pem"; print >out}' 
 ROOT_CA_ISSUED_CERT=$(ls tmpcert*.pem | tail -1) 
 ROOT_CA_CERT="/etc/ssl/certs/$(openssl x509 -in $ROOT_CA_ISSUED_CERT -noout -issuer_hash).0" 
 cat tmpcert*.pem $ROOT_CA_CERT > certchain.pem CERT=$(echo $(base64 certchain.pem) | sed 's\ \\g') rm tmpcert1.pem tmpcert2.pem 
 kubectl --kubeconfig $USER_CLUSTER_KUBECONFIG patch clientconfig default -n kube-public --type json -p "[{ \"op\" 
 : 
  
 \"replace\", \"path\" 
 : 
  
 \"/spec/authentication/0/oidc/certificateAuthorityData\", \"value\":\"${CERT}\"}]" 
< /pre></section> 
 Replace the following 
 : 
 * <var>YOUR_OIDC_IDENTITY_PROVICER</var>: The address of your OIDC provider 
 : 
 * <var>YOUR_YOUR_USER_CLUSTER_KUBECONFIG</var> 
 : 
  
 The path of your user cluster 
  
 kubeconfig file. 
 ## gkectl check-config</code> validation fails: can't find F5 BIG-IP partitions 
< dl 
>
< dt>Symptoms</dt> 
< dd><p>Validation fails because F5 BIG-IP partitions can't be found, even though they exist.</p></dd> 
< dt>Potential causes</dt> 
< dd><p>An issue with the F5 BIG-IP API can cause validation to fail.</p></dd> 
< dt>Resolution</dt> 
< dd><p>Try running <code>gkectl check-config</code> again.</p></dd> 
< /dl 
> ## Disruption for workloads with PodDisruptionBudgets {:#workloads_pdbs_disruption} 
 Upgrading clusters can cause disruption or downtime for workloads that use 
 [ 
 PodDisruptionBudgets 
 ] 
 (https://kubernetes.io/docs/concepts/workloads/pods/disruptions/){:.external} 
  
 (PDBs). 
 ## Nodes fail to complete their upgrade process 
 If you have `PodDisruptionBudget` objects configured that are unable to 
 allow any additional disruptions, node upgrades might fail to upgrade to the 
 control plane version after repeated attempts. To prevent this failure, we 
 recommend that you scale up the `Deployment` or `HorizontalPodAutoscaler` to 
 allow the node to drain while still respecting the `PodDisruptionBudget` 
 configuration. 
 To see all `PodDisruptionBudget` objects that do not allow any disruptions 
 : 
 

docker pull gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440

docker tag gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440 \
    PRIVATE_REGISTRY_HOST/stackdriver-operator:v0.0.440

docker push PRIVATE_REGISTRY_HOST/stackdriver-operator:v0.0.440

docker pull gcr.io/gke-on-prem-release/fluent-bit:v1.6.10-gke.3

docker tag gcr.io/gke-on-prem-release/fluent-bit:v1.6.10-gke.3 \
    PRIVATE_REGISTRY_HOST/fluent-bit:v1.6.10-gke.3

docker push PRIVATE_REGISTRY_HOST/fluent-bit:v1.6.10-gke.3

docker pull gcr.io/gke-on-prem-release/prometheus:2.18.1-gke.0

docker tag gcr.io/gke-on-prem-release/prometheus:2.18.1-gke.0 \
    PRIVATE_REGISTRY_HOST/prometheus:2.18.1-gke.0

docker push PRIVATE_REGISTRY_HOST/prometheus:2.18.1-gke.0

kubectl  --kubeconfig [CLUSTER_KUBECONFIG] 
-n kube-system patch deployment stackdriver-operator -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"stackdriver-operator","image":"gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440"}]}}}}'

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-cleanup
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: fluent-bit-cleanup
  template:
    metadata:
      labels:
        app: fluent-bit-cleanup
    spec:
      containers:
      - name: fluent-bit-cleanup
        image: debian:10-slim
        command: ["bash", "-c"]
        args:
        - |
          rm -rf /var/log/fluent-bit-buffers/
          echo "Fluent Bit local buffer is cleaned up."
          sleep 3600
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        securityContext:
          privileged: true
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - key: node-role.gke.io/observability
        effect: NoSchedule
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

kubectl --kubeconfig [CLUSTER_KUBECONFIG] 
logs -n kube-system -l app=fluent-bit-cleanup | grep "cleaned up" | wc -l

kubectl --kubeconfig [CLUSTER_KUBECONFIG] 
-n kube-system get pods -l app=fluent-bit-cleanup --no-headers | wc -l

kubectl --kubeconfig [CLUSTER_KUBECONFIG] 
-n kube-system delete ds fluent-bit-cleanup

kubectl --kubeconfig [CLUSTER_KUBECONFIG] 
-n kube-system rollout restart ds/stackdriver-log-forwarder