/var/log/audit/is filled with audit logs. You can check the disk
usage by runningsudo du -h -d 1 /var/log/audit.
Cause
Since Anthos v1.8, the Ubuntu image is hardened with CIS Level2 Benchmark. And
one of the compliance rules,4.1.2.2 Ensure audit logs are not automatically deleted,
ensures the auditd settingmax_log_file_action = keep_logs. This results in all the
audit rules kept on the disk.
Workaround
Admin workstation
For the admin workstation, you can manually change the auditd settings to rotate the
logs automatically, and then restart the auditd service:
sed -i 's/max_log_file_action = keep_logs/max_log_file_action = rotate/g' /etc/audit/auditd.conf
sed -i 's/num_logs = .*/num_logs = 250/g' /etc/audit/auditd.conf
systemctl restart auditd
The above setting would make auditd automatically rotate its logs once it has
generated more than 250 files (each with 8M size).
Cluster nodes
For cluster nodes, apply the following DaemonSet to your cluster to prevent potential issues:
apiVersion:apps/v1kind:DaemonSetmetadata:name:change-auditd-log-actionnamespace:kube-systemspec:selector:matchLabels:app:change-auditd-log-actiontemplate:metadata:labels:app:change-auditd-log-actionspec:hostIPC:truehostPID:truecontainers:-name:update-audit-ruleimage:ubuntucommand:["chroot","/host","bash","-c"]args:-|whiletrue;doif$(grep-q"max_log_file_action = keep_logs"/etc/audit/auditd.conf);thenecho"updating auditd max_log_file_action to rotate with a max of 250 files"sed-i's/max_log_file_action = keep_logs/max_log_file_action = rotate/g'/etc/audit/auditd.confsed-i's/num_logs = .*/num_logs = 250/g'/etc/audit/auditd.confecho"restarting auditd"systemctlrestartauditdelseecho"auditd setting is expected, skip update"fisleep600donevolumeMounts:-name:hostmountPath:/hostsecurityContext:privileged:truevolumes:-name:hosthostPath:path:/
Note that making this auditd config change would violate CIS Level2 rule4.1.2.2 Ensure audit logs are not automatically deleted.
User cluster upgrade/update fails due to 'failed to register user cluster'
Category
Upgrade, Update
Identified Versions
1.7.0+, 1.8.0+
Symptoms
Rungkectl diagnose clusterwhen a previousgkectlcommand timed out in the following cases.
Upgrading user clusters with GKE connect enabled to 1.8 versions.
Runninggkectl update clusteron 1.8 user clusters with GKE connect enabled.
Runninggkectl update clusterto enable GKE connect on 1.8 user clusters.
Note that the functionality of GKE connect should not be affected. In other words, if GKE connect was functional before the command, it should remain functional.
Cause
TheConnect Agentversion20210514-00-00used in 1.8 versions is out of support.
Workaround
Please contact Google support to mitigate the issue.
systemd-timesyncd not running after reboot on Ubuntu Node
Category
OS
Identified Versions
1.7.1-1.7.5, 1.8.0-1.8.4, 1.9.0+
Symptoms
systemctl status systemd-timesyncdshould show that the service is dead:
chronywas incorrectly installed on Ubuntu OS image, and there's conflict
betweenchronyandsystemd-timesyncd, wheresystemd-timesyncdwould become
inactive andchronybecome active everytime Ubuntu VM got rebooted. However,systemd-timesyncdshould be the default ntp client for the VM.
Workaround
Option 1: Manually runrestart systemd-timesyncdevery time when VM got rebooted.
Option 2: Deploy the following Daemonset so thatsystemd-timesyncdwill always
be restarted if it's dead.
apiVersion:apps/v1kind:DaemonSetmetadata:name:ensure-systemd-timesyncdspec:selector:matchLabels:name:ensure-systemd-timesyncdtemplate:metadata:labels:name:ensure-systemd-timesyncdspec:hostIPC:truehostPID:truecontainers:-name:ensure-systemd-timesyncd# Use your preferred image.image:ubuntucommand:-/bin/bash--c-|while true; doecho $(date -u)echo "Checking systemd-timesyncd status..."chroot /host systemctl status systemd-timesyncdif (( $? != 0 )) ; thenecho "Restarting systemd-timesyncd..."chroot /host systemctl start systemd-timesyncdelseecho "systemd-timesyncd is running."fi;sleep 60donevolumeMounts:-name:hostmountPath:/hostresources:requests:memory:"10Mi"cpu:"10m"securityContext:privileged:truevolumes:-name:hosthostPath:path:/````## ClientConfig custom resource`gkectl update` reverts any manual changes that you have made to the ClientConfigcustom resource. We strongly recommend that you back up the ClientConfigresource after every manual change.## gkectl check-config</code> validation fails: can't find F5 BIG-IP partitions<dl>
<dt>Symptoms</dt><dd><p>Validation fails because F5 BIG-IP partitions can't be found, even though they exist.</p></dd><dt>Potential causes</dt><dd><p>An issue with the F5 BIG-IP API can cause validation to fail.</p></dd><dt>Resolution</dt><dd><p>Try running <code>gkectl check-config</code> again.</p></dd></dl>## Disruption for workloads with PodDisruptionBudgets {:#workloads_pdbs_disruption}Upgrading clusters can cause disruption or downtime for workloads that use[PodDisruptionBudgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/){:.external}(PDBs).## Nodes fail to complete their upgrade processIf you have `PodDisruptionBudget` objects configured that are unable toallow any additional disruptions, node upgrades might fail to upgrade to thecontrol plane version after repeated attempts. To prevent this failure, werecommend that you scale up the `Deployment` or `HorizontalPodAutoscaler` toallow the node to drain while still respecting the `PodDisruptionBudget`configuration.To see all `PodDisruptionBudget` objects that do not allow any disruptions:
kubectl get poddisruptionbudget --all-namespaces -o jsonpath='{range .items[?(@.status.disruptionsAllowed==0)]}{.metadata.name}/{.metadata.namespace}{"\n"}{end}'
```
User cluster installation failed because of cert-manager/ca-injector's leader election issue in Anthos 1.8.2 and 1.8.3
You might see an installation failure due tocert-manager-cainjectorin crashloop, when the apiserver/etcd is slow. The following command,
Second, patch thecert-manager-cainjectorDeployment to disable leader election, which is safe because we only have one replica running. It is not required for a single replica.
# Ensure that we run only 1 cainjector replica, even during rolling updates.
kubectl patch --kubeconfigUSER_CLUSTER_KUBECONFIG-n kube-system deployment cert-manager-cainjector --type=strategic --patch '
spec:
strategy:
rollingUpdate:
maxSurge: 0
'
# Add a command line flag for cainjector: `--leader-elect=false`
kubectl patch --kubeconfigUSER_CLUSTER_KUBECONFIG-n kube-system deployment cert-manager-cainjector --type=json --patch '[
{
"op": "add",
"path": "/spec/template/spec/containers/0/args/-",
"value": "--leader-elect=false"
}
]'
Keepmonitoring-operatorreplicas at 0 as a mitigation until the installation is finished. Otherwise it will revert the change.
After the installation is finished and the cluster is up and running, turn on themonitoring-operatorfor day-2 operations:
Note that after upgrading to 1.8.4 and above (or 1.9.1 and above, if upgrading to 1.9), these steps will no longer be necessary since Anthos will disable leader-election for cainjector.
Until then, if you face this issue during each upgrade, it will be necessary to perform the same mitigation steps again.
Renewal of certificates might be required before an admin cluster upgrade
Before you begin the admin cluster upgrade process, you should make sure that your admin cluster certificates are currently valid, and renew these certificates if they are not.
Admin cluster certificate renewal process
Make sure that OpenSSL is installed on the admin workstation before you begin.
Set theKUBECONFIGvariable:
KUBECONFIG=ABSOLUTE_PATH_ADMIN_CLUSTER_KUBECONFIG
ReplaceABSOLUTE_PATH_ADMIN_CLUSTER_KUBECONFIGwith the absolute path to the admin cluster kubeconfig file.
Get the IP address and SSH keys for the admin master node:
Replaceclient-certificate-dataandclient-key-datain kubeconfig withclient-certificate-dataandclient-key-datain thenew_admin.conffile that you created.
Back up old certificates:
This is an optional, but recommended, step.
# ssh into admin master if you didn't in the previous step
ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}"
# on admin master
sudo tar -czvf backup.tar.gz /etc/kubernetes
logout
# on worker node
sudo scp -i ~/.ssh/admin-cluster.key \
ubuntu@"${MASTER_NODE_IP}":/home/ubuntu/backup.tar.gz .
Renew the certificates with kubeadm:
# ssh into admin master
ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}"
# on admin master
sudo kubeadm alpha certs renew all
Restart static Pods running on the admin master node:
# on admin master
cd /etc/kubernetes
sudo mkdir tempdir
sudo mv manifests/*.yaml tempdir/
sleep 5
echo "remove pods"
# ensure kubelet detect those change remove those pods
# wait until the result of this command is empty
sudo docker ps | grep kube-apiserver
# ensure kubelet start those pods again
echo "start pods again"
sudo mv tempdir/*.yaml manifests/
sleep 30
# ensure kubelet start those pods again
# should show some results
sudo docker ps | grep -e kube-apiserver -e kube-controller-manager -e kube-scheduler -e etcd
# clean up
sudo rm -rf tempdir
logout
Renew the certificates of admin cluster worker nodes
Check node certificates expiration date
kubectl get nodes -o wide
# find the oldest node, fill NODE_IP with the internal ip of that node
ssh -i ~/.ssh/admin-cluster.key ubuntu@"${NODE_IP}"
openssl x509 -enddate -noout -in /var/lib/kubelet/pki/kubelet-client-current.pem
logout
If the certificate is about to expire, renew node certificates bymanual node repair.
You must validate the renewed certificates, and validate the certificate of kube-apiserver.
# Get the IP address of kube-apiserver
cat $KUBECONFIG | grep server
# Get the current kube-apiserver certificate
openssl s_client -showcerts -connect: | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > current-kube-apiserver.crt
# check expiration date of this cert
openssl x509 -in current-kube-apiserver.crt -noout -enddate
/etc/cron.daily/aide script uses up all space in /run, causing a crashloop in Pods
Starting from Google Distributed Cloud 1.7.2, the Ubuntu OS images are hardened withCIS L1 Server Benchmark.
.
As a result, the cron script/etc/cron.daily/aidehas been installed so that an aide check is scheduled to
ensure the CIS L1 Server rule "1.4.2 Ensure filesystem integrity is regularly checked".
If you see one or more Pods crashlooping on a node, rundf -h /runon the node. If the command output shows 100% space usage,
then you are likely experiencing this issue.
This issue is fixed in version 1.8.1. For the 1.7.2 and 1.8.0 versions, you can resolve this issue manually with either of the following two workarounds:
Periodically remove the log files at/run/aide/cron.daily.old*(recommended).
If you use thegkectl upgrade loadbalancerto attempt to update some parameters of the Seesaw load balancer in version 1.8.0, this will not work in either DHCP or IPAM mode. If your setup includes this configuration, do not upgrade to version 1.8.0, but instead to version 1.8.1 or later.
Cannot log in to admin workstation due to password expiry issue
You might experience this issue if you are using one of the following versions of Google Distributed Cloud.
1.7.2-gke.2
1.7.3-gke.2
1.8.0-gke.21
1.8.0-gke.24
1.8.0-gke.25
1.8.1-gke.7
1.8.2-gke.8
You might get the following error when you attempt to SSH into your Anthos VMs, including the admin workstation, cluster nodes, and Seesaw nodes:
WARNING: Your password has expired.
This error occurs because the ubuntu user password on the VMs has expired. You must manually reset the user password's expiration time to a large value before logging into the VMs.
Prevention of password expiry error
If you are running the affected versions listed above, and the user password hasn't expired yet, you should extend the expiration time before seeing the SSH error.
Run the following command on each Anthos VM:
sudo chage -M 99999 ubuntu
Mitigation of password expiry error
If the user password has already expired and you can't log in to the VMs to extend the expiration time, perform the following mitigation steps for each component.
Admin workstation
Use a temporary VM to perform the following steps. You can create an admin workstation using the1.7.1-gke.4 versionto use as the temporary VM.
Ensure the temporary VM and the admin workstation are in a power off state.
Attach the boot disk of the admin workstation to the temporary VM. The boot disk is the one with the label "Hard disk 1".
Mount the boot disk inside the VM by running these commands. Substitute your own boot disk identifier fordev/sdc1.
sudo mkdir -p /mnt/boot-disk
sudo mount /dev/sdc1 /mnt/boot-disk
Set the ubuntu user expiration date to a large value such as99999days.
sudo chroot /mnt/boot-disk chage -M 99999 ubuntu
Shut down the temporary VM.
Power on the admin workstation. You should now be able to SSH as usual.
After you run this command, wait for the user cluster control plane VMs to finish recreation and to be ready before you continue with the next steps.
User cluster worker VMs
Run the following command from the admin workstation to recreate the VMs.
for md in `kubectl --kubeconfig=USER_CLUSTER_KUBECONFIGget machinedeployments -l set=node -o name`; do kubectl patch --kubeconfig=USER_CLUSTER_KUBECONFIG$md --type=json -p="[{'op': 'add', 'path': '/spec/template/spec/metadata/annotations', 'value': {"kubectl.kubernetes.io/restartedAt": "version1"}}]"; done
Seesaw VMs
Run the following commands from the admin workstation to recreate the Seesaw VMs. There will be some downtime. If HA is enabled for the load balancer, the maximum down time is two seconds.
Restarting or upgrading vCenter for versions lower than 7.0U2
If the vCenter, for versions lower than 7.0U2, is restarted, after an upgrade or otherwise,
the network name in VM Information from vCenter is incorrect, and results in the machine being in anUnavailablestate. This eventually leads to the nodes being auto-repaired to create new ones.
1. The issue is fixed in vCenter versions 7.0U2 and above.
2. For lower versions:
Right-click the host, and then select Connection > Disconnect. Next, reconnect, which forces an update of the
VM's portgroup.
In versions 1.8.0-1.8.3, thegkectl create-config admin/clustercommand panics with the messagepanic: invalid version: "latest".
As a workaround, usegkectl create-config admin/cluster --gke-on-prem-version=DESIRED_CLUSTER_VERSION. ReplaceDESIRED_CLUSTER_VERSIONwith the desired version, such as 1.8.2-gke.8.
Creating/upgrading admin cluster timeout
This issue affects 1.8.0-1.8.3.
Your admin cluster creation or admin cluster upgrade might time out with the following error:
Error getting kubeconfig: error running remote command 'sudo cat /etc/kubernetes/admin.conf': error: Process exited with status 1, stderr: 'cat: /etc/kubernetes/admin.conf: No such file or directory
In addition, the log atnodes/ADMIN_MASTER_NODE/files/var/log/startup.login the external cluster snapshot ends with this message:
This error happens when the network is slow between the admin control-plane VM and the container registry. Make sure to inspect your network or proxy setup to reduce the latency and increase the bandwidth.
SSH connection closed by remote host
For Google Distributed Cloud version 1.7.2 and above, the Ubuntu OS images are hardened withCIS L1 Server Benchmark.
To meet the CIS rule "5.2.16 Ensure SSH Idle Timeout Interval is configured",/etc/ssh/sshd_confighas the following settings:
ClientAliveInterval 300
ClientAliveCountMax 0
The purpose of these settings is to terminate a client session after 5 minutes of idle time. However, theClientAliveCountMax 0value causes
unexpected behavior. When you use the ssh session on the admin workstation, or a cluster node, the SSH connection might be disconnected
even your ssh client is not idle, such as when running a time-consuming command, and your command could get terminated with the following message:
Connection to [IP] closed by remote host.
Connection to [IP] closed.
As a workaround, you can either:
Usenohupto prevent your command being terminated on SSH disconnection,
Conflict withcert-managerwhen upgrading to version 1.8.2 or above
If you have your owncert-managerinstallation with Google Distributed Cloud, you might experience a failure when you attempt to upgrade to versions 1.8.2 or above. This is a result of a conflict between your version ofcert-manager, which is likely installed in thecert-managernamespace, and themonitoring-operatorversion.
If you try to install another copy ofcert-managerafter upgrading to Google Distributed Cloud version 1.8.2 or above, the installation might fail due to a conflict with the existing one managed bymonitoring-operator.
Themetrics-cacluster issuer, which control-plane and observability components rely on for creation and rotation of cert secrets, requires ametrics-cacert secret to be stored in the cluster resource namespace. This namespace iskube-systemfor the monitoring-operator installation, and likely to becert-managerfor your installation.
If you have experienced an installation failure, follow these steps to upgrade successfully to version 1.8.2 or later:
Avoid conflicts during upgrade
Uninstall your version ofcert-manager. If you defined your own resources, you may want tobackupthem.
Copy themetrics-cacert-manager.io/v1 Certificate and themetrics-pki.cluster.localIssuer resources fromkube-systemto the cluster resource namespace of your installed cert-manager. Your installed cert-manager namespace iscert-managerif using theupstream default cert-manager installation, but that depends on your installation.
In general, you shouldn't need to re-install cert-manager in admin clusters because admin clusters only run Google Distributed Cloud control plane workloads. In the rare cases that you also need to install your own cert-manager in admin clusters, please follow the following instructions to avoid conflicts. Please note, if you are anApigeecustomer and you only need cert-manager forApigee, you do not need to run the admin cluster commands.
Copy themetrics-cacert-manager.io/v1 Certificate and themetrics-pki.cluster.localIssuer resources fromkube-systemto the cluster resource namespace of your installed cert-manager. Your installed cert-manager namespace iscert-managerif using theupstream default cert-manager installation, but that depends on your installation.
False positives in docker, containerd, and runc vulnerability scanning
The docker, containerd, and runc in the Ubuntu OS images shipped with
Google Distributed Cloud are pinned to special versions usingUbuntu PPA.
This ensures that any container runtime changes will be qualified by Google Distributed Cloud before each release.
However, the special versions are unknown to theUbuntu CVE Tracker,
which is used as the vulnerability feeds by various CVE scanning tools. Therefore, you will see false positives in docker, containerd, and runc
vulnerability scanning results.
For example, you might see the following false positives from your CVE scanning
results. These CVEs are already fixed in the latest patch versions of Google Distributed Cloud.
Starting from Google Distributed Cloud version 1.7.2, the Ubuntu OS images are hardened withCIS L1 Server Benchmark.
As a result, the cron script/etc/cron.daily/aidehas been installed so that anaidecheck is scheduled so as to
ensure that the CIS L1 Server rule "1.4.2 Ensure filesystem integrity is regularly checked" is followed.
The cron job runs daily at 6:25 AM UTC. Depending on the number of files on the filesystem,
you may experience CPU and memory usage spikes around that time that are caused by thisaideprocess.
If the spikes are affecting your workload, you can disable the daily cron job:
`sudo chmod -x /etc/cron.daily/aide`.
Cisco ACI doesn't work with Direct Server Return (DSR)
Seesaw runs in DSR mode, and by default it doesn't work in Cisco ACI because of data-plane IP learning. A possible workaround is to disable IP learning by adding the Seesaw IP address as a L4-L7 Virtual IP in the Cisco Application Policy Infrastructure Controller (APIC).
You can configure the L4-L7 Virtual IP option by going toTenant > Application Profiles > Application EPGsoruSeg EPGs. Failure to disable IP learning will result in IP endpoint flapping between different locations in the Cisco API fabric.
A service account bearer token that is too long can break Seesaw load balancer logs
If your logging-monitoring service account bearer token is larger than 512 KB, it can break theSeesaw load balancer logs. To fix this issue, upgrade to version 1.9 or later.
Connectivity issues between Pods due toanetddaemons in software deadlock
Clusters withenableDataplaneV2set totruecan experience connectivity issues between Pods due toanetddaemons (running as a Daemonset) entering a software deadlock. While in this state,anetddaemons will see stale nodes (previously deleted nodes) as peers and miss newly added nodes as new peers.
If you have experienced this issue, complete the following steps to restart theanetddaemons to refresh the peer nodes, and connectivity should be restored.
Find allanetddaemons in the cluster:
kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG-n kube-system get pods -o wide | grep anetd
Check whetheranetddaemons currently see stale peers:
kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG-n kube-system exec -itANETD_XYZ-- cilium-health status
If your work station does not have access to user cluster worker nodes, it will get the following failures when runninggkectl diagnose, it is safe to ignore them.
Checking user cluster certificates...FAILURE
Reason: 3 user cluster certificates error(s).
Unhealthy Resources:
Node kubelet CA and certificate on node xxx: failed to verify kubelet certificate on node xxx: dial tcp xxx.xxx.xxx.xxx:10250: connect: connection timed out
Node kubelet CA and certificate on node xxx: failed to verify kubelet certificate on node xxx: dial tcp xxx.xxx.xxx.xxx:10250: connect: connection timed out
Node kubelet CA and certificate on node xxx: failed to verify kubelet certificate on node xxx: dial tcp xxx.xxx.xxx.xxx:10250: connect: connection timed out
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[],[],null,[]]