Health checks are a way to test and monitor the operation of your existing
clusters. Health checks run on their own, periodically. You can also use bmctl
to run health checks on demand. This document describes each check, in what
circumstances it runs automatically, how and when to run it manually, and how to
interpret results.
What's checked?
There are two categories of Distributed Cloud health checks:
-
Node machine checks
-
Cluster-wide checks
The following sections outline what gets checked for each category. These checks are used for both periodic and on-demand health checks.
Node machine checks
This section describes what's evaluated by health checks for node machines. These checks confirm that node machines are configured properly and that they have sufficient resources and connectivity for cluster creation, cluster upgrades, and cluster operation.
These checks correspond to the Bare Metal HealthCheck
custom resources named bm-system- NODE_IP_ADDRESS
-machine
(for example, bm-system-192.0.2.54-machine
) that run in the admin cluster in the cluster
namespace. For more information about health check resources, see HealthCheck
custom resources
.
Common machine checks consist of the following:
-
Cluster machines are using a supported operating system (OS).
-
OS version is supported.
-
OS is using a supported kernel version.
-
Kernel has the BPF Just In Time (JIT) compiler option enabled (
CONFIG_BPF_JIT=y). -
For Ubuntu, Uncomplicated Firewall (UFW) is disabled.
-
Node machines meet the minimum CPU requirements.
-
Node machines have more than 20% of CPU resources available.
-
Node machines meet the minimum memory requirements.
-
Node machines meet the minimum disk storage requirements.
-
Time synchronization is configured on node machines.
-
Default route for routing packets to the default gateway is present in nodes.
-
Domain Name System (DNS) is functional (this check is skipped if the cluster is configured to run behind a proxy).
-
If the cluster is configured to use a registry mirror, the registry mirror is reachable.
Machine Google Cloud checks consist of the following:
-
Container Registry,
gcr.iois reachable (this check is skipped if the cluster is configured to use a registry mirror). -
Google APIs are reachable.
Machine health checks consist of the following:
-
kubeletis active and running on node machines. -
containerdis active and running on node machines. -
Container Network Interface (CNI) health endpoint status is healthy.
-
Pod CIDRs don't overlap with node machine IP addresses.
For more information about the node machine requirements, see Cluster node machine prerequisites .
Cluster-wide checks
This section describes what's evaluated by health checks for a cluster.
Network checks
The following client-side cluster node network checks run automatically as part
of periodic health checks. Network checks can't be run on-demand. These checks
correspond to the Bare Metal HealthCheck
custom resources named bm-system-network
that run in the admin cluster in the cluster namespace. For
more information about health check resources, see HealthCheck
custom
resources
.
- If the cluster uses bundled load balancing, nodes in the load balancing node pool must have Layer 2 address resolution protocol (ARP) connectivity. ARP is required for VIP discovery.
For information about protocols and port usage for your Google Distributed Cloud clusters, see Network requirements .
The network checks for a preflight check differ from the network health checks. For a list of the network checks for a preflight check, see either Preflight checks for cluster creation or Preflight checks for cluster upgrades .
Kubernetes
Kubernetes checks, which run automatically as part of preflight and periodic health checks, can also be run on-demand . These health checks don't return an error if any of the listed control plane components are missing. The check only returns errors if the components exist and have errors at command-execution time.
These checks correspond to the Bare Metal HealthCheck
custom resources named bm-system-kubernetes
resources running in the admin cluster in the cluster
namespace. For more information about health check resources, see HealthCheck
custom resources
.
-
API server is functioning.
-
The
anetdoperator is configured correctly. -
All control plane nodes are operable.
-
The following control plane components are functioning properly:
-
anthos-cluster-operator -
controller-manager -
cluster-api-provider -
ais -
capi-kubeadm-bootstrap-system -
cert-manager -
kube-dns
-
Add-ons
Add-ons checks run automatically as part of preflight checks and periodic health checks and can be run on-demand . This health check doesn't return an error if any of the listed add-ons are missing. The check only returns errors if the add-ons exist and have errors at command-execution time.
These checks correspond to Bare Metal HealthCheck
custom resources named bm-system-add-ons
resources running in the admin cluster in the cluster
namespace. For more information about health check resources, see HealthCheck
custom resources
.
-
Cloud Logging Stackdriver components and Connect Agent are operable:
-
stackdriver-log-aggregator -
stackdriver-log-forwarder -
stackdriver-metadata-agent -
stackdriver-prometheus-k8 -
gke-connect-agent
-
HealthCheck
custom resources
When a health check runs, Distributed Cloud creates a HealthCheck
custom
resource. HealthCheck
custom resources are persistent and provide a structured
record of the health check activities and outcomes. There are two categories of HeathCheck
custom resources:
-
Bare Metal
HealthCheckcustom resources (API Version: baremetal.cluster.gke.io/v1): these resources provide details about periodic health checks. These resources are on the admin cluster in cluster namespaces. Bare MetalHealthCheckresources are responsible for creating health check cron jobs and jobs. These resources are consistently updated with the most recent results. -
Anthos
HealthCheckcustom resources (API Version: anthos.gke.io/v1): these resources are used to report health check metrics. These resources are in thekube-systemnamespace of each cluster. Updates of these resources are best effort . If an update fails to an issue, such as a transient network error, the failure is ignored.
The following table lists the types of resources that are created for either HealthCheck
category:
| Bare Metal HealthChecks | GKE Enterprise HealthChecks | Severity |
|---|---|---|
| Type: machine Name: |
Type: machine Name: |
Critical |
| Type: network Name: |
Type: network Name: |
Critical |
| Type: kubernetes Name: |
Type: kubernetes Name: |
Critical |
| Type: add-ons Name: |
Type: add-ons Name: |
Optional |
To retrieve HealthCheck
status:
-
To read the results of periodic health checks, you can get the associated custom resources:
kubectl get healthchecks.baremetal.cluster.gke.io --kubeconfig ADMIN_KUBECONFIG --all-namespacesReplace
ADMIN_KUBECONFIGwith the path of the admin cluster kubeconfig file.The following sample shows the health checks that run periodically and whether the checks passed when they last ran:
NAMESPACE NAME PASS AGE cluster-test-admin001 bm-system-192.0.2.52-machine true 11d cluster-test-admin001 bm-system-add-ons true 11d cluster-test-admin001 bm-system-kubernetes true 11d cluster-test-admin001 bm-system-network true 11d cluster-test-user001 bm-system-192.0.2.53-machine true 56d cluster-test-user001 bm-system-192.0.2.54-machine true 56d cluster-test-user001 bm-system-add-ons true 56d cluster-test-user001 bm-system-kubernetes true 56d cluster-test-user001 bm-system-network true 56d -
To read details for a specific health check, use
kubectl describe:kubectl describe healthchecks.baremetal.cluster.gke.io HEALTHCHECK_NAME --kubeconfig ADMIN_KUBECONFIG --namespace CLUSTER_NAMESPACEReplace the following:
-
HEALTHCHECK_NAME: the name of the health check. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the cluster.
When you review the resource, the
Status:section contains the following important fields:-
Pass: indicates whether or not the last health check job passed. -
Checks: contains information about the most recent health check job. -
Failures: contains information about the most recent failed job. -
Periodic: contains information such as when was the last time a health check job was scheduled and instrumented.
The following
HealthChecksample is for a successful machine check:Name : bm-system-192.0.2.54-machine Namespace : cluster-test-user001 Labels : baremetal.cluster.gke.io/periodic-health-check=true machine=192.0.2.54 type=machine Annotations : < none > API Version : baremetal.cluster.gke.io/v1 Kind : HealthCheck Metadata : Creation Timestamp : 2023-09-22T18:03:27Z ... Spec : Anthos Bare Metal Version : 1.15.0 Cluster Name : nuc-user001 Interval In Seconds : 3600 Node Addresses : 192.168.1.54 Type : machine Status : Check Image Version : 1.15.0-gke.26 Checks : 192.168.1.54 : Job UID : 345b74a6-ce8c-4300-a2ab-30769ea7f855 Message : Pass : true ... Cluster Spec : Anthos Bare Metal Version : 1.15.0 Bypass Preflight Check : false Cluster Network : Bundled Ingress : true Pods : Cidr Blocks : 10.0.0.0/16 Services : Cidr Blocks : 10.96.0.0/20 ... Conditions : Last Transition Time : 2023-11-22T17:53:18Z Observed Generation : 1 Reason : LastPeriodicHealthCheckFinished Status : False Type : Reconciling Node Pool Specs : node-pool-1 : Cluster Name : nuc-user001 ... Pass : true Periodic : Last Schedule Time : 2023-11-22T17:53:18Z Last Successful Instrumentation Time : 2023-11-22T17:53:18Z Start Time : 2023-09-22T18:03:28Z Events : Type Reason Age From Message ---- ------ ---- ---- ------- Normal HealthCheckJobFinished 6m4s (x2 over 6m4s) healthcheck-controller health check job bm-system-192.0.2.54-machine-28344593 finishedThe following
HealthChecksample is for a failed machine check:Name : bm-system-192.0.2.57-machine Namespace : cluster-user-cluster1 ... API Version : baremetal.cluster.gke.io/v1 Kind : HealthCheck ... Status : Checks : 192.0.2.57 : Job UID : 492af995-3bd5-4441-a950-f4272cb84c83 Message : following checks failed, ['check_kubelet_pass'] Pass : false Failures : Category : AnsibleJobFailed Description: Job : machine-health-check. Details: Target: 1192.0.2.57. View logs with : [ kubectl logs -n cluster-user-test bm-system-192.0.2.57-machine-28303170-qgmhn ] . Reason : following checks failed, ['check_kubelet_pass'] Pass : false Periodic : Last Schedule Time : 2023-10-24T23:04:21Z Last Successful Instrumentation Time : 2023-10-24T23:31:30Z ... -
-
To get a list of health checks for metrics, use the following command:
kubectl get healthchecks.anthos.gke.io --kubeconfig CLUSTER_KUBECONFIG --namespace kube-systemReplace
CLUSTER_KUBECONFIGwith the path of the target cluster kubeconfig file.The following sample shows the response format:
NAMESPACE NAME COMPONENT NAMESPACE STATUS LAST_COMPLETED kube-system bm-system-10.200.0.3-machine Healthy 56m kube-system bm-system-add-ons-add-ons Healthy 48m kube-system bm-system-kubernetes Healthy 57m kube-system bm-system-kubernetes-1.15.1-non-periodic Healthy 25d kube-system bm-system-network Healthy 32m kube-system check-kubernetes-20231114-190445-non-periodic Healthy 3h6m kube-system component-status-controller-manager Healthy 5s kube-system component-status-etcd-0 Healthy 5s kube-system component-status-etcd-1 Healthy 5s kube-system component-status-scheduler Healthy 5s
Health check cron jobs
For periodic health checks, each bare metal HealthCheck
custom resource has a
corresponding CronJob
with the same name. This CronJob
is responsible for scheduling the corresponding
health check to run at set intervals.
To retrieve information about cron jobs:
-
Get a list of cron jobs that have run for a given cluster:
kubectl get cronjobs --kubeconfig ADMIN_KUBECONFIG --namespace CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the cluster.
The following sample shows a typical response:
NAMESPACE NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cluster-test-admin bm-system-10.200.0.3-machine 17 */1 * * * False 0 11m 25d cluster-test-admin bm-system-add-ons 25 */1 * * * False 0 3m16s 25d cluster-test-admin bm-system-kubernetes 16 */1 * * * False 0 12m 25d cluster-test-admin bm-system-network 41 */1 * * * False 0 47m 25dThe values in the
SCHEDULEcolumn indicate the schedule for each health check job run in schedule syntax . For example, thebm-system-kubernetesjob runs at 17 minutes past the hour (17) every hour (*/1) of every day (* * *). The time intervals for periodic health checks aren't editable, but it's useful for troubleshooting to know when they're supposed to run. -
-
Retrieve details for a specific
CronJobcustom resource:kubectl describe cronjob CRONJOB_NAME --kubeconfig ADMIN_KUBECONFIG --namespace CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the cluster.
The following sample shows a successful
CronJob:Name : bm-system-network Namespace : cluster-test-admin Labels : AnthosBareMetalVersion=1.15.1 baremetal.cluster.gke.io/check-name=bm-system-network baremetal.cluster.gke.io/periodic-health-check=true controller-uid=2247b728-f3f5-49c2-86df-9e5ae9505613 type=network Annotations: target : node-network Schedule : 41 */1 * * * Concurrency Policy : Forbid Suspend : False Successful Job History Limit : 1 Failed Job History Limit : 1 Starting Deadline Seconds : < unset > Selector : < unset > Parallelism : < unset > Completions : 1 Active Deadline Seconds : 3600s Pod Template : Labels : baremetal.cluster.gke.io/check-name=bm-system-network Annotations: target : node-network Service Account : ansible-runner Containers : ansible-runner : Image : gcr.io/anthos-baremetal-release/ansible-runner:1.15.1-gke.5 Port : < none > Host Port : < none > Command : cluster Args : -execute-command=network-health-check -login-user=root -controlPlaneLBPort=443 Environment : < none > Mounts : /data/configs from inventory-config-volume (ro) /etc/ssh-key from ssh-key-volume (ro) Volumes : inventory-config-volume : Type : ConfigMap (a volume populated by a ConfigMap) Name : bm-system-network-inventory-bm-system-ne724a7cc3584de0635099 Optional : false ssh-key-volume : Type : Secret (a volume populated by a Secret) SecretName : ssh-key Optional : false Last Schedule Time : Tue, 14 Nov 2023 18:41:00 +0000 Active Jobs : < none > Events : Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 48m cronjob-controller Created job bm-system-network-28333121 Normal SawCompletedJob 47m cronjob-controller Saw completed job : bm-system-network-28333121, status : Complete Normal SuccessfulDelete 47m cronjob-controller Deleted job bm-system-network-28333061 -
Health check logs
When health checks run, they generate logs. Whether you run health checks with bmctl
or they run automatically as part of periodic health checks, logs are
sent to Cloud Logging. When run health checks on demand
,
log files are created in a time-stamped folder in the log/
directory of your
cluster folder on your admin workstation. For example, if you run the bmctl
check kubernetes
command for a cluster named test-cluster
, you find logs
in a directory like bmctl-workspace/test-cluster/log/check-kubernetes-20231103-165923
.
View logs locally
You can use kubectl
to view logs for periodic health checks:
-
Get pods and find the specific health check pod you're interested in:
kubectl get pods --kubeconfig ADMIN_KUBECONFIG --namespace CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the cluster.
The following sample response shows some health check pods:
NAME READY STATUS RESTARTS AGE bm-system-10.200.0.4-machine-28353626-lzx46 0/1 Completed 0 12m bm-system-10.200.0.5-machine-28353611-8vjw2 0/1 Completed 0 27m bm-system-add-ons-28353614-gxt8f 0/1 Completed 0 24m bm-system-check-kernel-gce-user001-02fd2ac273bc18f008192e177x2c 0/1 Completed 0 75m bm-system-cplb-init-10.200.0.4-822aa080-7a2cdd71a351c780bf8chxk 0/1 Completed 0 74m bm-system-cplb-update-10.200.0.4-822aa082147dbd5220b0326905lbtj 0/1 Completed 0 67m bm-system-gcp-check-create-cluster-202311025828f3c13d12f65k2xfj 0/1 Completed 0 77m bm-system-kubernetes-28353604-4tc54 0/1 Completed 0 34m bm-system-kubernetes-check-bm-system-kub140f257ddccb73e32c2mjzn 0/1 Completed 0 63m bm-system-machine-gcp-check-10.200.0.4-6629a970165889accb45mq9z 0/1 Completed 0 77m ... bm-system-network-28353597-cbwk7 0/1 Completed 0 41m bm-system-network-health-check-gce-user05e0d78097af3003dc8xzlbd 0/1 Completed 0 76m bm-system-network-preflight-check-create275a0fdda700cb2b44b264c 0/1 Completed 0 77m -
-
Retrieve pod logs:
kubectl logs POD_NAME --kubeconfig ADMIN_KUBECONFIG --namespace CLUSTER_NAMESPACEReplace the following:
-
POD_NAME: the name of the health check pod. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the cluster.
The following sample shows part of a pod log for a successful node machine health check:
... TASK [Summarize health check] ************************************************** Wednesday 29 November 2023 00:26:22 +0000 (0:00:00.419) 0:00:19.780 **** ok: [10.200.0.4] => { "results": { "check_cgroup_pass": "passed", "check_cni_pass": "passed", "check_containerd_pass": "passed", "check_cpu_pass": "passed", "check_default_route": "passed", "check_disks_pass": "passed", "check_dns_pass": "passed", "check_docker_pass": "passed", "check_gcr_pass": "passed", "check_googleapis_pass": "passed", "check_kernel_version_pass": "passed", "check_kubelet_pass": "passed", "check_memory_pass": "passed", "check_pod_cidr_intersect_pass": "passed", "check_registry_mirror_reachability_pass": "passed", "check_time_sync_pass": "passed", "check_ubuntu_1804_kernel_version": "passed", "check_ufw_pass": "passed", "check_vcpu_pass": "passed" } } ...The following sample shows part of a failed node machine health check pod log. The sample shows that the
kubeletcheck (check_kubelet_pass) failed, indicating that thekubeletisn't running on this node.... TASK [Reach a final verdict] *************************************************** Thursday 02 November 2023 17:30:19 +0000 (0:00:00.172) 0:00:17.218 ***** fatal: [10.200.0.17]: FAILED! => {"changed": false, "msg": "following checks failed, ['check_kubelet_pass']"} ... -
View logs in Cloud Logging
Health check logs are streamed to Cloud Logging and can be viewed in Logs Explorer. Periodic health checks are classed as Pods in the console logs.
-
In the Google Cloud console, go to the Logs Explorerpage in the Loggingmenu.
-
In the Queryfield, enter the following basic query:
resource . type = "k8s_container" resource . labels . pod_name =~ "bm-system.*-machine.*" -
The Query resultswindow shows the logs for node machine health checks.
Here's a list of queries for periodic health checks:
-
Node machine
resource . type = "k8s_container" resource . labels . pod_name =~ "bm-system.*-machine.*" -
Network
resource . type = "k8s_container" resource . labels . pod_name =~ "bm-system-network.*" -
Kubernetes
resource . type = "k8s_container" resource . labels . pod_name =~ "bm-system-kubernetes.*" -
Add-ons
resource . type = "k8s_container" resource . labels . pod_name =~ "bm-system-add-ons.*"
Periodic health checks
By default, the periodic health checks run hourly and check the following cluster components:
You can check the cluster health by looking at the Bare Metal HealthCheck
( healthchecks.baremetal.cluster.gke.io
) custom resources on the admin cluster.
The Network, Kubernetes, and Add-ons checks are cluster-level checks, so there
is a single resource for each check. A Machine check is run for each node in the
target cluster, so there is a resource for each node.
-
To list Bare Metal
HealthCheckresources for a given cluster, run the following command:kubectl get healthchecks.baremetal.cluster.gke.io --kubeconfig = ADMIN_KUBECONFIG \ --namespace = CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the target cluster of the health check.
The following sample response shows the format:
NAMESPACE NAME PASS AGE cluster-test-user001 bm-system-192.0.2.53-machine true 56d cluster-test-user001 bm-system-192.0.2.54-machine true 56d cluster-test-user001 bm-system-add-ons true 56d cluster-test-user001 bm-system-kubernetes true 56d cluster-test-user001 bm-system-network true 56dThe
Passfield forhealthchecks.baremetal.cluster.gke.ioindicates whether the last health check passed (true) or failed (false). -
For more information about checking the status for periodic health checks, see HealthCheck
custom resources
and Health check logs
.
Disable periodic health checks
Periodic health checks are enabled by default on all clusters. You can disable
periodic health checks for a cluster by setting the periodicHealthCheck.enable
field to false
in Cluster resource.
To disable periodic health checks:
-
Edit the cluster configuration file and add the
periodicHealthCheck.enablefield to the Cluster spec and set its value tofalse:apiVersion : v1 kind : Namespace metadata : name : cluster-user-basic --- apiVersion : baremetal.cluster.gke.io/v1 kind : Cluster metadata : name : user-basic namespace : cluster-user-basic spec : type : user profile : default ... periodicHealthCheck : enable : false ... -
Update the cluster by running the
bmctl updatecommand:bmctl update cluster -c CLUSTER_NAME --kubeconfig = ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster you want to update. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
-
To verify that periodic health checks have been disabled, run the following command to confirm that the corresponding
healthchecks.baremetal.cluster.gke.ioresources have been deleted:kubectl get healthchecks.baremetal.cluster.gke.io --kubeconfig = ADMIN_KUBECONFIG \ --namespace = CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the target cluster of the health check.
-
Re-enable periodic health checks
Periodic health checks are enabled by default on all clusters. If you've
disabled periodic health checks, you can re-enable them by setting the periodicHealthCheck.enable
field to true
in Cluster resource.
To re-enable periodic health checks:
-
Edit the cluster configuration file and add the
periodicHealthCheck.enablefield to the Cluster spec and set its value totrue:apiVersion : v1 kind : Namespace metadata : name : cluster-user-basic --- apiVersion : baremetal.cluster.gke.io/v1 kind : Cluster metadata : name : user-basic namespace : cluster-user-basic spec : type : user profile : default ... periodicHealthCheck : enable : true ... -
Update the cluster by running the
bmctl updatecommand:bmctl update cluster -c CLUSTER_NAME --kubeconfig = ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster you want to update. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
-
To verify that periodic health checks are enabled, run the following command to confirm that the corresponding
healthchecks.baremetal.cluster.gke.ioresources are present:kubectl get healthchecks.baremetal.cluster.gke.io --kubeconfig = ADMIN_KUBECONFIG \ --namespace = CLUSTER_NAMESPACEReplace the following:
-
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file. -
CLUSTER_NAMESPACE: the namespace of the target cluster of the health check.
It may take a couple of minutes for the resources to appear.
-
On-demand health checks
The following sections describe the health checks that you can run on demand
with bmctl check
. When you use bmctl check
to run health checks, the
following rules apply:
-
When you check a user cluster with a
bmctl checkcommand, specify the path of the kubeconfig file for the admin cluster with the--kubeconfigflag. -
Logs are generated in a time-stamped directory in the cluster log folder on your admin workstation (by default,
bmctl-workspace/ CLUSTER_NAME /log). -
Health check logs are also sent to Cloud Logging. For more information about the logs, see Health check logs .
For more information about options for bmctl
commands, see bmctl command
reference
.
Add-ons
Check that the specified Kubernetes add-ons for the specified cluster are operable.
-
To check add-ons for a cluster:
bmctl check add-ons --cluster CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that you're checking. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
For a list of what's checked, see Add-ons in the What's checked section of this document.
This check generates log files in a check-addons- TIMESTAMP
directory in the cluster log folder on your admin workstation. Logs are also
sent to Cloud Logging. For more information about the logs, see Health check
logs
.
Cluster
Check all cluster nodes, node networking, Kubernetes, and add-ons for the
specified cluster. You provide the cluster name, and bmctl
looks for the
cluster configuration file at bmctl-workspace/ CLUSTER_NAME
/ CLUSTER_NAME
.yaml
, by default.
-
To check the health of a cluster:
bmctl check cluster --cluster CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that you're checking. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
For a list of what's checked, see the following sections in the What's checked section of this document:
This check generates log files in a check-cluster- TIMESTAMP
directory in the cluster log folder on your admin workstation. Logs are also
sent to Cloud Logging. For more information about the logs, see Health check
logs
.
Config
Check the cluster configuration file. This check expects that you have generated
the configuration file and edited it to specify the cluster configuration
details for your cluster. The purpose of this command is to determine whether
any configuration setting is wrong, missing, or has any syntax errors. You
provide the cluster name, and bmctl
looks for the cluster configuration file
at bmctl-workspace/ CLUSTER_NAME
/ CLUSTER_NAME
.yaml
, by default.
-
To check a cluster configuration file:
bmctl check config --cluster CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that you're checking. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
This command checks the YAML syntax of the cluster configuration file, Google Cloud access, and permissions for the service account specified in the cluster configuration file.
This check generates log files in a check-config- TIMESTAMP
directory in the cluster log folder on your admin workstation. Logs are also
sent to Cloud Logging. For more information about the logs, see Health check
logs
.
GCP
Check that all cluster node machines can access Container Registry ( gcr.io
) and
the Google APIs endpoint ( googleapis.com
).
-
To check the cluster access to required Google Cloud resources:
bmctl check gcp --cluster CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that you're checking. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
This check generates log files in a check-gcp- TIMESTAMP
directory in the cluster log folder on your admin workstation. Logs are also
sent to Cloud Logging. For more information about the logs, see Health check
logs
.
Kubernetes
Check the health of critical Kubernetes operators running in the control plane. This check verifies that critical operators are working properly and that their pods aren't crashing. This health check doesn't return an error if any of the control plane components are missing: it only returns errors if the components exist and have errors at command-execution time.
-
To check the health of Kubernetes components in your cluster:
bmctl check kubernetes --cluster CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that contains the nodes you're checking. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
For a list of what's checked, see Kubernetes in the What's checked section of this document.
This check generates log files in a check-kubernetes- TIMESTAMP
directory in the cluster log folder on your admin workstation. Logs are also
sent to Cloud Logging. For more information about the logs, see Health check
logs
.
Nodes
Check cluster node machines to confirm that they're configured properly and that they have sufficient resources and connectivity for cluster upgrades and cluster operation.
-
To check the health of node machines in your cluster:
bmctl check nodes --cluster CLUSTER_NAME --addresses NODE_IP_ADDRESSES --kubeconfig ADMIN_KUBECONFIGReplace the following:
-
CLUSTER_NAME: the name of the cluster that contains the nodes you're checking. -
NODE_IP_ADDRESSES: a comma-separated list of IP addresses for node machines. -
ADMIN_KUBECONFIG: the path of the admin cluster kubeconfig file.
-
For a list of what's checked, see Node machine checks in the What's checked section of this document.
This check generates log files for each cluster node machine in a check-nodes- TIMESTAMP
directory in the cluster log folder
on your admin workstation. Logs are also sent to Cloud Logging. For more
information about the logs, see Health check logs
.
Preflight
For information about using bmctl
to run preflight checks, see Run on-demand preflight checks for cluster creation
and Run on-demand preflight checks for cluster upgrades
.
VM Runtime preflight check
The VM Runtime on Google Distributed Cloud preflight check validates a set of node machine
prerequisites before using VM Runtime on Google Distributed Cloud and VMs. If
VM Runtime on Google Distributed Cloud preflight check fails, VM creation is blocked. When spec.enabled
is set to true
in the VMRuntime
custom resource, the
VM Runtime on Google Distributed Cloud preflight check runs automatically.
apiVersion
:
vm.cluster.gke.io/v1
kind
:
VMRuntime
metadata
:
name
:
vmruntime
spec
:
enabled
:
true
...
For more information, see VM Runtime on Google Distributed Cloud preflight check .
