Logging and monitoring

Google Distributed Cloud includes multiple options for cluster logging and monitoring, including cloud-based managed services, open source tools, and validated compatibility with third-party commercial solutions. This document explains these options and provides some basic guidance on selecting the proper solution for your environment.

Options for Google Distributed Cloud

You have several logging and monitoring options for your Google Distributed Cloud:

Cloud Logging and Cloud Monitoring

Google Cloud Observability (formerly Stackdriver) is the built-in observability solution for Google Cloud. It offers a fully managed logging solution, metrics collection, monitoring, dashboarding, and alerting. Cloud Monitoring monitors Google Distributed Cloud clusters in a similar way as cloud-based GKE clusters.

You can configure the in-cluster agents for the scope of monitoring and logging, as well as the level of metrics collected:

  • Scope of logging and monitoring can be set to system components only (the default) or for system components and applications
  • Level of metrics collected can be configured for an optimized set of metrics or for full metrics

See Configuring logging and monitoring agents for Anthos clusters on VMware on this document for more information.

Cloud Logging and Cloud Monitoring provide an ideal solution for customers wanting a single, easy-to-configure, powerful cloud-based observability solution. We highly recommend Logging and Monitoring when running workloads only on GKE on VMware, or workloads on GKE and GKE on VMware. For applications with components running on GKE on VMware and traditional on-premises infrastructure, you might consider other solutions for an end-to-end view of those applications.

Third-party solutions

Google has worked with several third-party logging and monitoring solution providers to help their products work well with GKE on VMware. These include Datadog, Elastic, and Splunk. Additional validated third parties will be added in the future.

For more information about using third-party solutions with GKE on VMware, see the following:

How logging and monitoring for GKE on VMware works

Logging and monitoring agents are installed and activated in each cluster when you create a new admin or user cluster. The agents collect data about system components—the scope of which you can configure.

To view the collected data on the Google Cloud console, you must configure the Google Cloud project that stores the logs and metrics you want to view.

The logging and monitoring agents on each cluster include:

  • GKE metrics agent( gke-metrics-agent ). A DaemonSet that sends metrics to the Cloud Monitoring API.

  • Log forwarder( stackdriver-log-forwarder ). A Fluent Bit DaemonSet that forwards logs from each machine to Cloud Logging. The log forwarder buffers the log entries on the node locally and resends them for up to four hours. If the buffer gets full or if the log forwarder can't reach the Cloud Logging API for more than four hours, then logs are dropped.

  • Global GKE metrics agent( gke-metrics-agent-global ). A Deployment that sends metrics to the Cloud Monitoring API.

  • Metadata agent( stackdriver-metadata-agent ). A Deployment that sends metadata for Kubernetes resources such as pods, deployments, or nodes to the Stackdriver Resource Metadata API; this data is used to enrich metric queries by enabling you to query by deployment name, node name, or even Kubernetes service name.

  • kube-state-metricsA Deployment that listens to API server and generate metrics about the states of objects.

  • node-exporterA DaemonSet that generates hardware and OS metrics.

You can see all the Deployment agents by running the following command:

kubectl --kubeconfig CLUSTER_KUBECONFIG 
get deployments -l "managed-by=stackdriver" --all-namespaces

where CLUSTER_KUBECONFIG is the path to your kubeconfig file for the cluster.

The output of this command is similar to the following:

gke-metrics-agent-global                      1/1     Running   0   4h31m
stackdriver-metadata-agent-cluster-level      1/1     Running   0   4h31m

You can see all the DaemonSet agents by running the following command:

kubectl --kubeconfig CLUSTER_KUBECONFIG 
get daemonsets -l "managed-by=stackdriver" --all-namespaces

The output of this command is similar to the following:

gke-metrics-agent                             1/1     Running   0   4h31m
stackdriver-log-forwarder                     1/1     Running   0   4h31m

Configuring logging and monitoring agents for GKE on VMware

The agents installed with GKE on VMware collect data about system components, subject to your settings and configuration, for the purposes of maintaining and troubleshooting issues with your clusters.

System components only (default scope)

Upon installation, agents collect logs and metrics, including performance details (for example, CPU and memory utilization) and similar metadata, for Google-provided system components. These include all workloads in the admin cluster, and for user clusters, workloads in the kube-system, gke-system, gke-connect, istio-system, and config-management-system namespaces. You can configure or disable the agents as described in the following sections.

The scope of logs and metrics collected can be expanded to include applications as well. For instructions to enable application logging and monitoring, see Enabling Logging and Monitoring for user applications .

Optimized metrics (default metrics)

By default, the metrics agents running in the cluster collect and report an optimized set of container, kubelet and kube-state-metrics metrics to Google Cloud Observability (formerly Stackdriver).

Fewer resources are needed to collect this optimized set of metrics, which improves overall performance and scalability. This is especially important for container-level and kube-level metrics, due to the large quantity of objects to monitor.

Excluded container metrics

The following container metrics are excluded from the optimized metrics:

  • container_cpu_cfs_periods_total
  • container_cpu_cfs_throttled_periods_total
  • container_cpu_load_average_10s
  • container_cpu_system_seconds_total
  • container_cpu_user_seconds_total
  • container_fs_io_current
  • container_fs_io_time_seconds_total
  • container_fs_io_time_weighted_seconds_total
  • container_fs_read_seconds_total
  • container_fs_reads_bytes_total
  • container_fs_reads_merged_total
  • container_fs_reads_total
  • container_fs_sector_reads_total
  • container_fs_sector_writes_total
  • container_fs_write_seconds_total
  • container_fs_writes_bytes_total
  • container_fs_writes_merged_total
  • container_fs_writes_total
  • container_last_seen
  • container_memory_cache
  • container_memory_failcnt
  • container_memory_mapped_file
  • container_memory_max_usage_bytes
  • container_memory_swap
  • container_network_receive_packets_dropped_total
  • container_network_receive_packets_total
  • container_network_transmit_packets_dropped_total
  • container_network_transmit_packets_total
  • container_start_time_seconds
  • container_spec_cpu_period
  • container_spec_cpu_quota
  • container_spec_cpu_shares
  • container_spec_memory_limit_bytes
  • container_spec_memory_reservation_limit_bytes
  • container_spec_memory_swap_limit_bytes
  • container_start_time_seconds
  • container_tasks_state

The complete set of GKE on VMware metrics is documented in GKE Enterprise metrics .