Once these steps are done, all the user application logs are sent to Cloud Logging.
The next step is to annotate the user application for metrics collection.
Annotate a user application for metrics collection
To annotate a user application to be scraped and the logs sent to Cloud Monitoring, you must add correspondingannotationsto the metadata for the service, Pod, and endpoints.
In this section, you deploy a sample application with both logs and prometheus-compatible metrics.
Save the following Service and Deployment manifests to a file namedmy-app.yaml. Notice that the Service has the annotationprometheus.io/scrape: "true":
The Pods type metric has a default metric selector for the labels of the target Pods, which is how kube-controller-maneger works. In this example, you cna query the example_monitoring_up metric with a selector of{matchLabels: {app: example-monitoring}}as they are available in the target Pods. Any other selector specified is added to the list. To avoid the default selector, you canremove any labels on the target Pod or use the Object type metric.
Check that the user-defined application metrics are used by HPA
Check that the user defined application metrics are used by HPA:
Name: example-monitoring-hpa
Namespace: default
Labels:Annotations: CreationTimestamp: Mon, 19 Jul 2021 16:00:40 -0800
Reference: Deployment/example-monitoring
Metrics: ( current / target )
"example_monitoring_up" on pods: 1 / 20
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric example_monitoring_up
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Costs
Using custom metrics for HPA does not incur any additional charges. Users are charged only for application metrics and logs. SeeGoogle Cloud's operations suite pricingfor details. The Pod for enabling custom metrics consumes an extra 15m CPU and 20MB memory.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eThis guide outlines the process of configuring user-defined metrics for Horizontal Pod Autoscaling (HPA) in Google Distributed Cloud, enabling application scaling based on custom metrics.\u003c/p\u003e\n"],["\u003cp\u003eTo enable logging and monitoring for user applications, set \u003ccode\u003eenableStackdriverForApplications\u003c/code\u003e and \u003ccode\u003eenableCustomMetricsAdapter\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e within the \u003ccode\u003estackdriver\u003c/code\u003e object configuration.\u003c/p\u003e\n"],["\u003cp\u003eUser applications must be annotated with \u003ccode\u003eprometheus.io/scrape: "true"\u003c/code\u003e in their metadata to enable metrics collection and log sending to Cloud Monitoring.\u003c/p\u003e\n"],["\u003cp\u003eDeploying an HPA object allows the use of custom metrics, such as \u003ccode\u003eexample_monitoring_up\u003c/code\u003e, to define scaling behavior, with no additional charges for using these custom metrics for HPA.\u003c/p\u003e\n"],["\u003cp\u003eUsing custom metrics on HPA requires application metrics and logs to be enabled and the Pod enabling custom metrics will consume an extra 15m CPU and 20MB memory.\u003c/p\u003e\n"]]],[],null,["# Enabling user-defined custom metrics for Horizontal Pod autoscaling\n\n| **Preview**\n|\n|\n| This feature is subject to the \"Pre-GA Offerings Terms\" in the General Service Terms section\n| of the [Service Specific Terms](/terms/service-terms#1).\n|\n| Pre-GA features are available \"as is\" and might have limited support.\n|\n| For more information, see the\n| [launch stage descriptions](/products#product-launch-stages).\n\nThis topic describes how to configure user-defined metrics for Horizontal Pod autoscaling (HPA) in Google Distributed Cloud.\n\nEnabling Logging and Monitoring for user applications\n-----------------------------------------------------\n\nThe configuration for Logging and Monitoring is\nheld in a Stackdriver object named `stackdriver`.\n\n1. Open the `stackdriver` object for editing:\n\n ```\n kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver\n ```\n\n Replace \u003cvar translate=\"no\"\u003eUSER_CLUSTER_KUBECONFIG\u003c/var\u003e with the path of your user\n cluster kubeconfig file.\n2. Under `spec`, set both `enableStackdriverForApplications` and `enableCustomMetricsAdapter` to `true`:\n\n ```\n apiVersion: addons.sigs.k8s.io/v1alpha1\n kind: Stackdriver\n metadata:\n name: stackdriver\n namespace: kube-system\n spec:\n projectID: project-id\n clusterName: cluster-name\n clusterLocation: cluster-location\n proxyConfigSecretName: secret-name\n enableStackdriverForApplications: true\n enableCustomMetricsAdapter: true\n enableVPC: stackdriver-enable-VPC\n optimizedMetrics: true\n ```\n3. Save and close the edited file.\n\nOnce these steps are done, all the user application logs are sent to Cloud Logging.\n\nThe next step is to annotate the user application for metrics collection.\n\nAnnotate a user application for metrics collection\n--------------------------------------------------\n\nTo annotate a user application to be scraped and the logs sent to Cloud Monitoring, you must add corresponding `annotations` to the metadata for the service, Pod, and endpoints.\n\n\u003cbr /\u003e\n\n```\n metadata:\n name: \"example-monitoring\"\n namespace: \"default\"\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/path: \"\" - Overriding metrics path (default \"/metrics\")\n \n```\n\n\u003cbr /\u003e\n\nDeploy an example user application\n----------------------------------\n\nIn this section, you deploy a sample application with both logs and prometheus-compatible metrics.\n\n1. Save the following Service and Deployment manifests to a file named `my-app.yaml`. Notice that the Service has the annotation `prometheus.io/scrape: \"true\"`:\n\n kind: Service\n apiVersion: v1\n metadata:\n name: \"example-monitoring\"\n namespace: \"default\"\n annotations:\n prometheus.io/scrape: \"true\"\n spec:\n selector:\n app: \"example-monitoring\"\n ports:\n - name: http\n port: 9090\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: \"example-monitoring\"\n namespace: \"default\"\n labels:\n app: \"example-monitoring\"\n spec:\n replicas: 1\n selector: \n matchLabels:\n app: \"example-monitoring\"\n template: \n metadata: \n labels:\n app: \"example-monitoring\"\n spec:\n containers:\n - image: gcr.io/google-samples/prometheus-example-exporter:latest\n name: prometheus-example-exporter\n imagePullPolicy: Always\n command:\n - /bin/sh\n - -c\n - ./prometheus-example-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090\n resources:\n requests:\n cpu: 100m\n\n1. Create the Deployment and the Service:\n\n ```\n kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml\n ```\n\nUse the custom metrics in HPA\n-----------------------------\n\nDeploy the HPA object to use the metric exposed in the previous step. See [Autoscaling on multiple metrics and custom metrics](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics) for more advanced information about different type of custom metrics.\n\n```\napiVersion: autoscaling/v2beta2\nkind: HorizontalPodAutoscaler\nmetadata:\n name: example-monitoring-hpa\n namespace: default\nspec:\n scaleTargetRef:\n apiVersion: apps/v1\n kind: Deployment\n name: example-monitoring\n minReplicas: 1\n maxReplicas: 5\n metrics:\n - type: Pods\n pods:\n metric:\n name: example_monitoring_up\n target:\n type: AverageValue\n averageValue: 20\n```\n\nThe Pods type metric has a default metric selector for the labels of the target Pods, which is how kube-controller-maneger works. In this example, you cna query the example_monitoring_up metric with a selector of\n`{matchLabels: {app: example-monitoring}}` as they are available in the target Pods. Any other selector specified is added to the list. To avoid the default selector, you canremove any labels on the target Pod or use the Object type metric.\n\nCheck that the user-defined application metrics are used by HPA\n---------------------------------------------------------------\n\nCheck that the user defined application metrics are used by HPA:\n\n```\nkubectl --kubeconfig=USER_CLUSTER_KUBECONFIG describe hpa example-monitoring-hpa\n```\n\nThe output will look like this:\n\n\u003cbr /\u003e\n\n```\nName: example-monitoring-hpa\nNamespace: default\nLabels: \nAnnotations: CreationTimestamp: Mon, 19 Jul 2021 16:00:40 -0800\nReference: Deployment/example-monitoring\nMetrics: ( current / target )\n \"example_monitoring_up\" on pods: 1 / 20\nMin replicas: 1\nMax replicas: 5\nDeployment pods: 1 current / 1 desired\nConditions:\n Type Status Reason Message\n\n\u003cbr /\u003e\n\n\n\n\n\n*** ** * ** ***\n\n\n\n\nAbleToScale True ReadyForNewScale recommended size matches current size\n ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric example_monitoring_up\n ScalingLimited False DesiredWithinRange the desired count is within the acceptable range\n \n```\n\n\u003cbr /\u003e\n\nCosts\n-----\n\nUsing custom metrics for HPA does not incur any additional charges. Users are charged only for application metrics and logs. See [Google Cloud's operations suite pricing](https://cloud.google.com/stackdriver/pricing) for details. The Pod for enabling custom metrics consumes an extra 15m CPU and 20MB memory."]]
