Regional external Application Load Balancer logging and monitoring

Logging

You can enable, disable, and view logs for an external Application Load Balancer backend service .

You enable or disable logging for each backend service. You can configure whether to log all requests or a randomly sampled fraction.

You must ensure that you don't have a logs exclusion that applies to external Application Load Balancers. For information about how to verify that Cloud HTTP Load Balancer logs are allowed, see Exclusion filters .

Logs sampling and collection

The requests (and corresponding responses) handled by load balancer backend virtual machine (VM) instances are sampled. These sampled requests are then processed to generate logs. You control the fraction of the requests that are emitted as log entries according to the logConfig.sampleRate parameter . When logConfig.sampleRate is 1.0 (100%), this means that logs are generated for all of the requests and written to Cloud Logging.

Optional fields

Log records contain required fields and optional fields. The What is logged section lists which fields are optional and which are required. All required fields are always included. You can customize which optional fields you keep.

• If you select include all optional, all optional fields in the log record format are included in the logs. When new optional fields are added to the record format, the logs automatically include the new fields.

• If you select exclude all optional, all optional fields are omitted.

• If you select custom, you can specify the optional fields that you want to include, such as tls.protocol,tls.cipher,orca_load_report.cpu_utilization,orca_load_report.mem_utilization .

For information about customizing optional fields, see Enable logging on a new backend service .

Enabling logging on a new backend service

gcloud compute backend-services create BACKEND_SERVICE 
\
    --region= REGION 
\
    --enable-logging \
    --logging-sample-rate= VALUE 
\
    --load-balancing-scheme=EXTERNAL_MANAGED \
    --logging-optional= LOGGING_OPTIONAL_MODE 
\
    --logging-optional-fields= OPTIONAL_FIELDS 

Enabling logging on an existing backend service

gcloud compute backend-services update BACKEND_SERVICE 
\
    --region= REGION 
\
    --enable-logging \
    --logging-sample-rate= VALUE 
\
    --logging-optional= LOGGING_OPTIONAL_MODE 
\
    --logging-optional-fields= OPTIONAL_FIELDS 

Disabling or modifying logging on an existing backend service

gcloud compute backend-services update BACKEND_SERVICE 
\
    --region= REGION 
\
    --no-enable-logging

gcloud compute backend-services update BACKEND_SERVICE 
\
    --global \
    --enable-logging \
    --logging-sample-rate= VALUE 
\
    --logging-optional= LOGGING_OPTIONAL_MODE 
\
    --logging-optional-fields= OPTIONAL_FIELDS 

gcloud compute backend-services update BACKEND_SERVICE 
\
    --global \
    --enable-logging \
    --logging-sample-rate= VALUE 
\
    --logging-optional= LOGGING_OPTIONAL_MODE 
\
    --logging-optional-fields=

gcloud compute backend-services update BACKEND_SERVICE 
\
    --global | --region= REGION 
\
    --logging-sample-rate= VALUE 

View logs

HTTP(S) logs are indexed first by a forwarding rule , then by a URL map .

To view logs, go to the Logs Explorerpage:

Go to Logs Explorer

• To view all logs, in the Resourcefilter menu, select Cloud HTTP Load Balancer  > All forwarding rules.

• To view logs for one forwarding rule, select a single forwarding rule name.

• To view logs for one URL map, select a forwarding rule, and then select a URL map.

Log fields of type boolean typically only appear if they have a value of true . If a boolean field has a value of false , that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks. For regional external Application Load Balancers , you can export logs-based metrics using resource logs ( resource.type="http_external_regional_lb_rule" ).

What is logged

External Application Load Balancer log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log records contain required fields, which are the default fields of every log record.

Log records contain optional fields that add additional information about your HTTP(S) traffic. Optional fields can be omitted to save storage costs.

TlsInfo field format

MtlsInfo field format

Resource labels

The following table lists the resource labels for resource.type="http_external_regional_lb_rule" .

proxyStatus error field

The proxyStatus field contains a string that specifies why the load balancer returned an error. There are two parts in the proxyStatus field, proxyStatus error and proxyStatus details . This section describes the strings that are supported in the proxyStatus error field.

The proxyStatus error field is applicable to the following load balancers:

• Regional external Application Load Balancer
• Cross-region internal Application Load Balancer
• Regional internal Application Load Balancer

proxyStatus details field

The proxyStatus field contains a string that specifies why the load balancer returned an error. There are two parts in the proxyStatus field, proxyStatus error and proxyStatus details . The proxyStatus details field is optional and is shown only when additional information is available. This section describes the strings that are supported in the proxyStatus details field.

The proxyStatus details field is applicable to the following load balancers:

• Regional external Application Load Balancer
• Regional internal Application Load Balancer
• Cross-region internal Application Load Balancer

Failed TLS connection log entries

When the TLS connection between the client and the load balancer fails before any backend is selected, log entries record the errors. You can configure the backend services with different log sample rates. When a TLS connection fails, the failed TLS connection log sample rate is the highest sample rate for any backend service. For example, if you have configured two backend services with logging sample rate as 0.3 and 0.5 , the failed TLS connection log sample rate is 0.5 .

You can identify failed TLS connections by checking for these log entry details:

• proxyStatus error type is tls_alert_received , tls_certificate_error , tls_protocol_error , or connection_terminated .
• There is no backend information.

The following sample shows a failed TLS log entry with the proxyStatus error field:

json_payload:    {
   @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
   proxyStatus: "error="tls_alert_received"; details="server_to_client: handshake_failure""
   log_name: "projects/529254013417/logs/mockservice.googleapis.com%20name"
   }
   http_request {
    latency {
      nanos: 12412000
    }
    protocol: "HTTP/1.0"
    remote_ip: "127.0.0.2"
   }
  resource {
    type: "mock_internal_http_lb_rule"
    labels {
      backend_name: ""
      backend_scope: ""
      backend_scope_type: "UNKNOWN"
      backend_target_name: ""
      backend_target_type: "UNKNOWN"
      backend_type: "UNKNOWN"
      forwarding_rule_name: "l7-ilb-https-forwarding-rule-dev"
      matched_url_path_rule: "UNKNOWN"
      network_name: "lb-network"
      region: "REGION"
      target_proxy_name: "l7-ilb-https-proxy-dev"
      url_map_name: ""
    }
  }
  timestamp: "2023-08-15T16:49:30.850785Z"

Authorization policy request logs

The authz_info object in the Load Balancer Log Entry JSON payload contains information about authorization policies. You can configure log-based metrics for traffic allowed or denied by these policies. Check more authorization policies log details .

Interacting with the logs

You can interact with the external Application Load Balancer logs by using the Cloud Logging API. The Logging API provides ways to interactively filter logs that have specific fields set. It exports matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information about the Logging API, see Logging API overview .

Monitoring

The load balancer exports monitoring data to Monitoring .

You can use monitoring metrics to do the following:

• Evaluate a load balancer's configuration, usage, and performance
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Cloud Monitoring API .

Defining alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

1. In the Google Cloud console, go to the notifications Alerting page:

   Go to Alerting

   If you use the search bar to find this page, then select the result whose subheading is Monitoring .

2. If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
3. From the Alerting page, select Create policy .
4. To select the metric, expand the Select a metric menu and then do the following:
   1. To limit the menu to relevant entries, enter Regional External Application Load Balancer Rule into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
   2. For the Resource type , select Regional External Application Load Balancer Rule .
   3. Select a Metric category and a Metric , and then select Apply .
5. Click Next .
6. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Create metric-threshold alerting policies .
7. Click Next .
8. Optional: To add notifications to your alerting policy, click Notification channels . In the dialog, select one or more notification channels from the menu, and then click OK .
9. Optional: Update the Incident autoclose duration . This field determines when Monitoring closes incidents in the absence of metric data.
10. Optional: Click Documentation , and then add any information that you want included in a notification message.
11. Click Alert name and enter a name for the alerting policy.
12. Click Create Policy .

Defining Cloud Monitoring custom dashboards

You can create custom Cloud Monitoring dashboards for the load balancer's metrics:

1. In the Google Cloud console, go to the Monitoringpage.

   Go to Monitoring

2. Select Dashboards  > Create Dashboard.

3. Click Add Chart, and then give the chart a title.

4. To identify the time series to be displayed, choose a resource type and metric type:

   1. In the Resource & Metricsection, click the chart, and then in the Select a metricsection, select from the available options:
   2. For a regional external Application Load Balancer, select the resource type Regional External Application Load Balancer Rule.
   3. Click Apply.

5. To specify monitoring filters, click Filters  > Add filter.

6. Click Save.

Metric reporting frequency and retention

Metrics for the external Application Load Balancers are exported to Cloud Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks.

The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics

You can monitor the following metrics for external Application Load Balancers.

The following metrics for regional external Application Load Balancers are reported into Cloud Monitoring . These metrics are prepended with loadbalancing.googleapis.com/ .

Filtering dimensions for metrics

You can apply filters for metrics for external Application Load Balancers.

Metrics are aggregated for each regional external Application Load Balancer. You can filter aggregated metrics by using the following dimensions for resource.type="http_external_regional_lb_rule" .

What's next

• Read conceptual information about external Application Load Balancers .