Add custom traces and metrics to your app with OpenTelemetry

This document describes how to add observability code to your application by using OpenTelemetry . OpenTelemetry provides instrumentation libraries that generate telemetry for popular frameworks. You can augment the library-generated telemetry by adding custom instrumentation that measures your application-specific behavior.

The principles and concepts described in this document can be applied to apps written in all languages supported by OpenTelemetry. To learn more about instrumentation, see the following documents:

The sample code, which is the same Go app that is described in Go instrumentation sample , is available in GitHub. To view the full sample, click More , and then select View on GitHub .

Before you begin

Enable the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs.

Enable the APIs

Create custom traces

To generate custom traces from your application, you add instrumentation code that creates OpenTelemetry spans . In OpenTelemetry, spans are the building blocks for traces.

To create a span, do the following:

Modify your app to acquire an OpenTelemetry Tracer . In OpenTelemetry, a tracer is a creator of spans. You can acquire a tracer as demonstrated in the following code:

  const 
  
 scopeName 
  
 = 
  
 "github.com/GoogleCloudPlatform/golang-samples/opentelemetry/instrumentation/app/work" 
 var 
  
 ( 
  
 meter 
  
 = 
  
 otel 
 . 
 Meter 
 ( 
 scopeName 
 ) 
  
 tracer 
  
 = 
  
 otel 
 . 
 Tracer 
 ( 
 scopeName 
 ) 
  
 sleepHistogram 
  
 metric 
 . 
 Float64Histogram 
  
 subRequestsHistogram 
  
 metric 
 . 
 Int64Histogram 
 )

The tracer name, which is represented by scopeName , identifies the instrumentation scope of the generated traces.

Use the tracer instance to create spans. In the following code sample, the computeSubrequests function generates a span whenever it is called:

  func 
  
 computeSubrequests 
 ( 
 r 
  
 * 
 http 
 . 
 Request 
 , 
  
 subRequests 
  
 int 
 ) 
  
 error 
  
 { 
  
 // Add custom span representing the work done for the subrequests 
  
 ctx 
 , 
  
 span 
  
 := 
  
 tracer 
 . 
 Start 
 ( 
 r 
 . 
 Context 
 (), 
  
 "subrequests" 
 ) 
  
 defer 
  
 span 
 . 
 End 
 () 
  
 // Make specified number of http requests to the /single endpoint. 
  
 for 
  
 i 
  
 := 
  
 0 
 ; 
  
 i 
 < 
 subRequests 
 ; 
  
 i 
 ++ 
  
 { 
  
 if 
  
 err 
  
 := 
  
 callSingle 
 ( 
 ctx 
 ); 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 err 
  
 } 
  
 } 
  
 // record number of sub-requests made 
  
 subRequestsHistogram 
 . 
 Record 
 ( 
 ctx 
 , 
  
 int64 
 ( 
 subRequests 
 )) 
  
 return 
  
 nil 
 }

In the previous code sample, the span generated from the computeSubrequests function represents the work done by the entire function. This is because the first step of the function is to start a new span using tracer.Start and the defer keyword before the span.End() ensures that the span is ended right before the function exits.

Create custom metrics

To generate metrics from your application, you add instrumentation code that records measurements taken during your app's execution.

To create metrics, do the following:

Modify your app to acquire an OpenTelemetry Meter . In OpenTelemetry, a meter provides access to metric instruments for recording metrics. You can acquire a meter as demonstrated in the following code:

  const 
  
 scopeName 
  
 = 
  
 "github.com/GoogleCloudPlatform/golang-samples/opentelemetry/instrumentation/app/work" 
 var 
  
 ( 
  
 meter 
  
 = 
  
 otel 
 . 
 Meter 
 ( 
 scopeName 
 ) 
  
 tracer 
  
 = 
  
 otel 
 . 
 Tracer 
 ( 
 scopeName 
 ) 
  
 sleepHistogram 
  
 metric 
 . 
 Float64Histogram 
  
 subRequestsHistogram 
  
 metric 
 . 
 Int64Histogram 
 )

The meter name, which is represented by scopeName , identifies the instrumentation scope of the generated metrics.

Use the meter instance to create instruments which can record metrics. For example, in the following code, we use the meter to create an OpenTelemetry Histogram :

  sleepHistogram 
 , 
  
 err 
  
 = 
  
 meter 
 . 
 Float64Histogram 
 ( 
 "example.sleep.duration" 
 , 
  
 metric 
 . 
 WithDescription 
 ( 
 "Sample histogram to measure time spent in sleeping" 
 ), 
  
 metric 
 . 
 WithExplicitBucketBoundaries 
 ( 
 0.05 
 , 
  
 0.075 
 , 
  
 0.1 
 , 
  
 0.125 
 , 
  
 0.150 
 , 
  
 0.2 
 ), 
  
 metric 
 . 
 WithUnit 
 ( 
 "s" 
 )) 
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 panic 
 ( 
 err 
 ) 
 }

This previous code generates a histogram named sleepHistogram .

Use the sleepHistogram instance to record the sleep time, which is determined when the function randomSleep is invoked:

  func 
  
 randomSleep 
 ( 
 r 
  
 * 
 http 
 . 
 Request 
 ) 
  
 time 
 . 
 Duration 
  
 { 
  
 // simulate the work by sleeping 100 to 200 ms 
  
 sleepTime 
  
 := 
  
 time 
 . 
 Duration 
 ( 
 100 
 + 
 rand 
 . 
 Intn 
 ( 
 100 
 )) 
  
 * 
  
 time 
 . 
 Millisecond 
  
 time 
 . 
 Sleep 
 ( 
 sleepTime 
 ) 
  
 hostValue 
  
 := 
  
 attribute 
 . 
 String 
 ( 
 "host.value" 
 , 
  
 r 
 . 
 Host 
 ) 
  
 // custom histogram metric to record time slept in seconds 
  
 sleepHistogram 
 . 
 Record 
 ( 
 r 
 . 
 Context 
 (), 
  
 sleepTime 
 . 
 Seconds 
 (), 
  
 metric 
 . 
 WithAttributes 
 ( 
 hostValue 
 )) 
  
 return 
  
 sleepTime 
 }

The recorded metrics from these instruments are exported from your application based on your OpenTelemetry exporter configuration.

Add custom traces and metrics to your app with OpenTelemetry Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Create custom traces

Create custom metrics

What's next

Add custom traces and metrics to your app with OpenTelemetry