Set up trace collection using OpenTelemetry

This document shows you how to set up client-side and end-to-end tracing using OpenTelemetry . Before you can opt in for end-to-end tracing, you must set up client-side tracing. For more information, see Trace collection overview .

Before you begin

To ensure that the service account your application uses has the necessary permissions to set up trace collection, ask your administrator to grant the following IAM roles to the service account your application uses on your project:

Important: You must grant these roles to the service account your application uses, not to your user account. Failure to grant the roles to the correct principal might result in permission errors.
- All: Cloud Telemetry Traces Writer ( roles/telemetry.tracesWriter )
Verify that the Cloud Trace and Telemetry API are enabled on your project. For more information on enabling APIs, see Enabling APIs .

Configure client-side tracing

To export traces using the OpenTelemetry Protocol ( OTLP ), configure your application. You can send data to either an OpenTelemetry Collector or directly to Cloud Trace through the Telemetry API . Both methods use the same dependencies and configuration. They differ only in the OTLP endpoint the exporter uses.

Export traces using OpenTelemetry Protocol

To export traces using OpenTelemetry Protocol, configure the OpenTelemetry SDK and OTLP exporter:

Add the necessary dependencies to your application using the following code:

Java

 <dependency>  
<groupId>com.google.cloud</groupId>  
<artifactId>google-cloud-spanner</artifactId>
</dependency>
<dependency>  
<groupId>io.opentelemetry</groupId>  
<artifactId>opentelemetry-sdk</artifactId>
</dependency>
<dependency>  
<groupId>io.opentelemetry</groupId>  
<artifactId>opentelemetry-sdk-trace</artifactId>
</dependency>
<dependency>  
<groupId>io.opentelemetry</groupId>  
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Go

 go 
 . 
 opentelemetry 
 . 
 io 
 / 
 otel 
  
 v1 
 .28.0 
 go 
 . 
 opentelemetry 
 . 
 io 
 / 
 otel 
 / 
 sdk 
  
 v1 
 .28.0 
 go 
 . 
 opentelemetry 
 . 
 io 
 / 
 otel 
 / 
 exporters 
 / 
 otlp 
 / 
 otlptrace 
 / 
 otlptracegrpc 
  
 v1 
 .28.0

Node.js

 "@opentelemetry/exporter-trace-otlp-grpc" 
 : 
  
 "^0.57.0" 
 , 
 "@opentelemetry/sdk-trace-base" 
 : 
  
 "^1.26.0" 
 , 
 "@opentelemetry/sdk-trace-node" 
 : 
  
 "^1.26.0" 
 ,

Python

 pip 
 install 
 opentelemetry 
 - 
 api 
 opentelemetry 
 - 
 sdk 
 pip 
 install 
 opentelemetry 
 - 
 exporter 
 - 
 otlp

Configure the OpenTelemetry object and enable tracing.

Java

  Resource 
  
 resource 
  
 = 
  
 Resource 
  
 . 
 getDefault 
 (). 
 merge 
 ( 
 Resource 
 . 
 builder 
 (). 
 put 
 ( 
 "service.name" 
 , 
  
 "My App" 
 ). 
 build 
 ()); 
 OtlpGrpcSpanExporter 
  
 otlpGrpcSpanExporter 
  
 = 
  
 OtlpGrpcSpanExporter 
  
 . 
 builder 
 () 
  
 . 
 setEndpoint 
 ( 
 otlpEndpoint 
 ) 
  
 // Replace with your OTLP endpoint 
  
 . 
 build 
 (); 
 // Using a batch span processor 
 // You can use `.setScheduleDelay()`, `.setExporterTimeout()`, 
 // `.setMaxQueueSize`(), and `.setMaxExportBatchSize()` to further customize. 
 BatchSpanProcessor 
  
 otlpGrpcSpanProcessor 
  
 = 
  
 BatchSpanProcessor 
 . 
 builder 
 ( 
 otlpGrpcSpanExporter 
 ). 
 build 
 (); 
 // Create a new tracer provider 
 sdkTracerProvider 
  
 = 
  
 SdkTracerProvider 
 . 
 builder 
 () 
  
 // Use Otlp exporter or any other exporter of your choice. 
  
 . 
 addSpanProcessor 
 ( 
 otlpGrpcSpanProcessor 
 ) 
  
 . 
 setResource 
 ( 
 resource 
 ) 
  
 . 
 setSampler 
 ( 
 Sampler 
 . 
 traceIdRatioBased 
 ( 
 0.1 
 )) 
  
 . 
 build 
 (); 
 // Export to a collector that is expecting OTLP using gRPC. 
 OpenTelemetry 
  
 openTelemetry 
  
 = 
  
 OpenTelemetrySdk 
 . 
 builder 
 () 
  
 . 
 setTracerProvider 
 ( 
 sdkTracerProvider 
 ). 
 build 
 (); 
 // Enable OpenTelemetry traces before Injecting OpenTelemetry 
 SpannerOptions 
 . 
 enableOpenTelemetryTraces 
 (); 
 // Inject OpenTelemetry object via Spanner options or register as GlobalOpenTelemetry. 
 SpannerOptions 
  
 options 
  
 = 
  
 SpannerOptions 
 . 
 newBuilder 
 () 
  
 . 
 setOpenTelemetry 
 ( 
 openTelemetry 
 ) 
  
 . 
 build 
 (); 
 Spanner 
  
 spanner 
  
 = 
  
 options 
 . 
 getService 
 ();

Go

  // Ensure that your Go runtime version is supported by the OpenTelemetry-Go 
 // compatibility policy before enabling OpenTelemetry instrumentation. 
 // Enable OpenTelemetry traces by setting environment variable GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING 
 // to the case-insensitive value "opentelemetry" before loading the client library. 
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
 // Create a new resource to uniquely identify the application 
 res 
 , 
  
 err 
  
 := 
  
 resource 
 . 
 Merge 
 ( 
 resource 
 . 
 Default 
 (), 
  
 resource 
 . 
 NewWithAttributes 
 ( 
 semconv 
 . 
 SchemaURL 
 , 
  
 semconv 
 . 
 ServiceName 
 ( 
 "My App" 
 ), 
  
 semconv 
 . 
 ServiceVersion 
 ( 
 "0.1.0" 
 ), 
  
 )) 
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 log 
 . 
 Fatal 
 ( 
 err 
 ) 
 } 
 // Create a new OTLP exporter. 
 defaultOtlpEndpoint 
  
 := 
  
 "http://localhost:4317" 
  
 // Replace with the endpoint on which OTLP collector is running 
 traceExporter 
 , 
  
 err 
  
 := 
  
 otlptracegrpc 
 . 
 New 
 ( 
 ctx 
 , 
  
 otlptracegrpc 
 . 
 WithEndpoint 
 ( 
 defaultOtlpEndpoint 
 )) 
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 log 
 . 
 Fatal 
 ( 
 err 
 ) 
 } 
 // Create a new tracer provider 
 tracerProvider 
  
 := 
  
 sdktrace 
 . 
 NewTracerProvider 
 ( 
  
 sdktrace 
 . 
 WithResource 
 ( 
 res 
 ), 
  
 sdktrace 
 . 
 WithBatcher 
 ( 
 traceExporter 
 ), 
  
 sdktrace 
 . 
 WithSampler 
 ( 
 sdktrace 
 . 
 TraceIDRatioBased 
 ( 
 0.1 
 )), 
 ) 
 // Register tracer provider as global 
 otel 
 . 
 SetTracerProvider 
 ( 
 tracerProvider 
 )

Node.js

  const 
  
 { 
 NodeTracerProvider 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/sdk-trace-node' 
 ); 
 const 
  
 { 
  
 OTLPTraceExporter 
 , 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/exporter-trace-otlp-grpc' 
 ); 
 const 
  
 { 
  
 BatchSpanProcessor 
 , 
  
 TraceIdRatioBasedSampler 
 , 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/sdk-trace-base' 
 ); 
 const 
  
 { 
 Resource 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/resources' 
 ); 
 const 
  
 { 
 Spanner 
 } 
  
 = 
  
 require 
 ( 
 ' @google-cloud/spanner 
' 
 ); 
 // Define a Resource with service metadata 
 const 
  
 resource 
  
 = 
  
 new 
  
 Resource 
 ({ 
  
 'service.name' 
 : 
  
 'my-service' 
 , 
  
 'service.version' 
 : 
  
 '1.0.0' 
 , 
 }); 
 // Create an OTLP gRPC trace exporter 
 const 
  
 traceExporter 
  
 = 
  
 new 
  
 OTLPTraceExporter 
 ({ 
  
 url 
 : 
  
 'http://localhost:4317' 
 , 
  
 // Default OTLP gRPC endpoint 
 }); 
 // Create a provider with a custom sampler 
 const 
  
 provider 
  
 = 
  
 new 
  
 NodeTracerProvider 
 ({ 
  
 sampler 
 : 
  
 new 
  
 TraceIdRatioBasedSampler 
 ( 
 1.0 
 ), 
  
 // Sample 100% of traces 
  
 resource 
 , 
  
 spanProcessors 
 : 
  
 [ 
 new 
  
 BatchSpanProcessor 
 ( 
 traceExporter 
 )], 
 }); 
 // Uncomment following line to register tracerProvider globally or pass it in Spanner object 
 // provider.register(); 
 // Create the Cloud Spanner Client. 
 const 
  
 spanner 
  
 = 
  
 new 
  
  Spanner 
 
 ({ 
  
 projectId 
 : 
  
 projectId 
 , 
  
 observabilityOptions 
 : 
  
 { 
  
 tracerProvider 
 : 
  
 provider 
 , 
  
 enableExtendedTracing 
 : 
  
 true 
 , 
  
 enableEndToEndTracing 
 : 
  
 true 
 , 
  
 }, 
 });

Python

  # Setup OpenTelemetry, trace and OTLP exporter. 
 tracer_provider 
 = 
 TracerProvider 
 ( 
 sampler 
 = 
 ALWAYS_ON 
 ) 
 otlp_exporter 
 = 
 OTLPSpanExporter 
 ( 
 endpoint 
 = 
 "http://localhost:4317" 
 ) 
 tracer_provider 
 . 
 add_span_processor 
 ( 
 BatchSpanProcessor 
 ( 
 otlp_exporter 
 )) 
 # Setup the Cloud Spanner Client. 
 spanner_client 
 = 
 spanner 
 . 
 Client 
 ( 
 project_id 
 , 
 observability_options 
 = 
 dict 
 ( 
 tracer_provider 
 = 
 tracer_provider 
 , 
 enable_extended_tracing 
 = 
 True 
 , 
 enable_end_to_end_tracing 
 = 
 True 
 ), 
 )

Configure end-to-end tracing

This section provides instructions for configuring end-to-end tracing on Spanner client libraries:

Add the necessary dependencies to your application using the following code:

Java

The existing client-side tracing dependencies are sufficient for configuring end-to-end tracing. You don't need any additional dependencies.

Go

In addition to the dependencies you need for client-side tracing, you also need the following dependency:
go.opentelemetry.io/otel/propagation v1.28.0

Node.js

The existing client-side tracing dependencies are sufficient for configuring end-to-end tracing. You don't need any additional dependencies.

Python

The existing client-side tracing dependencies are sufficient for configuring end-to-end tracing. You don't need any additional dependencies.

Opt in for end-to-end tracing.

Java

 SpannerOptions 
  
 options 
  
 = 
  
 SpannerOptions 
 . 
 newBuilder 
 () 
  
 . 
 setOpenTelemetry 
 ( 
 openTelemetry 
 ) 
  
 . 
 setEnableEndToEndTracing 
 ( 
 /* enableEndtoEndTracing= */ 
  
 true 
 ) 
  
 . 
 build 
 ();

Go

Use the EnableEndToEndTracing option in the client configuration to opt in.

 client 
 , 
  
 _ 
  
 := 
  
 spanner 
 . 
 NewClientWithConfig 
 ( 
 ctx 
 , 
  
 "projects/test-project/instances/test-instance/databases/test-db" 
 , 
  
 spanner 
 . 
 ClientConfig 
 { 
 SessionPoolConfig 
 : 
  
 spanner 
 . 
 DefaultSessionPoolConfig 
 , 
 EnableEndToEndTracing 
 : 
  
 true 
 , 
 }, 
  
 clientOptions 
 ... 
 )

Node.js

 const 
  
 spanner 
  
 = 
  
 new 
  
 Spanner 
 ({ 
 projectId 
 : 
  
 projectId 
 , 
 observabilityOptions 
 : 
  
 { 
 tracerProvider 
 : 
  
 openTelemetryTracerProvider 
 , 
 enableEndToEndTracing 
 : 
  
 true 
 , 
 } 
 })

Python

 observability_options 
 = 
 { 
 "tracer_provider" 
 : 
 tracer_provider 
 , 
 "enable_end_to_end_tracing" 
 : 
 True 
 , 
 } 
 spanner 
 = 
 spanner 
 . 
 Client 
 ( 
 project_id 
 , 
 observability_options 
 = 
 observability_options 
 )

Set the trace context propagation in OpenTelemetry.

Java

 OpenTelemetry 
  
 openTelemetry 
  
 = 
  
 OpenTelemetrySdk 
 . 
 builder 
 () 
  
 . 
 setTracerProvider 
 ( 
 sdkTracerProvider 
 ) 
  
 . 
 setPropagators 
 ( 
 ContextPropagators 
 . 
 create 
 ( 
 W3CTraceContextPropagator 
 . 
 getInstance 
 ())) 
  
 . 
 buildAndRegisterGlobal 
 ();

Go

 // Register the TraceContext propagator globally. 
 otel 
 . 
 SetTextMapPropagator 
 ( 
 propagation 
 . 
 TraceContext 
 {})

Node.js

 const 
  
 { 
 propagation 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/api' 
 ); 
 const 
  
 { 
 W3CTraceContextPropagator 
 } 
  
 = 
  
 require 
 ( 
 '@opentelemetry/core' 
 ); 
 propagation 
 . 
 setGlobalPropagator 
 ( 
 new 
  
 W3CTraceContextPropagator 
 ());

Python

 from 
  
 opentelemetry.propagate 
  
 import 
 set_global_textmap 
 from 
  
 opentelemetry.trace.propagation.tracecontext 
  
 import 
 TraceContextTextMapPropagator 
 set_global_textmap 
 ( 
 TraceContextTextMapPropagator 
 ())

End-to-end tracing attributes

End-to-end traces can include the following information:

Attribute name	Description
service.name	The attribute value is always `spanner_api_frontend` .
cloud.region	The Google Cloud cloud region of the Spanner API frontend that serves your application request.
gcp.spanner.server.query.fingerprint	The attribute value is the query fingerprint. To debug this query further, see the `TEXT_FINGERPRINT` column in the Query statistics tables.
gcp.spanner.server.paxos.participantcount	The number of participants involved in the transaction. For more information, see Life of Spanner Reads & Writes .
gcp.spanner.isolation_level	The attribute value is the isolation level of the transaction. Possible values are `SERIALIZABLE` and `REPEATABLE_READ` . For more information, see Isolation levels overview .

Sample trace

An end-to-end trace lets you view the following details:

The latency between your application and Spanner. You can calculate network latency to see if you have any network issues.
The Spanner API frontend cloud region where your application requests are being served from. You can use this to check for cross-region calls between your application and Spanner.

In the following example, your application request is being served by the Spanner API frontend in the us-west1 region and the network latency is 8.542ms (55.47ms - 46.928ms).

View a end-to-end trace.

What's next

For more information about OpenTelemetry, see OpenTelemetry documentation .

Set up trace collection using OpenTelemetry Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Configure client-side tracing

Export traces using OpenTelemetry Protocol

Java

Go

Node.js

Python

Java

Go

Node.js

Python

Configure end-to-end tracing

Java

Go

Node.js

Python

Java

Go

Node.js

Python

Java

Go

Node.js

Python

End-to-end tracing attributes

Sample trace

What's next

Set up trace collection using OpenTelemetry