Connect to Cloud Storage using gRPC

gRPC is a high performance, open source universal RPC framework developed by Google that you can use to define your services using Protocol Buffers. You can use gRPC to interact with Cloud Storage. gRPC utilizes direct connectivity between Compute Engine instances and Cloud Storage buckets, bypassing Google Front Ends (GFEs).

You can connect to Cloud Storage using gRPC through the following supported clients:

Enable gRPC on a client library

C++

Before you begin

  1. Ensure you have the following versions installed:

    • gRPC version 1.65.1 or later

    • C++ client library version v2.30.0 or later

    • C++ version 14 or later

    For installation instructions, see Setting up a C++ development environment .

  2. Set up authentication .

  3. Ensure each Compute Engine instance has a service account attached to it, even if the service account has no permissions. This service account is used to represent the Compute Engine instance in the Application Layer Transport Security (ALTS) handshake process and is required for direct connectivity .

Configure the C++ client library

  1. Create a gRPC client using gcs::MakeGrpcClient() :

       
     namespace 
      
     gcs 
      
     = 
      
     google 
     :: 
     cloud 
     :: 
     storage 
     ; 
      
     void 
      
     App 
     () 
      
     { 
      
     auto 
      
     client 
      
     = 
      
     gcs 
     : 
     : 
     MakeGrpcClient 
     (); 
      
     // 
      
     application 
      
     code 
      
     } 
     
    

    The C++ client library automatically uses direct connectivity when it detects that the application is running on Google Cloud.

  2. To configure the C++ client library to use gRPC, enable the Cloud Storage gRPC client to update your build system configuration for CMake or Bazel.

    CMake

    1. Enable the Cloud Storage gRPC client plugin at compile-time.

       cmake -DGOOGLE_CLOUD_CPP_ENABLE=storage_grpc [other options here] 
      
    2. In your codebase, for the target_link_libraries() command replace google-cloud-cpp::storage with google-cloud-cpp::storage_grpc

      For example, the quickstart program for gRPC uses the following code:

       add_executable(quickstart_grpc quickstart_grpc.cc)
      target_link_libraries(quickstart_grpc google-cloud-cpp::storage_grpc) 
      

    Bazel

    Replace the dependencies from @google_cloud_cpp//:storage to @google_cloud_cpp//:storage_grpc .

    For example, the quickstart program for gRPC uses the following code:

     cc_binary(
          name = "quickstart",
          srcs = [
              "quickstart.cc",
          ],
          deps = [
              "@com_github_googleapis_google_cloud_cpp//:storage_grpc",
          ],
      ) 
    

Java

Before you begin

  1. Ensure you have the following versions installed:

    • Java client libraries:

      • com.google.cloud:google-cloud-storage:2.43.1 or later.
      • com.google.cloud:libraries-bom:26.48 or later.
    • Java 8 or later

    For installation instructions, see Setting up a Java development environment .

  2. Set up authentication .

  3. Ensure each Compute Engine instance has a service account attached to it, even if the service account has no permissions. This service account is used to represent the Compute Engine instance in the Application Layer Transport Security (ALTS) handshake process and is required for direct connectivity .

Update your project to use the BOM

To make sure that your projects have compatible versions of Google Cloud client libraries, use the versions specified in the Google Cloud libraries Bill of Materials (BOM) . To update your project to use the BOM, use any of the following methods:

Standalone Cloud Storage

If you are using the Cloud Storage client library independently (without other Google Cloud libraries), use the Cloud Storage client library-specific BOM.

Maven

Import the BOM in the dependencyManagement section of your pom.xml file.

The following example shows how you would import the BOM and include the google-cloud-storage artifact.

 < dependencyManagement 
>  
< dependencies 
>  
< dependency 
>  
< groupId>com.google.cloud</groupId> 
  
< artifactId>google-cloud-storage-bom</artifactId> 
  
< version>2.43.1</version> 
  
< type>pom</type> 
  
< scope>import</scope> 
  
< /dependency 
>  
< /dependencies 
>
< /dependencyManagement 
>

< dependencies 
>  
< dependency 
>  
< groupId>com.google.cloud</groupId> 
  
< artifactId>google-cloud-storage</artifactId> 
  
< /dependency 
>
< /dependencies 
> 

Gradle

Add a platform dependency on com.google.cloud:google-cloud-storage-bom :

 implementation platform('com.google.cloud:google-cloud-storage-bom:2.43.1')
implementation 'com.google.cloud:google-cloud-storage' 

Cloud Storage with other Google Cloud libraries

If you are using the Cloud Storage client library along with other Google Cloud libraries, use the Google Cloud client libraries BOM.

Maven

Import the BOM in the dependencyManagement section of your pom.xml file.

The following example shows how to import the BOM and include the libraries-bom artifact.

 < dependencyManagement 
>  
< dependencies 
>  
< dependency 
>  
< groupId>com.google.cloud</groupId> 
  
< artifactId>libraries-bom</artifactId> 
  
< version>26.48.0</version> 
  
< type>pom</type> 
  
< scope>import</scope> 
  
< /dependency 
>  
< /dependencies 
>
< /dependencyManagement 
>

< dependencies 
>  
< dependency 
>  
< groupId>com.google.cloud</groupId> 
  
< artifactId>google-cloud-storage</artifactId> 
  
< /dependency 
>
< /dependencies 
> 

Gradle

Add a platform dependency on com.google.cloud:libraries-bom :

 implementation platform('com.google.cloud:libraries-bom:26.48.0')
implementation 'com.google.cloud:google-cloud-storage' 

Create a gRPC client

The following sample uses a gRPC centric builder. The gRPC Java client automatically uses direct connectivity when it detects that the application is running on Google Cloud.

  // Imports the Google Cloud client library 
 import 
  
 com.google.cloud.storage. Bucket 
 
 ; 
 import 
  
 com.google.cloud.storage. BucketInfo 
 
 ; 
 import 
  
 com.google.cloud.storage. Storage 
 
 ; 
 import 
  
 com.google.cloud.storage. StorageOptions 
 
 ; 
 public 
  
 class 
 QuickstartGrpcSample 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 ... 
  
 args 
 ) 
  
 throws 
  
 Exception 
  
 { 
  
 // Create an instance of options which will use the Google Cloud Storage gRPC API for all 
  
 // operations 
  
  StorageOptions 
 
  
 options 
  
 = 
  
  StorageOptions 
 
 . 
  grpc 
 
 (). 
 build 
 (); 
  
 // Instantiates a client in a try-with-resource to automatically cleanup underlying resources 
  
 try 
  
 ( 
  Storage 
 
  
 storage 
  
 = 
  
 options 
 . 
  getService 
 
 ()) 
  
 { 
  
 // The name for the new bucket 
  
 String 
  
 bucketName 
  
 = 
  
 args 
 [ 
 0 
 ] 
 ; 
  
 // "my-new-bucket"; 
  
 // Creates the new bucket using a request to the gRPC API 
  
  Bucket 
 
  
 bucket 
  
 = 
  
 storage 
 . 
 create 
 ( 
  BucketInfo 
 
 . 
 of 
 ( 
 bucketName 
 )); 
  
 System 
 . 
 out 
 . 
 printf 
 ( 
 "Bucket %s created.%n" 
 , 
  
 bucket 
 . 
 getName 
 ()); 
  
 } 
  
 } 
 } 
 

Go

Before you begin

  1. Setup your development environment .

  2. Ensure you use Cloud Storage Go client library version 1.46.0 or later.

  3. Set up authentication .

  4. Ensure each Compute Engine instance has a service account attached to it, even if the service account has no permissions. This service account is used to represent the Compute Engine instance in the Application Layer Transport Security (ALTS) handshake process and is required for direct connectivity .

Create a gRPC client

To use the client, you need to call the NewGRPCClient constructor in your application instead of NewClient .

  // Sample storage-quickstart creates a Google Cloud Storage bucket using 
 // gRPC API. 
 package 
  
 main 
 import 
  
 ( 
  
 "context" 
  
 "fmt" 
  
 "log" 
  
 "time" 
  
 "cloud.google.com/go/storage" 
 ) 
 func 
  
 main 
 () 
  
 { 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 // Use your Google Cloud Platform project ID and Cloud Storage bucket 
  
 projectID 
  
 := 
  
 "project-id" 
  
 bucketName 
  
 := 
  
 "bucket-name" 
  
 // Creates a gRPC enabled client. 
  
 client 
 , 
  
 err 
  
 := 
  
 storage 
 . 
  NewGRPCClient 
 
 ( 
 ctx 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 log 
 . 
 Fatalf 
 ( 
 "Failed to create client: %v" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 client 
 . 
 Close 
 () 
  
 // Creates the new bucket. 
  
 ctx 
 , 
  
 cancel 
  
 := 
  
 context 
 . 
 WithTimeout 
 ( 
 ctx 
 , 
  
 time 
 . 
 Second 
 * 
 10 
 ) 
  
 defer 
  
 cancel 
 () 
  
 if 
  
 err 
  
 := 
  
 client 
 . 
  Bucket 
 
 ( 
 bucketName 
 ). 
  Create 
 
 ( 
 ctx 
 , 
  
 projectID 
 , 
  
 nil 
 ); 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 log 
 . 
 Fatalf 
 ( 
 "Failed to create bucket: %v" 
 , 
  
 err 
 ) 
  
 } 
  
 fmt 
 . 
 Printf 
 ( 
 "Bucket %v created.\n" 
 , 
  
 bucketName 
 ) 
 } 
 

The Go client library automatically uses direct connectivity when it detects that the application is running on Google Cloud.

For information about how to use a gRPC client, see gRPC API .

Configure VPC Service Controls

If you are using Virtual Private Cloud with restricted virtual IP addresses ( VPC Service Controls ) to enhance the security of your network, you need to update your firewall rules to enable direct connectivity for optimal performance between your Compute Engine instance and Cloud Storage bucket.

To do so, add allowlist firewall rules to permit traffic on all ports for the following CIDR blocks:

  • For IPv4 traffic: 34.126.0.0/18
  • For IPv6 traffic: 2001:4860:8040::/42

In addition to the preceding rules, retain the existing allowlist rule for 199.36.153.4/30 .

If you have restrictions on firewall rule modifications and you cannot update them, then you can force the traffic to avoid direct connectivity by using storage.googleapis.com as the Cloud Storage endpoint instead of google-c2p://storage.googleapis.com .

For example, for C++ use, .set<google::cloud::EndpointOption>(storage.googleapis.com) instead of google-c2p:///storage.googleapis.com .

You can configure Cloud Storage client libraries to to generate gRPC related metrics in Cloud Monitoring. The gRPC related metrics can help you to do the following:

  • Monitor and optimize the performance of gRPC requests to Cloud Storage.

  • Troubleshoot and debug issues.

  • Gain insights into your application's usage and behavior.

For information about how to generate gRPC related metrics, see Use client-side metrics .

If gathering metrics isn't necessary for your use case, you can choose to opt-out of metrics collection. For instructions, see Opt-out of client-side metrics .

Limitations

  • IPv6 requests cannot be sent over legacy networks .

  • Direct connectivity is not supported when using the following GKE versions on IPv4-only clusters:

    • 1.28, 1.28.0-gke.100 or later until 1.28.5-gke.1199000
    • 1.27, 1.27.4-gke.1900 or later
    • 1.26, 1.26.10-gke.1238000 or later
    • 1.25, 1.25.15-gke.1045000 or later
  • gRPC does not support the notifications , hmacKeys , and serviceAccount methods.

  • HTTP-specific client constructor options, such as WithHTTPClient , are not supported for Go client library.

Troubleshooting direct connectivity

To learn how to check for direct connectivity and troubleshoot when it's not available, see Direct connectivity .

What's next

Design a Mobile Site
View Site in Mobile | Classic
Share by: