Payload unwrapping for Pub/Sub push subscriptions

When building your Pub/Sub system, payload unwrapping can help you connect to other systems that don't adhere to all system requirements of a standard Pub/Sub push endpoint implementation.

Some potential use cases for payload unwrapping are as follows:

  • You don't want to write Pub/Sub-specific message parsing code for your HTTP push endpoints.
  • You prefer receiving Pub/Sub message metadata as HTTP headers instead of the metadata in the HTTP POST body.
  • You want to send Pub/Sub messages and exclude the Pub/Sub metadata, for example when sending data to a third-party API.

How payload unwrapping works

Payload unwrapping is a feature that strips Pub/Sub messages of all message metadata, except for the message data. By sending raw message data, subscribers can process the message without having to adhere to any system requirements of Pub/Sub.

  • With payload unwrapping, the message data is delivered directly as the HTTP body.
  • Without payload unwrapping, Pub/Sub delivers a JSON object that contains multiple message metadata fields and a message data field. In this case, the JSON must be parsed to retrieve the message data and then base64 decoded.

Write metadata

After enabling payload unwrapping, you can use the write metadata option which adds previously removed message metadata into the request header .

  • Write metadata enabled.Add message metadata back into the request header. Also delivers the raw, decoded message data.
  • Write metadata disabled.Only delivers the raw, decoded message data.

Write metadata is exposed through the Pub/Sub, the Google Cloud CLI argument --push-no-wrapper-write-metadata , and the API property NoWrapper . By default, this value is null.

Before you begin

Example for wrapped and unwrapped messages

The following examples illustrate the difference between sending a wrapped and unwrapped HTTP message. In these examples, the message data contains the string {"status": "Hello there"} .

For this example, a subscription is created with the payload unwrapping feature enabled and publishes a message to mytopic . It uses an ordering key with a value of some-key and the media type is declared as application/json .

gcloud pubsub topics publish mytopic
   --message='{"status": "Hello there"}'
   --ordering-key="some-key"
   --attribute "Content-Type=application/json"

The following sections show the difference between a wrapped and unwrapped message.

Wrapped message

The following example shows a standard Pub/Sub wrapped message. In this case, payloading unwrapping isn't enabled.

Publish Push Endpoint Receives
data="{"status": "Hello there"}"
ordering_key="some-key"
attributes=
  {
     {"Content-Type", "application/json"}
  }
Content-Length: 361
Content-Type: application/json
User-Agent: CloudPubSub-Google
Host: subscription-project.uc.r.appspot.com

{
  "message": {
      "attributes": {
          "Content-Type": "application/json"
      },
      "data": "eyJzdGF0dXMiOiAiSGVsbG8gdGhlcmUifQ==", //  Base64 - {"status": "Hello there"}
      "messageId": "2070443601311540",
      "message_id": "2070443601311540",
      "publishTime": "2021-02-26T19:13:55.749Z",
      "publish_time": "2021-02-26T19:13:55.749Z"
  },
  "subscription": "projects/myproject/..."
}

Unwrapped message with write metadata disabled

The following example shows an unwrapped message with the write metadata option disabled. In this case, the x-goog-pubsub-* headers and message attributes aren't included.

Publish Push Endpoint Receives
data="{"status": "Hello there"}"
ordering_key="some-key"
attributes=
  {
     {"Content-Type", "application/json"}
  }
Content-Length: 25
User-Agent: CloudPubSub-Google
Host: subscription-project.uc.r.appspot.com

{"status": "Hello there"}

The following example shows an unwrapped message with the write metadata option enabled. In this case, the x-goog-pubsub-* headers and message attributes are included.

Publish Push Endpoint Receives
data="{"status": "Hello there"}"
ordering_key="some-key"
attributes=
  {
     {"Content-Type", "application/json"}
  }
x-goog-pubsub-subscription-name: "projects/myproject/..."
x-goog-pubsub-message-id: "2070443601311540"
x-goog-pubsub-publish-time: "2021-02-26T19:13:55.749Z"
x-goog-pubsub-ordering-key: "some-key"
Content-Type: application/json
Content-Length: 12
User-Agent: CloudPubSub-Google
Host: subscription-project.uc.r.appspot.com

{"status": "Hello there"}

Configure payload unwrapping

You can enable payload unwrapping push delivery for a subscription using the Google Cloud console Subscription Detailspage, the Google Cloud CLI, or the Client Libraries.

Console

  1. In the Google Cloud console, go to the Subscriptionspage.

    Open Pub/Sub subscriptions

  2. Click Create subscription.

  3. In the Subscription IDfield, enter a name.

    For information on how to name a subscription, see Guidelines to name a topic or a subscription .

  4. Select a topic from the drop-down menu. The subscription receives messages from the topic.

  5. For Delivery type, select Push.

  6. To enable payload unwrapping, select Enable payload unwrapping.

  7. (Optional) To preserve metadata of messages in the request header, select Write metadata. You must enable this option to set a Content-Type header for your messages.

  8. Specify an endpoint URL.

  9. Retain all other default values.

  10. Click Create.

gcloud

To configure a subscription with payload unwrapping that includes standard HTTP headers, run the following gcloud pubsub subscriptions create command:

gcloud pubsub subscriptions create SUBSCRIPTION 
\
  --topic TOPIC 
\
  --push-endpoint= PUSH_ENDPOINT 
\
  --push-no-wrapper

Replace the following:

  • SUBSCRIPTION : the name or ID of your pull subscription.
  • TOPIC : the ID of the topic.
  • PUSH_ENDPOINT : the URL to use as the endpoint for this subscription. For example, https://myproject.appspot.com/myhandler
  • --push-no-wrapper : delivers the message data directly as the HTTP body.

To configure a subscription with payload unwrapping and control the use of x-goog-pubsub-* headers, run following command:

gcloud pubsub subscriptions create SUBSCRIPTION 
\
  --topic TOPIC 
\
  --push-endpoint= PUSH_ENDPOINT 
\
  --push-no-wrapper \
  --push-no-wrapper-write-metadata
  • --push-no-wrapper-write-metadata : When true, writes the Pub/Sub message metadata to x-goog-pubsub-<KEY>:<VAL> headers of the HTTP request. Writes the Pub/Sub message attributes to <KEY>:<VAL> headers of the HTTP request.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Python API reference documentation .

  from 
  
 google.cloud 
  
 import 
 pubsub_v1 
 # TODO(developer) 
 # project_id = "your-project-id" 
 # topic_id = "your-topic-id" 
 # subscription_id = "your-subscription-id" 
 # endpoint = "https://my-test-project.appspot.com/push" 
 publisher 
 = 
 pubsub_v1 
 . 
  PublisherClient 
 
 () 
 subscriber 
 = 
 pubsub_v1 
 . 
  SubscriberClient 
 
 () 
 topic_path 
 = 
 publisher 
 . 
 topic_path 
 ( 
 project_id 
 , 
 topic_id 
 ) 
 subscription_path 
 = 
 subscriber 
 . 
 subscription_path 
 ( 
 project_id 
 , 
 subscription_id 
 ) 
 no_wrapper 
 = 
 pubsub_v1 
 . 
 types 
 . 
  PushConfig 
 
 . 
  NoWrapper 
 
 ( 
 write_metadata 
 = 
 True 
 ) 
 push_config 
 = 
 pubsub_v1 
 . 
 types 
 . 
  PushConfig 
 
 ( 
 push_endpoint 
 = 
 endpoint 
 , 
 no_wrapper 
 = 
 no_wrapper 
 ) 
 # Wrap the subscriber in a 'with' block to automatically call close() to 
 # close the underlying gRPC channel when done. 
 with 
 subscriber 
 : 
 subscription 
 = 
 subscriber 
 . 
 create_subscription 
 ( 
 request 
 = 
 { 
 "name" 
 : 
 subscription_path 
 , 
 "topic" 
 : 
 topic_path 
 , 
 "push_config" 
 : 
 push_config 
 , 
 } 
 ) 
 print 
 ( 
 f 
 "Push no wrapper subscription created: 
 { 
 subscription 
 } 
 ." 
 ) 
 print 
 ( 
 f 
 "Endpoint for subscription is: 
 { 
 endpoint 
 } 
 " 
 ) 
 print 
 ( 
 f 
 "No wrapper configuration for subscription is: 
 { 
 no_wrapper 
 } 
 " 
 ) 
 

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Java API reference documentation .

  /* 
 * Copyright 2016 Google LLC 
 * 
 * Licensed under the Apache License, Version 2.0 (the "License"); 
 * you may not use this file except in compliance with the License. 
 * You may obtain a copy of the License at 
 * 
 * http://www.apache.org/licenses/LICENSE-2.0 
 * 
 * Unless required by applicable law or agreed to in writing, software 
 * distributed under the License is distributed on an "AS IS" BASIS, 
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
 * See the License for the specific language governing permissions and 
 * limitations under the License. 
 */ 
 package 
  
 pubsub 
 ; 
 import 
  
 com.google.cloud.pubsub.v1. SubscriptionAdminClient 
 
 ; 
 import 
  
 com.google.pubsub.v1. PushConfig 
 
 ; 
 import 
  
 com.google.pubsub.v1. PushConfig 
. NoWrapper 
 
 ; 
 import 
  
 com.google.pubsub.v1. Subscription 
 
 ; 
 import 
  
 com.google.pubsub.v1. SubscriptionName 
 
 ; 
 import 
  
 com.google.pubsub.v1. TopicName 
 
 ; 
 import 
  
 java.io.IOException 
 ; 
 public 
  
 class 
 CreateUnwrappedPushSubscriptionExample 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 ... 
  
 args 
 ) 
  
 throws 
  
 Exception 
  
 { 
  
 // TODO(developer): Replace these variables before running the sample. 
  
 String 
  
 projectId 
  
 = 
  
 "your-project-id" 
 ; 
  
 String 
  
 subscriptionId 
  
 = 
  
 "your-subscription-id" 
 ; 
  
 String 
  
 topicId 
  
 = 
  
 "your-topic-id" 
 ; 
  
 String 
  
 pushEndpoint 
  
 = 
  
 "https://my-test-project.appspot.com/push" 
 ; 
  
 createPushSubscriptionExample 
 ( 
 projectId 
 , 
  
 subscriptionId 
 , 
  
 topicId 
 , 
  
 pushEndpoint 
 ); 
  
 } 
  
 public 
  
 static 
  
 void 
  
 createPushSubscriptionExample 
 ( 
  
 String 
  
 projectId 
 , 
  
 String 
  
 subscriptionId 
 , 
  
 String 
  
 topicId 
 , 
  
 String 
  
 pushEndpoint 
 ) 
  
 throws 
  
 IOException 
  
 { 
  
 try 
  
 ( 
  SubscriptionAdminClient 
 
  
 subscriptionAdminClient 
  
 = 
  
  SubscriptionAdminClient 
 
 . 
 create 
 ()) 
  
 { 
  
  TopicName 
 
  
 topicName 
  
 = 
  
  TopicName 
 
 . 
 of 
 ( 
 projectId 
 , 
  
 topicId 
 ); 
  
  SubscriptionName 
 
  
 subscriptionName 
  
 = 
  
  SubscriptionName 
 
 . 
 of 
 ( 
 projectId 
 , 
  
 subscriptionId 
 ); 
  
  NoWrapper 
 
  
 noWrapper 
  
 = 
  
  NoWrapper 
 
 . 
 newBuilder 
 () 
  
 // Determines if message metadata is added to the HTTP headers of 
  
 // the delivered message. 
  
 . 
 setWriteMetadata 
 ( 
 true 
 ) 
  
 . 
 build 
 (); 
  
  PushConfig 
 
  
 pushConfig 
  
 = 
  
  PushConfig 
 
 . 
 newBuilder 
 (). 
  setPushEndpoint 
 
 ( 
 pushEndpoint 
 ). 
  setNoWrapper 
 
 ( 
 noWrapper 
 ). 
 build 
 (); 
  
 // Create a push subscription with default acknowledgement deadline of 10 seconds. 
  
 // Messages not successfully acknowledged within 10 seconds will get resent by the server. 
  
  Subscription 
 
  
 subscription 
  
 = 
  
 subscriptionAdminClient 
 . 
 createSubscription 
 ( 
 subscriptionName 
 , 
  
 topicName 
 , 
  
 pushConfig 
 , 
  
 10 
 ); 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Created push subscription: " 
  
 + 
  
 subscription 
 . 
  getName 
 
 ()); 
  
 } 
  
 } 
 } 
 

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub C++ API reference documentation .

  namespace 
  
 pubsub 
  
 = 
  
 :: 
 google 
 :: 
 cloud 
 :: 
 pubsub 
 ; 
 namespace 
  
 pubsub_admin 
  
 = 
  
 :: 
 google 
 :: 
 cloud 
 :: 
 pubsub_admin 
 ; 
 []( 
 pubsub_admin 
 :: 
 SubscriptionAdminClient 
  
 client 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 project_id 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 topic_id 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 subscription_id 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 endpoint 
 ) 
  
 { 
  
 google 
 :: 
 pubsub 
 :: 
 v1 
 :: 
 Subscription 
  
 request 
 ; 
  
 request 
 . 
 set_name 
 ( 
  
 pubsub 
 :: 
 Subscription 
 ( 
 project_id 
 , 
  
 subscription_id 
 ). 
 FullName 
 ()); 
  
 request 
 . 
 set_topic 
 ( 
 pubsub 
 :: 
 Topic 
 ( 
 project_id 
 , 
  
 topic_id 
 ). 
 FullName 
 ()); 
  
 request 
 . 
 mutable_push_config 
 () 
 - 
> set_push_endpoint 
 ( 
 endpoint 
 ); 
  
 request 
 . 
 mutable_push_config 
 () 
 - 
> mutable_no_wrapper 
 () 
 - 
> set_write_metadata 
 ( 
  
 true 
 ); 
  
 auto 
  
 sub 
  
 = 
  
 client 
 . 
 CreateSubscription 
 ( 
 request 
 ); 
  
 if 
  
 ( 
 sub 
 . 
 status 
 (). 
 code 
 () 
  
 == 
  
 google 
 :: 
 cloud 
 :: 
 StatusCode 
 :: 
 kAlreadyExists 
 ) 
  
 { 
  
 std 
 :: 
 cout 
 << 
 "The subscription already exists 
 \n 
 " 
 ; 
  
 return 
 ; 
  
 } 
  
 if 
  
 ( 
 ! 
 sub 
 ) 
  
 throw 
  
 std 
 :: 
 move 
 ( 
 sub 
 ). 
 status 
 (); 
  
 std 
 :: 
 cout 
 << 
 "The subscription was successfully created: " 
 << 
 sub 
 - 
> DebugString 
 () 
 << 
 " 
 \n 
 " 
 ; 
 } 
 

Go

The following sample uses the major version of the Go Pub/Sub client library (v2). If you are still using the v1 library, see the migration guide to v2 . To see a list of v1 code samples, see the deprecated code samples .

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Go API reference documentation .

  import 
  
 ( 
  
 "context" 
  
 "fmt" 
  
 "io" 
  
 "cloud.google.com/go/pubsub/v2" 
  
 "cloud.google.com/go/pubsub/v2/apiv1/pubsubpb" 
 ) 
 // createPushNoWrapperSubscription creates a push subscription where messages are delivered in the HTTP body. 
 func 
  
 createPushNoWrapperSubscription 
 ( 
 w 
  
 io 
 . 
 Writer 
 , 
  
 projectID 
 , 
  
 topic 
 , 
  
 subscription 
 , 
  
 endpoint 
  
 string 
 ) 
  
 error 
  
 { 
  
 // projectID := "my-project-id" 
  
 // topic := "projects/my-project-id/topics/my-topic" 
  
 // subscription := "projects/my-project/subscriptions/my-sub" 
  
 // endpoint := "https://my-test-project.appspot.com/push" 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 client 
 , 
  
 err 
  
 := 
  
 pubsub 
 . 
 NewClient 
 ( 
 ctx 
 , 
  
 projectID 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "pubsub.NewClient: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 client 
 . 
 Close 
 () 
  
 pbSubscription 
  
 := 
  
& pubsubpb 
 . 
 Subscription 
 { 
  
 Name 
 : 
  
 subscription 
 , 
  
 Topic 
 : 
  
 topic 
 , 
  
 AckDeadlineSeconds 
 : 
  
 10 
 , 
  
 PushConfig 
 : 
  
& pubsubpb 
 . 
 PushConfig 
 { 
  
 PushEndpoint 
 : 
  
 endpoint 
 , 
  
 Wrapper 
 : 
  
& pubsubpb 
 . 
 PushConfig_NoWrapper_ 
 { 
  
 NoWrapper 
 : 
  
& pubsubpb 
 . 
 PushConfig_NoWrapper 
 { 
  
 // Determines if message metadata is added to the HTTP headers of 
  
 // the delivered message. 
  
 WriteMetadata 
 : 
  
 true 
 , 
  
 }, 
  
 }, 
  
 }, 
  
 } 
  
 sub 
 , 
  
 err 
  
 := 
  
 client 
 . 
 SubscriptionAdminClient 
 . 
 CreateSubscription 
 ( 
 ctx 
 , 
  
 pbSubscription 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "CreateSubscription: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 fmt 
 . 
 Fprintf 
 ( 
 w 
 , 
  
 "Created push no wrapper subscription: %v\n" 
 , 
  
 sub 
 ) 
  
 return 
  
 nil 
 } 
 

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Node.js API reference documentation .

  /** 
 * TODO(developer): Uncomment these variables before running the sample. 
 */ 
 // const pushEndpoint = 'YOUR_ENDPOINT_URL'; 
 // const topicNameOrId = 'YOUR_TOPIC_NAME_OR_ID'; 
 // const subscriptionNameOrId = 'YOUR_SUBSCRIPTION_NAME_OR_ID'; 
 // Imports the Google Cloud client library 
 const 
  
 { 
 PubSub 
 } 
  
 = 
  
 require 
 ( 
 '@ 
 google 
 - 
 cloud 
 / 
 pubsub 
 ' 
 ); 
 // Creates a client; cache this for further use 
 const 
  
 pubSubClient 
  
 = 
  
 new 
  
 PubSub 
 (); 
 async 
  
 function 
  
 createPushSubscriptionNoWrapper 
 ( 
  
 pushEndpoint 
 , 
  
 topicNameOrId 
 , 
  
 subscriptionNameOrId 
 , 
 ) 
  
 { 
  
 const 
  
 options 
  
 = 
  
 { 
  
 pushConfig 
 : 
  
 { 
  
 // Set to an HTTPS endpoint of your choice. If necessary, register 
  
 // (authorize) the domain on which the server is hosted. 
  
 pushEndpoint 
 , 
  
 // When true, writes the Pub/Sub message metadata to 
  
 // `x-goog-pubsub-<KEY>:<VAL>` headers of the HTTP request. Writes the 
  
 // Pub/Sub message attributes to `<KEY>:<VAL>` headers of the HTTP request. 
  
 noWrapper 
 : 
  
 { 
  
 writeMetadata 
 : 
  
 true 
 , 
  
 }, 
  
 }, 
  
 }; 
  
 await 
  
 pubSubClient 
  
 . 
 topic 
 ( 
 topicNameOrId 
 ) 
  
 . 
 createSubscription 
 ( 
 subscriptionNameOrId 
 , 
  
 options 
 ); 
  
 console 
 . 
 log 
 ( 
 `Subscription ${subscriptionNameOrId} created.` 
 ); 
 } 
 

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Node.js API reference documentation .

  /** 
 * TODO(developer): Uncomment these variables before running the sample. 
 */ 
 // const pushEndpoint = 'YOUR_ENDPOINT_URL'; 
 // const topicNameOrId = 'YOUR_TOPIC_NAME_OR_ID'; 
 // const subscriptionNameOrId = 'YOUR_SUBSCRIPTION_NAME_OR_ID'; 
 // Imports the Google Cloud client library 
 import 
  
 { 
 PubSub 
 , 
  
 CreateSubscriptionOptions 
 } 
  
 from 
  
 '@ 
 google 
 - 
 cloud 
 / 
 pubsub 
 ' 
 ; 
 // Creates a client; cache this for further use 
 const 
  
 pubSubClient 
  
 = 
  
 new 
  
 PubSub 
 (); 
 async 
  
 function 
  
 createPushSubscriptionNoWrapper 
 ( 
  
 pushEndpoint 
 : 
  
 string 
 , 
  
 topicNameOrId 
 : 
  
 string 
 , 
  
 subscriptionNameOrId 
 : 
  
 string 
 , 
 ) 
  
 { 
  
 const 
  
 options 
 : 
  
 CreateSubscriptionOptions 
  
 = 
  
 { 
  
 pushConfig 
 : 
  
 { 
  
 // Set to an HTTPS endpoint of your choice. If necessary, register 
  
 // (authorize) the domain on which the server is hosted. 
  
 pushEndpoint 
 , 
  
 // When true, writes the Pub/Sub message metadata to 
  
 // `x-goog-pubsub-<KEY>:<VAL>` headers of the HTTP request. Writes the 
  
 // Pub/Sub message attributes to `<KEY>:<VAL>` headers of the HTTP request. 
  
 noWrapper 
 : 
  
 { 
  
 writeMetadata 
 : 
  
 true 
 , 
  
 }, 
  
 }, 
  
 }; 
  
 await 
  
 pubSubClient 
  
 . 
 topic 
 ( 
 topicNameOrId 
 ) 
  
 . 
 createSubscription 
 ( 
 subscriptionNameOrId 
 , 
  
 options 
 ); 
  
 console 
 . 
 log 
 ( 
 `Subscription ${subscriptionNameOrId} created.` 
 ); 
 } 
 

Set a content-type header in your message

After enabling payload unwrapping, Pub/Sub doesn't automatically set a media type header field in your request. If you don't explicitly set a Content-Type header field, the web server processing your request might set a default value of application/octet-stream or interpret the request in an unexpected manner.

If you require a Content-Type header, make sure that you explicitly declare it at publish time to every individual published message. To do this, you must first enable Write metadata. This result of enabling Write metadatais shown in the provided examples .

What's next

Design a Mobile Site
View Site in Mobile | Classic
Share by: