Console
    Start free

    Create a subscription with SMTs

    This document explains how to create a Pub/Sub subscription with Single Message Transforms (SMTs).

    Subscription SMTs allow for lightweight modifications to message data and attributes directly within Pub/Sub. This feature enables data cleaning, filtering, or format conversion before the messages are delivered to a subscriber client.

    To create a subscription with SMTs, you can use the Google Cloud console, the Google Cloud CLI, the client library, or the Pub/Sub API.

    Before you begin

    Required roles and permissions

    To get the permissions that you need to create a subscription with SMTs, ask your administrator to grant you the Pub/Sub Editor ( roles/pubsub.editor ) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations .

    This predefined role contains the permissions required to create a subscription with SMTs. To see the exact permissions that are required, expand the Required permissionssection:

    Required permissions

    The following permissions are required to create a subscription with SMTs:

    • Grant the create a subscription permission on the project: pubsub.subscriptions.create

    You might also be able to get these permissions with custom roles or other predefined roles .

    Depending on the subscription type, you might need additional permissions. To find out the exact list of permissions, refer to the document that discusses creating the specific subscription. For example, if you are creating a BigQuery subscription with SMTs, see Create BigQuery subscriptions .

    If you create a subscription in a different project than the topic, you must grant the roles/pubsub.subscriber role to the principal of the project containing the subscription in the project containing the topic.

    You can configure access control at the project level and at the individual resource level.

    Create a subscription with SMTs

    Before you create a subscription with SMTs, review the documentation for Properties of a subscription .

    To create a Pub/Sub subscription with one or more SMTs, perform the following steps. You can enable up to 5 SMTs per subscription.

    Console

    1. In the Google Cloud console, go to the Pub/Sub Subscriptionspage.

      Go to Subscriptions

    2. Click Create subscription.

    3. In the Subscription IDfield, enter an ID for your subscription. For more information about naming subscriptions, see the naming guidelines .

    4. Under Transforms, click Add a transform.

    5. Select the Transform type. For more information about the supported SMT types, see Types of SMTs .

    6. Set the configuration properties for the SMT. The set of properties depends on the type of SMT. For more information, see the documentation for that SMT type.

    7. Optional. To validate the SMT, click Validate. If the SMT is valid, the message "Validation passed" is displayed. Otherwise, an error message is displayed.

    8. To add another transform, click Add a transformand repeat the previous steps.

      To arrange the SMTs in a specific order, click Move upor Move down. To remove an SMT, click Delete.

    9. Optional. To test an SMT on a sample message, perform the following steps:

      1. Click Test transforms.

      2. In the Test transformwindow, select the function that you want to test.

      3. In the Input messagewindow, enter a sample message.

      4. To add an attribute to the message, click Add an attributeand enter the attribute's key and value. You can add multiple attributes.

      5. Click Test. The result of applying the SMT on the message is displayed under Output message.

      6. To close the Test transformswindow, click Close.

      If you create more than one SMT, you can test the entire sequence of transforms as follows:

      1. Test the first SMT in the sequence, as described in the previous steps.
      2. Select the next SMT. The input message is pre-populated with the output message from the previous test.
      3. Continue testing the SMTs in order, to make sure the entire sequence works as expected.

    10. Click Createto create the subscription.

    gcloud

    1. In the Google Cloud console, activate Cloud Shell.

      Activate Cloud Shell

      At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

    2. Create a YAML or JSON file that defines one or more SMTs. The YAML or JSON definition depends on the type of SMT. For more information, see Types of SMTs .

      If the file includes more than one SMT, Pub/Sub executes them in the order listed.

    3. Optional. To validate an SMT, run the gcloud pubsub message-transforms validate command:

       gcloud  
pubsub  
message-transforms  
validate  
 \ 
  
--message-transform-file = 
 TRANSFORM_FILE

      Replace the following:

      • TRANSFORM_FILE : The path to a YAML or JSON file that defines a single SMT. If you are creating multiple SMTs, you must validate them individually.

    4. Optional. To test one or more SMTs on a sample Pub/Sub message, run the gcloud pubsub message-transforms test command:

       gcloud  
pubsub  
message-transforms  
 test 
  
 \ 
  
--message-transforms-file = 
 TRANSFORMS_FILE 
  
 \ 
  
--message = 
 MESSAGE 
  
 \ 
  
--attribute = 
 ATTRIBUTES

      Replace the following:

      • TRANSFORMS_FILE : The path to a YAML or JSON file that defines one or more SMTs.
      • MESSAGE : The body of the sample message.
      • ATTRIBUTES : Optional. A comma-separated list of message attributes. Each attribute is a key-value pair formatted as KEY =" VALUE " .

      The command executes the SMTs in order, using the output from each SMT as the input to the next. The command outputs the results of each step.

    5. To create the subscription, run the gcloud pubsub subscriptions create command:

       gcloud  
pubsub  
subscriptions  
create  
 SUBSCRIPTION_ID 
  
 \ 
  
--topic = 
projects/ PROJECT_ID 
/topics/ TOPIC_ID 
  
 \ 
  
--message-transforms-file = 
 TRANSFORMS_FILE

      Replace the following:

      • SUBSCRIPTION_ID : The ID or name of the subscription you want to create. For guidelines on how to name a subscription, see Resource names . The name of a subscription is immutable.

      • PROJECT_ID : The ID of the project that contains the topic.

      • TOPIC_ID : The ID of the topic to subscribe to.

      • TRANSFORMS_FILE : The path to the YAML or JSON file that defines one or more SMTs.

    Java

    Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Java API reference documentation .

      import 
  
 com.google.cloud.pubsub.v1. SubscriptionAdminClient 
 
 ; 
 import 
  
 com.google.pubsub.v1. JavaScriptUDF 
 
 ; 
 import 
  
 com.google.pubsub.v1. MessageTransform 
 
 ; 
 import 
  
 com.google.pubsub.v1. ProjectSubscriptionName 
 
 ; 
 import 
  
 com.google.pubsub.v1. ProjectTopicName 
 
 ; 
 import 
  
 com.google.pubsub.v1. Subscription 
 
 ; 
 import 
  
 java.io.IOException 
 ; 
 public 
  
 class 
 CreateSubscriptionWithSmtExample 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 ... 
  
 args 
 ) 
  
 throws 
  
 Exception 
  
 { 
  
 // TODO(developer): Replace these variables before running the sample. 
  
 String 
  
 projectId 
  
 = 
  
 "your-project-id" 
 ; 
  
 String 
  
 topicId 
  
 = 
  
 "your-topic-id" 
 ; 
  
 String 
  
 subscriptionId 
  
 = 
  
 "your-subscription-id" 
 ; 
  
 createSubscriptionWithSmtExample 
 ( 
 projectId 
 , 
  
 topicId 
 , 
  
 subscriptionId 
 ); 
  
 } 
  
 public 
  
 static 
  
 void 
  
 createSubscriptionWithSmtExample 
 ( 
  
 String 
  
 projectId 
 , 
  
 String 
  
 topicId 
 , 
  
 String 
  
 subscriptionId 
 ) 
  
 throws 
  
 IOException 
  
 { 
  
 // UDF that removes the 'ssn' field, if present 
  
 String 
  
 code 
  
 = 
  
 "function redactSSN(message, metadata) {" 
  
 + 
  
 "  const data = JSON.parse(message.data);" 
  
 + 
  
 "  delete data['ssn'];" 
  
 + 
  
 "  message.data = JSON.stringify(data);" 
  
 + 
  
 "  return message;" 
  
 + 
  
 "}" 
 ; 
  
 String 
  
 functionName 
  
 = 
  
 "redactSSN" 
 ; 
  
  JavaScriptUDF 
 
  
 udf 
  
 = 
  
  JavaScriptUDF 
 
 . 
 newBuilder 
 (). 
  setCode 
 
 ( 
 code 
 ). 
  setFunctionName 
 
 ( 
 functionName 
 ). 
 build 
 (); 
  
  MessageTransform 
 
  
 transform 
  
 = 
  
  MessageTransform 
 
 . 
 newBuilder 
 (). 
  setJavascriptUdf 
 
 ( 
 udf 
 ). 
 build 
 (); 
  
 try 
  
 ( 
  SubscriptionAdminClient 
 
  
 subscriptionAdminClient 
  
 = 
  
  SubscriptionAdminClient 
 
 . 
 create 
 ()) 
  
 { 
  
  ProjectTopicName 
 
  
 topicName 
  
 = 
  
  ProjectTopicName 
 
 . 
 of 
 ( 
 projectId 
 , 
  
 topicId 
 ); 
  
  ProjectSubscriptionName 
 
  
 subscriptionName 
  
 = 
  
  ProjectSubscriptionName 
 
 . 
 of 
 ( 
 projectId 
 , 
  
 subscriptionId 
 ); 
  
  Subscription 
 
  
 subscription 
  
 = 
  
 subscriptionAdminClient 
 . 
 createSubscription 
 ( 
  
  Subscription 
 
 . 
 newBuilder 
 () 
  
 . 
 setName 
 ( 
 subscriptionName 
 . 
  toString 
 
 ()) 
  
 . 
 setTopic 
 ( 
 topicName 
 . 
  toString 
 
 ()) 
  
 // Add the UDF message transform 
  
 . 
 addMessageTransforms 
 ( 
 transform 
 ) 
  
 . 
 build 
 ()); 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "Created subscription with SMT: " 
  
 + 
  
 subscription 
 . 
 getAllFields 
 ()); 
  
 } 
  
 } 
 }

    Python

    Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Python API reference documentation .

      from 
  
 google.cloud 
  
 import 
 pubsub_v1 
 from 
  
 google.pubsub_v1.types 
  
 import 
  JavaScriptUDF 
 
 , 
  MessageTransform 
 
 # TODO(developer): Choose an existing topic. 
 # project_id = "your-project-id" 
 # topic_id = "your-topic-id" 
 # subscription_id = "your-subscription-id" 
 publisher 
 = 
 pubsub_v1 
 . 
  PublisherClient 
 
 () 
 subscriber 
 = 
 pubsub_v1 
 . 
  SubscriberClient 
 
 () 
 topic_path 
 = 
 publisher 
 . 
 topic_path 
 ( 
 project_id 
 , 
 topic_id 
 ) 
 subscription_path 
 = 
 subscriber 
 . 
 subscription_path 
 ( 
 project_id 
 , 
 subscription_id 
 ) 
 code 
 = 
 """function redactSSN(message, metadata) { 
 const data = JSON.parse(message.data); 
 delete data['ssn']; 
 message.data = JSON.stringify(data); 
 return message; 
 }""" 
 udf 
 = 
 JavaScriptUDF 
 ( 
 code 
 = 
 code 
 , 
 function_name 
 = 
 "redactSSN" 
 ) 
 transforms 
 = 
 [ 
 MessageTransform 
 ( 
 javascript_udf 
 = 
 udf 
 )] 
 with 
 subscriber 
 : 
 subscription 
 = 
 subscriber 
 . 
 create_subscription 
 ( 
 request 
 = 
 { 
 "name" 
 : 
 subscription_path 
 , 
 "topic" 
 : 
 topic_path 
 , 
 "message_transforms" 
 : 
 transforms 
 , 
 } 
 ) 
 print 
 ( 
 f 
 "Created subscription with SMT: 
 { 
 subscription 
 } 
 " 
 )

    Go

    The following sample uses the major version of the Go Pub/Sub client library (v2). If you are still using the v1 library, see the migration guide to v2 . To see a list of v1 code samples, see the deprecated code samples .

    Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Go API reference documentation .

      import 
  
 ( 
  
 "context" 
  
 "fmt" 
  
 "io" 
  
 "cloud.google.com/go/pubsub/v2" 
  
 "cloud.google.com/go/pubsub/v2/apiv1/pubsubpb" 
 ) 
 // createSubscriptionWithSMT creates a subscription with a single message transform function applied. 
 func 
  
 createSubscriptionWithSMT 
 ( 
 w 
  
 io 
 . 
 Writer 
 , 
  
 projectID 
 , 
  
 topicID 
 , 
  
 subID 
  
 string 
 ) 
  
 error 
  
 { 
  
 // projectID := "my-project-id" 
  
 // topicID := "my-topic" 
  
 // subID := "my-sub" 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 client 
 , 
  
 err 
  
 := 
  
 pubsub 
 . 
 NewClient 
 ( 
 ctx 
 , 
  
 projectID 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "pubsub.NewClient: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 client 
 . 
 Close 
 () 
  
 code 
  
 := 
  
 `function redactSSN(message, metadata) { 
 const data = JSON.parse(message.data); 
 delete data['ssn']; 
 message.data = JSON.stringify(data); 
 return message; 
 }` 
  
 transform 
  
 := 
  
& pubsubpb 
 . 
 MessageTransform 
 { 
  
 Transform 
 : 
  
& pubsubpb 
 . 
 MessageTransform_JavascriptUdf 
 { 
  
 JavascriptUdf 
 : 
  
& pubsubpb 
 . 
 JavaScriptUDF 
 { 
  
 FunctionName 
 : 
  
 "redactSSN" 
 , 
  
 Code 
 : 
  
 code 
 , 
  
 }, 
  
 }, 
  
 } 
  
 sub 
  
 := 
  
& pubsubpb 
 . 
 Subscription 
 { 
  
 Name 
 : 
  
 fmt 
 . 
 Sprintf 
 ( 
 "projects/%s/subscriptions/%s" 
 , 
  
 projectID 
 , 
  
 subID 
 ), 
  
 Topic 
 : 
  
 fmt 
 . 
 Sprintf 
 ( 
 "projects/%s/topics/%s" 
 , 
  
 projectID 
 , 
  
 topicID 
 ), 
  
 MessageTransforms 
 : 
  
 [] 
 * 
 pubsubpb 
 . 
 MessageTransform 
 { 
 transform 
 }, 
  
 } 
  
 sub 
 , 
  
 err 
  
 = 
  
 client 
 . 
 SubscriptionAdminClient 
 . 
 CreateSubscription 
 ( 
 ctx 
 , 
  
 sub 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "CreateSubscription: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 fmt 
 . 
 Fprintf 
 ( 
 w 
 , 
  
 "Created subscription with message transform: %v\n" 
 , 
  
 sub 
 ) 
  
 return 
  
 nil 
 }

    How SMTs interact with other subscription features

    Consider the following points when using a subscription SMT.

    Filtering

    If your subscription uses both SMTs and Pub/Sub's built-in filters , then the filter gets applied before the SMT. This has the following implications:

    • If your SMT alters the message attributes, the Pub/Sub filter isn't applied to the new set of attributes.
    • Your SMT isn't applied to messages that are filtered out by the Pub/Sub filter.
    • If your SMT filters out messages, be aware of the impact on monitoring your subscription backlog .
    • If you connect a subscription to a Dataflow pipeline , don't use a subscription SMT to filter out messages, because it disrupts Dataflow's autoscaling.

    Message ordering

    If you define an SMT on a subscription that has ordering enabled, and executing the SMT throws an error, then subsequent messages for the same ordering key aren't delivered to the subscriber. To avoid this problem, set up a dead-letter topic on the subscription to remove the unprocessed message from the message backlog.

    What's next
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: