Stream changes to Pub/Sub using optional Cloud Run function trigger

Create a Pub/Sub topic

1. In the Google Cloud console, go to the Pub/Sub Topicspage.

   Go to Topics

2. Click Create topic.

3. Set the ID to bigtable-change-stream-topic .

4. Select Use a schema.

5. In the Select a Pub/Sub schemadrop-down, click Create new schema. This opens a new tab where you define the schema.

   1. Set the schema ID to bigtable-change-stream-schema .
   2. Set the schema type to Avro .
   3. Paste the following as the schema definition. More information about the schema can be found on the template documentation page .

      
    { 
     
    "name" 
     
    : 
     
    "ChangelogEntryMessage" 
    , 
     
    "type" 
     
    : 
     
    "record" 
    , 
     
    "namespace" 
     
    : 
     
    "com.google.cloud.teleport.bigtable" 
    , 
     
    "fields" 
     
    : 
     
    [ 
     
    { 
     
    "name" 
     
    : 
     
    "rowKey" 
    , 
     
    "type" 
     
    : 
     
    "bytes" 
    }, 
     
    { 
     
    "name" 
     
    : 
     
    "modType" 
    , 
     
    "type" 
     
    : 
     
    { 
     
    "name" 
    : 
     
    "ModType" 
    , 
     
    "type" 
    : 
     
    "enum" 
    , 
     
    "symbols" 
    : 
     
    [ 
    "SET_CELL" 
    , 
     
    "DELETE_FAMILY" 
    , 
     
    "DELETE_CELLS" 
    , 
     
    "UNKNOWN" 
    ]} 
     
    }, 
     
    { 
     
    "name" 
    : 
     
    "isGC" 
    , 
     
    "type" 
    : 
     
    "boolean" 
     
    }, 
     
    { 
     
    "name" 
    : 
     
    "tieBreaker" 
    , 
     
    "type" 
    : 
     
    "int" 
    }, 
     
    { 
     
    "name" 
    : 
     
    "columnFamily" 
    , 
     
    "type" 
    : 
     
    "string" 
    }, 
     
    { 
     
    "name" 
    : 
     
    "commitTimestamp" 
    , 
     
    "type" 
     
    : 
     
    "long" 
    }, 
     
    { 
     
    "name" 
     
    : 
     
    "sourceInstance" 
    , 
     
    "type" 
     
    : 
     
    "string" 
    }, 
     
    { 
     
    "name" 
     
    : 
     
    "sourceCluster" 
    , 
     
    "type" 
     
    : 
     
    "string" 
    }, 
     
    { 
     
    "name" 
     
    : 
     
    "sourceTable" 
    , 
     
    "type" 
     
    : 
     
    "string" 
    }, 
     
    { 
     
    "name" 
    : 
     
    "column" 
    , 
     
    "type" 
     
    : 
     
    [ 
    "null" 
    , 
     
    "bytes" 
    ]}, 
     
    { 
     
    "name" 
    : 
     
    "timestamp" 
    , 
     
    "type" 
     
    : 
     
    [ 
    "null" 
    , 
     
    "long" 
    ]}, 
     
    { 
     
    "name" 
    : 
     
    "timestampFrom" 
    , 
     
    "type" 
     
    : 
     
    [ 
    "null" 
    , 
     
    "long" 
    ]}, 
     
    { 
     
    "name" 
    : 
     
    "timestampTo" 
    , 
     
    "type" 
     
    : 
     
    [ 
    "null" 
    , 
     
    "long" 
    ]}, 
     
    { 
     
    "name" 
     
    : 
     
    "value" 
    , 
     
    "type" 
     
    : 
     
    [ 
    "null" 
    , 
     
    "bytes" 
    ]} 
     
    ] 
     
    } 
    
   

   1. Click Createto create the schema.

6. Close the Create schematab, refresh the schema list, and select your newly defined schema.

7. Click Createto create the topic.

Optional: Create a Cloud Run function

You might want to process the Pub/Sub stream with a Cloud Run function.

1. On the Detailspage for the bigtable-change-stream-topic topic, click Trigger Cloud Function.
2. In the Function namefield, enter the name bt-ps-tutorial-function .
3. In the Source Codesection, click the Runtimedrop-down, and then select the runtime and programming language of your choice. A hello world is generated that prints out the change stream as it comes in. See the documentation to learn more about writing Cloud Run functions .
4. Use the default values for all other fields.
5. Click Deploy function.

Create a table with a change stream enabled

1. In the Google Cloud console, go to the Bigtable Instancespage.

   Go to Instances

2. Click the ID of the instance that you are using for this tutorial.

   If you don't have an instance available, create an instance with the default configurations in a region near you.

3. In the left navigation pane, click Tables.

4. Click Create a table.

5. Name the table change-streams-pubsub-tutorial .

6. Add a column family named cf .

7. Select Enable change stream.

8. Click Create.

Initialize a data pipeline to capture the change stream

1. On the Bigtable Tablespage, find your table change-streams-pubsub-tutorial .
2. In the Change streamcolumn, click Connect.
3. In the dialog, select Pub/Sub.
4. Click Create Dataflow job.
5. On the Dataflow Create jobpage, set the output Pub/Sub topic name to: bigtable-change-stream-topic .
6. Set the Bigtable application profile ID to default .
7. Click Run job.
8. Wait until the job status is Startingor Runningbefore proceeding. It takes around 5 minutes once the job is queued.

Write some data to Bigtable

1. In the Cloud Shell, write a few rows to Bigtable so the change log can write some data to the Pub/Sub stream. As long as you write the data after the job is created, the changes appear. You don't have to wait for the job status to become running .

    cbt  
   -instance = 
    BIGTABLE_INSTANCE_ID 
     
   -project = 
    YOUR_PROJECT_ID 
     
    \ 
     
    set 
     
   change-streams-pubsub-tutorial  
   user123  
   cf:col1 = 
   abc
   cbt  
   -instance = 
    BIGTABLE_INSTANCE_ID 
     
   -project = 
    YOUR_PROJECT_ID 
     
    \ 
     
    set 
     
   change-streams-pubsub-tutorial  
   user546  
   cf:col1 = 
   def
   cbt  
   -instance = 
    BIGTABLE_INSTANCE_ID 
     
   -project = 
    YOUR_PROJECT_ID 
     
    \ 
     
    set 
     
   change-streams-pubsub-tutorial  
   user789  
   cf:col1 = 
   ghi 
   

View the change logs in Pub/Sub

1. In the Google Cloud console, go to the Pub/Sub Subscriptionspage.

   Go to Subscriptions

2. Click the automatically created subscription for your topic bigtable-change-stream-topic . It should be named bigtable-change-stream-topic-sub .

3. Go to to the Messagestab.

4. Click Pull.

5. Explore the list of messages and view the data that you wrote.

   

Optional: View the changes in the Cloud Run functions logs

If you created a Cloud Run functions function, you can view the changes in the logs.

1. In the Google Cloud console, go to Cloud Run functions.

   Go to Cloud Run functions

2. Click your function bt-ps-tutorial-function .

3. Go to the Logstab.

4. Ensure that Severityis set to at least Infoso you can see the logs.

5. Explore the logs and view the data that you wrote.

The output looks similar to the following:

 Pub/Sub message: {"rowKey":"user789","modType":"SET_CELL","isGC":false,"tieBreaker":0,"columnFamily":"cf","commitTimestamp":1695653833064548,"sourceInstance":"YOUR-INSTANCE","sourceCluster":"YOUR-INSTANCE-c1","sourceTable":"change-streams-pubsub-tutorial","column":{"bytes":"col1"},"timestamp":{"long":1695653832278000},"timestampFrom":null,"timestampTo":null,"value":{"bytes":"ghi"}}