Bigtable subscriptions let you write Pub/Sub messages to a Bigtable table. This document provides an overview of Bigtable subscriptions, including the subscription workflow, delivery semantics, and error handling.
To create a Bigtable subscription, see Create a Bigtable subscription .
Overview
A Bigtable subscription is a type of export subscription that writes messages to an existing Bigtable table as they are received. By using an export subscription, you don't need to configure a separate subscriber client.
Consider using a Bigtable subscription if you want to write messages to Bigtable, and the messages don't need additional processing, or the processing can be performed by a Single Message Transform (SMT).
If you need to perform more complex transformations before writing messages to Bigtable, we recommend using a Dataflow pipeline with a pull subscription. For more information, see Read from Pub/Sub to Dataflow .
For more information about choosing a Pub/Sub subscription type, see Choose a subscription type .
Bigtable subscription workflow
The following image shows the workflow between a Bigtable subscription and Bigtable.
The workflow consists of the following steps:
-
A publisher sends a message to a Pub/Sub topic.
-
Pub/Sub forwards the message to the Bigtable subscription.
-
The subscription sends a write request to Bigtable. In the write request, the subscription specifies an app profile that uses single-cluster routing .
-
Bigtable routes the message to the Bigtable cluster that contains the table.
-
The write request is committed to the table.
-
After a successful completion of a write operation, the Bigtable API returns a success response.
If a write operation fails, it's handled as follows:
-
The Pub/Sub message is negatively acknowledged.
-
Pub/Sub resends the message.
-
If the message fails enough times, and a dead-letter topic is configured on the subscription, then the message is moved to the dead-letter topic.
Delivery semantics
Bigtable subscriptions support at-least-once delivery. If you require exact de-duplication of messages, consider implementing downstream processes to handle potential duplicates in Bigtable, or else use Dataflow's exactly-once processing capabilities.
Handle message failures
If a Pub/Sub message can't be written to Bigtable,
the message can't be acknowledged. To forward such undeliverable messages,
configure a dead-letter topic
on the
Bigtable subscription. The Pub/Sub message
forwarded to the dead-letter topic contains an attribute named CloudPubSubDeadLetterSourceDeliveryErrorMessage
that contains the reason for
the write failure.
If Pub/Sub can't write messages to Bigtable, then Pub/Sub backs off delivery of messages in a way similar to push backoff behavior.
Quotas and limits
Pub/Sub has quota limitations on the Bigtable subscriber throughput per region. For more information, see Pub/Sub quotas and limits .
For information about Bigtable limits, see Quotas and limits in the Bigtable documentation.
Pricing
To learn about the pricing for Bigtable subscriptions, see the Pub/Sub pricing page.
