This page explains how to publish messages to Lite topics. You can publish messages with the Pub/Sub Lite client library for Java.
After publishing messages and creating a Lite subscription to a Lite topic, you can receive messages from the Lite subscription.
Message format
A message consists of fields with the message data and metadata. Specify any of the following in the message:
- The message data
- An ordering key
- An event time
- Attributes with additional metadata
The client library automatically assigns the message to a partition, and the Pub/Sub Lite service adds the following fields to the message:
- A message ID unique within the partition
- A timestamp for when the Pub/Sub Lite service stores the message in the partition
Publishing messages
To publish messages, request a streaming connection to the Lite topic and then send messages over the streaming connection.
The following sample shows you how to publish messages to a Lite topic:
gcloud
This command requires Python 3.6 or greater, and requires the grpcio Python package to be installed. For MacOS, Linux, and Cloud Shell users, run:
sudo pip3 install grpcioexport CLOUDSDK_PYTHON_SITEPACKAGES = 1
To publish a message, use the gcloud pubsub lite-topics publish command:
gcloud
pubsub
lite-topics
publish
TOPIC_ID
\
--location =
LITE_LOCATION
\
--message =
MESSAGE_DATA
Replace the following:
- TOPIC_ID : the ID of the Lite topic
- LITE_LOCATION : the location of the Lite topic
- MESSAGE_DATA : a string with the message data
Go
Before running this sample, follow the Go setup instructions in Pub/Sub Lite Client Libraries .
Java
Before running this sample, follow the Java setup instructions in Pub/Sub Lite Client Libraries .
Python
Before running this sample, follow the Python setup instructions in Pub/Sub Lite Client Libraries .
The client library asynchronously sends messages and handles errors. If an error occurs, the client library sends the message again.
- The Pub/Sub Lite service closes the stream.
- The client library buffers the messages and reestablishes a connection to the Lite topic.
- The client library sends the messages in order.
After you publish a message, the Pub/Sub Lite service stores the message in a partition and returns the message ID to the publisher.
Using ordering keys
If messages have the same ordering key, the client library assigns the messages to the same partition. The ordering key must be a string of at most 1,024 bytes.
The ordering key is in the key
field of a message
.
You can set ordering keys with the client library.
gcloud
This command requires Python 3.6 or greater, and requires the grpcio Python package to be installed. For MacOS, Linux, and Cloud Shell users, run:
sudo pip3 install grpcioexport CLOUDSDK_PYTHON_SITEPACKAGES = 1
To publish a message, use the gcloud pubsub lite-topics publish command:
gcloud
pubsub
lite-topics
publish
TOPIC_ID
\
--location =
LITE_LOCATION
\
--ordering-key =
ORDERING_KEY
\
--message =
MESSAGE_DATA
Replace the following:
- TOPIC_ID : the ID of the Lite topic
- LITE_LOCATION : the location of the Lite topic
- ORDERING_KEY : a string used to assign messages to partitions
- MESSAGE_DATA : a string with the message data
Go
Before running this sample, follow the Go setup instructions in Pub/Sub Lite Client Libraries .
Java
Before running this sample, follow the Java setup instructions in Pub/Sub Lite Client Libraries .
Python
Before running this sample, follow the Python setup instructions in Pub/Sub Lite Client Libraries .
You can send multiple messages to the same partition using ordering keys, so subscribers receive the messages in order . The client library might assign multiple ordering keys to the same partition.
Set the event time
You can use event time to publish your Lite messages. Event time is a custom attribute that you can add to your message.
You can set the event timestamp with the client library or the gcloud CLI.
This command requires Python 3.6 or greater, and requires the grpcio Python package to be installed. For MacOS, Linux, and Cloud Shell users, run:
sudo pip3 install grpcioexport CLOUDSDK_PYTHON_SITEPACKAGES = 1
To publish a message, use the gcloud pubsub lite-topics publish command:
gcloud
pubsub
lite-topics
publish
TOPIC_ID
\
--location =
LITE_LOCATION
\
--event-time =
EVENT_TIME
\
--message =
MESSAGE_DATA
Replace the following:
-
TOPIC_ID : the ID of the Lite topic
-
LITE_LOCATION : the location of the Lite topic
-
EVENT_TIME : a user-specified event time. For more information about time formats, run
gcloud topic datetimes. -
MESSAGE_DATA : a string with the message data
Using attributes
Message attributes are key-value pairs with metadata about the message. The attributes can be text or byte strings.
The attributes are in the attributes
field
of a message. You can set attributes with the client library.
gcloud
This command requires Python 3.6 or greater, and requires the grpcio Python package to be installed. For MacOS, Linux, and Cloud Shell users, run:
sudo pip3 install grpcioexport CLOUDSDK_PYTHON_SITEPACKAGES = 1
To publish a message, use the gcloud pubsub lite-topics publish command:
gcloud
pubsub
lite-topics
publish
TOPIC_ID
\
--location =
LITE_LOCATION
\
--message =
MESSAGE_DATA
\
--attribute =
KEY
=
VALUE
,...
Replace the following:
- TOPIC_ID : the ID of the Lite topic
- LITE_LOCATION : the location of the Lite topic
- MESSAGE_DATA : a string with the message data
- KEY : the key of a message attribute
- VALUE : the value for the key of the message attribute
Go
Before running this sample, follow the Go setup instructions in Pub/Sub Lite Client Libraries .
Java
Before running this sample, follow the Java setup instructions in Pub/Sub Lite Client Libraries .
Python
Before running this sample, follow the Python setup instructions in Pub/Sub Lite Client Libraries .
Attributes can indicate how to process a message. Subscribers can parse the attributes
field of a message and process the message according to its
attributes.
Batching messages
The client library publishes messages in batches. Larger batches use fewer compute resources but increase latency. You can change the batch size with batching settings.
The following table lists the batching settings that you can configure:
| Setting | Description | Default |
|---|---|---|
|
Request size
|
The maximum size, in bytes, of the batch. | 3.5 MiB |
|
Number of messages
|
The maximum number of messages in a batch. | 1,000 messages |
|
Publish delay
|
The amount of time, in milliseconds, between adding the message to a batch and sending the batch to the Lite topic. | 50 milliseconds |
You can configure batching settings with the client library.
Go
Before running this sample, follow the Go setup instructions in Pub/Sub Lite Client Libraries .
Java
Before running this sample, follow the Java setup instructions in Pub/Sub Lite Client Libraries .
Python
Before running this sample, follow the Python setup instructions in Pub/Sub Lite Client Libraries .
When a publisher application starts, the client library creates a batch for each partition in a Lite topic. For example, if a Lite topic has two partitions, publishers create two batches and send each batch to a partition.
After you publish a message, the client library buffers it until the batch exceeds the maximum request size, the maximum number of messages, or the publish delay.
Ordering messages
Lite topics order messages in each partition by when you publish the messages. To assign messages to the same partition, use an ordering key .
Pub/Sub Lite delivers the messages from a partition in order, and subscribers can process the messages in order. For details, see Receiving messages .
Publish Idempotency
Pub/Sub Lite client libraries support idempotent publishing, from the following versions:
- java-pubsublite : version 1.10.0.
- python-pubsublite : version 1.8.0.
- google-cloud-go : pubsublite version 1.7.0.
If publishing of a message is retried due to network or server errors, it is stored exactly once. Idempotency is guaranteed only within the same session; it cannot be guaranteed if the same message is republished using a new publisher client. It does not incur any additional service costs or increase publish latency.
Enable or disable idempotent publishing
Idempotent publishing is enabled by default in the Pub/Sub Lite client libraries. It can be disabled using publisher client settings in the respective client library.
If idempotent publishing is enabled, the offset returned in a publish result
might be -1
. This value is returned when the message is identified as a
duplicate of an already successfully published message, but the server did not
have sufficient information to return the offset of the message at publish time.
Messages received by subscribers always have a valid offset.
Troubleshooting
Duplicates received
As idempotency is limited to a single session, duplicates might be received if you recreate the publisher client to publish the same messages.
A subscriber client might receive the same message multiple times if partitions are automatically assigned to subscribers by the Pub/Sub Lite service (the default setting). A message might be redelivered to another subscriber client when a reassignment occurs.
Publisher error
State for a publisher session is garbage collected in the server after 7 days of inactivity. If a session is resumed after this time period, the publisher client terminates with an error message similar to "Failed Precondition: Expected message to have publish sequence number of..." and does not accept new messages. Recreate the publisher client to resolve this error.
