This document shows you how to associate schemas for Pub/Sub topics.
Before you begin
- Understand how Pub/Sub schemas work .
- Create a schema .
Required roles and permissions
To get the permissions that
you need to associate and manage schemas,
ask your administrator to grant you the Pub/Sub Editor
( roles/pubsub.editor
)
IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations
.
This predefined role contains the permissions required to associate and manage schemas. To see the exact permissions that are required, expand the Required permissionssection:
Required permissions
The following permissions are required to associate and manage schemas:
- Create schema:
pubsub.schemas.create
- Attach schema to topic:
pubsub.schemas.attach
- Commit a schema revision:
pubsub.schemas.commit
- Delete a schema or a schema revision:
pubsub.schemas.delete
- Get a schema or schema revisions:
pubsub.schemas.get
- List schemas:
pubsub.schemas.list
- List schema revisions:
pubsub.schemas.listRevisions
- Rollback a schema:
pubsub.schemas.rollback
- Validate a message:
pubsub.schemas.validate
- Get the IAM policy for a schema:
pubsub.schemas.getIamPolicy
- Configure the IAM policy
for a schema:
pubsub.schemas.setIamPolicy
You might also be able to get these permissions with custom roles or other predefined roles .
You can grant roles and permissions to principals such as users, groups, domains, or service accounts. You can create a schema in one project and attach it to a topic located in a different project. Ensure that you have the required permissions for each project.
Guidelines to associate a schema with a topic
You can associate a schema with a topic when you create or edit a topic. Here are the guidelines for associating a schema with a topic:
-
You can associate a schema with one or more topics.
After a schema is associated with a topic, every message that the topic receives from the publishers must follow that schema.
-
When you associate a schema with a topic, you must also specify the encoding of the messages to be published as
BINARY
orJSON
. If using JSON with an Avro schema, pay close attention to the encoding rules for unions . -
If a schema associated with a topic has revisions, messages must match the encoding and validate against a revision within the available range. If they don't validate, the message fails to publish.
Revisions are tried in reverse chronological order that is based on creation time. To create a schema revision, see Commit a schema revision .
Validation logic for a message schema
When you associate a schema with a topic and if the schema has revisions, you can specify a subset range of revisions to use. If you don't specify a range, the entire range is used for validation.
If you don't specify a revision as First revision allowed, then the oldest existing revision for the schema is used for validation. If you don't specify a revision as Last revision allowed, then the newest existing revision for the schema is used.
Let's take the example of Schema S
that is attached to topic T
.
Schema S
has the revision IDs A
, B
, C
, and D
created in order,
where A
is the first or oldest revision. None of the schemas are identical
to each other, or rollbacks of an existing schema.
-
If you only set the First revision allowedfield as
B
, messages conforming only to schemaA
are rejected, while messages conforming to schemasB
,C
, andD
are accepted. -
If you only set the Last revision allowedfield as
C
, messages that conform to schemasA
,B
, andC
are accepted, and messages that conform only to schemaD
are rejected. -
If you set both fields First revision allowedas
B
and Last revision allowedasC
, messages that conform to schemasB
andC
are accepted. -
You can also set the first and last revision to the same revision ID. In this case, only messages that conform to that revision are accepted.
Create and associate a schema when you create a topic
You can create a topic with a schema using the Google Cloud console, the gcloud CLI, the Pub/Sub API, or the Cloud Client Libraries.
Console
-
In the Google Cloud console, go to the Pub/Sub topicspage.
-
Click Create topic.
-
In the Topic IDfield, enter an ID for your topic.
To name a topic, see the guidelines .
-
Check the box for Use a schema.
Retain the default settings for the remaining fields.
You can create a schema or use an existing schema.
-
If you are creating a schema, follow these steps: `
- For Select a Pub/Sub schema, select Create a new schema.
The Create schemapage is displayed in a secondary tab.
Follow the steps in Create a schema .
-
Go back to the Create topictab and click Refresh.
-
Search for your schema in the Select a Pub/Sub schemafield.
-
Select the message encoding as JSONor Binary.
The schema you just created has a revision ID. You can create additional schema revisions as discussed in Commit a schema revision .
-
If you are associating an already-created schema, follow these steps:
-
For Select a Pub/Sub schema, select an existing schema.
-
Select the message encoding as JSONor Binary.
-
-
Optional: If the selected schema has revisions, for Revision Range, use the drop-down menus for First revision allowedand Last revision allowed.
You can specify both fields, specify only one, or retain the default settings based on your requirements.
-
Retain the default settings for the remaining fields.
-
Click Createto save the topic and assign it to the selected schema.
gcloud
To create a topic assigned with a previously created schema, run the gcloud pubsub topics create
command:
gcloud pubsub topics create TOPIC_ID \ --message-encoding = ENCODING_TYPE \ --schema = SCHEMA_ID \ --first-revision-id = FIRST_REVISION_ID \ --last-revision-id = LAST_REVISION_ID \
Where:
- TOPIC_ID is the ID for the topic that you're creating.
- ENCODING_TYPE
is the encoding of messages validated against the
schema. This value must be set to either
JSON
orBINARY
. - SCHEMA_ID is the ID of an existing schema.
- FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both --first-revision-id
and --last-revision-id
are optional.
You can also assign a schema from a different Google Cloud project:
gcloud pubsub topics create TOPIC_ID \ --message-encoding = ENCODING_TYPE \ --schema = SCHEMA_ID \ --schema-project = SCHEMA_PROJECT \ --project = TOPIC_PROJECT
Where:
- SCHEMA_PROJECT is the project ID of the Google Cloud project for the schema.
- TOPIC_PROJECT is the project ID of the Google Cloud project for the topic.
REST
To create a topic, use the projects.topics.create
method:
Request:
The request must be authenticated with an access token in the Authorization
header. To obtain an access token for the current
Application Default Credentials: gcloud auth application-default print-access-token
.
PUT https://pubsub.googleapis.com/v1/projects/ PROJECT_ID /topics/ TOPIC_ID Authorization: Bearer ACCESS_TOKEN
Request body:
{ "schemaSettings" : { "schema" : " SCHEMA_NAME " , "encoding" : " ENCODING_TYPE " "firstRevisionId" : " FIRST_REVISION_ID " "lastRevisionId" : " LAST_REVISION_ID " } }
Where:
- PROJECT_ID is your project ID.
- TOPIC_ID is your topic ID.
- SCHEMA_NAME
is the name of the schema that messages published should be validated against. The format is:
projects/ PROJECT_ID /schemas/ SCHEMA_ID
. - ENCODING_TYPE
is the encoding of messages validated against the schema. This must be set to either
JSON
orBINARY
. - FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both firstRevisionId
and lastRevisionId
are optional.
Response:
{ "name" : "projects/ PROJECT_ID /topics/ TOPIC_ID " , "schemaSettings" : { "schema" : " SCHEMA_NAME " , "encoding" : " ENCODING_TYPE " "firstRevisionId" : " FIRST_REVISION_ID " "lastRevisionId" : " LAST_REVISION_ID " } }
Both firstRevisionId
and lastRevisionId
are omitted if not provided
in the request.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub C++ API reference documentation .
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub C# API reference documentation .
Go
The following sample uses the major version of the Go Pub/Sub client library (v2). If you are still using the v1 library, see the migration guide to v2 . To see a list of v1 code samples, see the deprecated code samples .
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Go API reference documentation .
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Java API reference documentation .
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Node.js API reference documentation .
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Node.js API reference documentation .
PHP
Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub PHP API reference documentation .
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Python API reference documentation .
Ruby
The following sample uses Ruby Pub/Sub client library v3. If you are still using the v2 library, see the migration guide to v3 . To see a list of Ruby v2 code samples, see the deprecated code samples .
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Ruby API reference documentation .
Edit a schema associated with a topic
You can edit a topic to attach a schema, remove a schema, or update the revision range used for validating messages. In general, if you have changes planned for the schema in use, you can commit a new revision and update the range of revisions used for the topic.
You can edit a schema associated with a topic using the Google Cloud console, the gcloud CLI, the Pub/Sub API, or the Cloud Client Libraries.
Console
-
In the Google Cloud console, go to the Pub/Sub topicspage.
-
Click the Topic IDof a topic.
-
In the topic details page, click Edit.
-
You can make the following changes to the schema.
It might take a few minutes for the changes to take place.
-
If you want to remove the schema from the topic, in the Edit topicpage, clear the Use a schemacheckbox.
-
If you want to change the schema, in the Schemasection, select the name of a schema.
Update the other fields as required.
- If you want to update the revision range, for Revision Range, use the drop-down menus for First revision allowedand Last revision allowed.
You can specify both fields, specify only one, or retain the default settings based on your requirements.
-
-
Click Updateto save the changes.
gcloud
gcloud pubsub topics update TOPIC_ID \ --message-encoding = ENCODING_TYPE \ --schema = SCHEMA_NAME \ --first-revision-id = FIRST_REVISION_ID \ --last-revision-id = LAST_REVISION_ID \
Where:
- TOPIC_ID is the ID for the topic that you're creating.
- ENCODING_TYPE
is the encoding of messages validated against the
schema. This value must be set to either
JSON
orBINARY
. - SCHEMA_NAME is the name of an existing schema.
- FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both --first-revision-id
and --last-revision-id
are optional.
REST
To update a topic, use the projects.topics.patch
method:
Request:
The request must be authenticated with an access token in the Authorization
header. To obtain an access token for the current
Application Default Credentials: gcloud auth application-default print-access-token
.
PATCH https://pubsub.googleapis.com/v1/projects/ PROJECT_ID /topics/ TOPIC_ID Authorization: Bearer ACCESS_TOKEN
Request body:
{ "schemaSettings" : { "schema" : " SCHEMA_NAME " , "encoding" : " ENCODING_TYPE " "firstRevisionId" : " FIRST_REVISION_ID " "lastRevisionId" : " LAST_REVISION_ID " "update_mask" : } }
Where:
- PROJECT_ID is your project ID.
- TOPIC_ID is your topic ID.
- SCHEMA_NAME
is the name of the schema that messages published should be validated against. The format is:
projects/ PROJECT_ID /schemas/ SCHEMA_ID
. - ENCODING_TYPE
is the encoding of messages validated against the schema. This must be set to either
JSON
orBINARY
. - FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both firstRevisionId
and lastRevisionId
are optional.
Response:
{ "name" : "projects/ PROJECT_ID /topics/ TOPIC_ID " , "schemaSettings" : { "schema" : " SCHEMA_NAME " , "encoding" : " ENCODING_TYPE " "firstRevisionId" : " FIRST_REVISION_ID " "lastRevisionId" : " LAST_REVISION_ID " } }
Both firstRevisionId
and lastRevisionId
they are not set after the
update.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub C++ API reference documentation .
Go
The following sample uses the major version of the Go Pub/Sub client library (v2). If you are still using the v1 library, see the migration guide to v2 . To see a list of v1 code samples, see the deprecated code samples .
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Go API reference documentation .
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Java API reference documentation .
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries . For more information, see the Pub/Sub Python API reference documentation .