This document explains how to create a Batch job that sends Pub/Sub notifications. You can use Pub/Sub to get notifications when a job or task state changes, or when a job or task enters a specific state. For more information, see Monitor jobs using notifications .
Before you begin
- If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users .
- Create or identify a Pub/Sub topic for Batch notifications .
- Configure a subscription to receive and use the notifications .
Required roles
-
To get the permissions that you need to create and run a job that sends notifications, ask your administrator to grant you the following IAM roles:
- Batch Job Editor
(
roles/batch.jobsEditor) on the project. - Service Account User
(
roles/iam.serviceAccountUser) on the job's service account , which by default is the default Compute Engine service account . - Pub/Sub Editor
(
roles/pubsub.editor) on the Pub/Sub topic or project.
For more information about granting roles, see Manage access to projects, folders, and organizations .
You might also be able to get the required permissions through custom roles or other predefined roles .
- Batch Job Editor
(
-
Unless you are using the default configuration for the job's service account , ensure that it has the necessary permissions.
To ensure that the job's service account has the necessary permissions to publish Pub/Sub notifications, ask your administrator to grant the job's service account the Pub/Sub Publisher (
roles/pubsub.publisher) IAM role on your Pub/Sub topic. - If you want a job to publish notifications to a Pub/Sub topic that
is in a different project than the job, then the Batch service agent
for the job's project must be granted permission to publish to that topic.
To ensure that the Batch service agent for the job's project has the necessary permissions to publish Pub/Sub notifications to a Pub/Sub topic in another project, ask your administrator to grant the Batch service agent for the job's project the Pub/Sub Publisher (
roles/pubsub.publisher) IAM role on the Pub/Sub topic.
Create and run a job that sends notifications
You can create a Batch job that sends Pub/Sub notifications by doing the following:
gcloud
Use the Google Cloud CLI to create a job
that
includes the notifications
field and one or more jobNotification
objects
in the main body of the JSON file:
{
...
"notifications"
:
[
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
ATTRIBUTES
}
}
]
...
}
Replace the following:
-
PROJECT_ID: the project ID of the project that contains the Pub/Sub topic. -
TOPIC_ID: the Pub/Sub topic ID of the topic you created when you enabled Pub/Sub notifications . -
ATTRIBUTES: specify one or more of the following attributes, which each let you receive notifications about the state of the job or all of its tasks.-
For notifications about all job state changes, specify the following:
"type" : "JOB_STATE_CHANGED" -
For notifications about a specific job state change, specify the following:
"type" : "JOB_STATE_CHANGED" , "newJobState" : " JOB_STATE "Replace
JOB_STATEwith one of the following job states:-
QUEUED -
SCHEDULED -
RUNNING -
SUCCEEDED -
FAILED
For more information about job states, see Job lifecycle .
-
-
For notifications about all task state changes, specify the following:
"type" : "TASK_STATE_CHANGED" -
For notifications about specific task state changes, specify the following:
"type" : "TASK_STATE_CHANGED" , "newTaskState" : " TASK_STATE "Replace
TASK_STATEwith one of the following task states:-
PENDING -
ASSIGNED -
RUNNING -
SUCCEEDED -
FAILED
For more information about task states, see Job lifecycle .
-
-
For example, suppose you want to receive notifications about all job state changes and any time a task fails. To do so, you can have a JSON configuration file that is similar to the following:
{
"taskGroups"
:
[
{
"taskSpec"
:
{
"runnables"
:
[
{
"script"
:
{
"text"
:
"echo Hello World! This is task $BATCH_TASK_INDEX."
}
}
]
},
"taskCount"
:
3
,
}
],
"logsPolicy"
:
{
"destination"
:
"CLOUD_LOGGING"
},
"notifications"
:
[
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
"type"
:
"JOB_STATE_CHANGED"
}
},
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
"type"
:
"TASK_STATE_CHANGED"
,
"newTaskState"
:
"FAILED"
}
}
]
}
API
Use the REST API to create a job
that
includes the notifications
field and one or more jobNotification
objects
in the main body of the JSON file:
{
...
"notifications"
:
[
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
ATTRIBUTES
}
}
]
...
}
Replace the following:
-
PROJECT_ID: the project ID of the project that contains the Pub/Sub topic. -
TOPIC_ID: the Pub/Sub topic ID of the topic you created when you enabled Pub/Sub notifications . -
ATTRIBUTES: specify one or more of the following attributes, which each let you receive notifications about the state of the job or all of its tasks.-
For notifications about all job state changes, specify the following:
"type" : "JOB_STATE_CHANGED" -
For notifications about a specific job state change, specify the following:
"type" : "JOB_STATE_CHANGED" , "newJobState" : " JOB_STATE "Replace
JOB_STATEwith one of the following job states:-
QUEUED -
SCHEDULED -
RUNNING -
SUCCEEDED -
FAILED
For more information about job states, see Job lifecycle .
-
-
For notifications about all task state changes, specify the following:
"type" : "TASK_STATE_CHANGED" -
For notifications about specific task state changes, specify the following:
"type" : "TASK_STATE_CHANGED" , "newTaskState" : " TASK_STATE "Replace
TASK_STATEwith one of the following task states:-
PENDING -
ASSIGNED -
RUNNING -
SUCCEEDED -
FAILED
For more information about task states, see Job lifecycle .
-
-
For example, suppose you want to receive notifications about all job state changes and any time a task fails. To do so, you can have a JSON configuration file that is similar to the following:
{
"taskGroups"
:
[
{
"taskSpec"
:
{
"runnables"
:
[
{
"script"
:
{
"text"
:
"echo Hello World! This is task $BATCH_TASK_INDEX."
}
}
]
},
"taskCount"
:
3
,
}
],
"logsPolicy"
:
{
"destination"
:
"CLOUD_LOGGING"
},
"notifications"
:
[
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
"type"
:
"JOB_STATE_CHANGED"
}
},
{
"pubsubTopic"
:
"projects/ PROJECT_ID
/topics/ TOPIC_ID
"
,
"message"
:
{
"type"
:
"TASK_STATE_CHANGED"
,
"newTaskState"
:
"FAILED"
}
}
]
}
Go
Java
Node.js
Python
After the job starts running, you can use its notifications. For example, if the Pub/Sub topic for your job has a subscription that streams notifications to BigQuery, you can analyze the Pub/Sub notifications in BigQuery .
What's next
- Learn more about how to monitor job status using Pub/Sub notifications and BigQuery .
- If you have issues creating or running a job, see Troubleshooting .
- View jobs and tasks .
- Learn about more job creation options .
