This document explains how to define the VM resources for a Batch job by specifying a Compute Engine VM instance template when you create the job.
The types of VM resources that a job runs on are automatically defined by Batch unless you define them by using one of the following methods:
- Define a job's VM resources directly by using the
instances[].policyfield .This method is demonstrated in most Batch documentation. -
Define a job's VM resources through a template by using the
instances[].instanceTemplatefield .This is the method explained in this document.Using a template is required to specify VM options that Batch doesn't provide job fields for. Using a template can also be convenient when you want to specify the same VM resources for multiple jobs.
Before you begin
- If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users .
- Create an instance template or identify an existing instance template .
-
To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:
- Batch Job Editor
(
roles/batch.jobsEditor) on the project - Service Account User
(
roles/iam.serviceAccountUser) on the job's service account , which by default is the default Compute Engine service account - Create a job from a Compute Engine VM instance template: Compute Viewer
(
roles/compute.viewer) on the VM instance template
For more information about granting roles, see Manage access to projects, folders, and organizations .
You might also be able to get the required permissions through custom roles or other predefined roles .
- Batch Job Editor
(
Create a job using a Compute Engine VM instance template
This section provides examples for how to create a basic script job from an existing VM instance template. You can create a job from a VM instance template using the gcloud CLI, Batch API, Go, Java, Node.js, Python, or C++.
gcloud
To create a job from a VM instance template using the
gcloud CLI, use the gcloud batch jobs submit
command
and specify the VM instance template in the job's JSON configuration file.
For example, to create a basic script job from a VM instance template:
-
Create a JSON file in the current directory named
hello-world-instance-template.jsonwith the following contents:{ "taskGroups" : [ { "taskSpec" : { "runnables" : [ { "script" : { "text" : "echo Hello world! This is task ${BATCH_TASK_INDEX}. This job has a total of ${BATCH_TASK_COUNT} tasks." } } ], "computeResource" : { "cpuMilli" : 2000 , "memoryMib" : 16 }, "maxRetryCount" : 2 , "maxRunDuration" : "3600s" }, "taskCount" : 4 , "parallelism" : 2 } ], "allocationPolicy" : { "instances" : [ { "installGpuDrivers" : INSTALL_GPU_DRIVERS , "instanceTemplate" : " INSTANCE_TEMPLATE_NAME " } ] }, "labels" : { "department" : "finance" , "env" : "testing" }, "logsPolicy" : { "destination" : "CLOUD_LOGGING" } }Replace the following:
-
INSTALL_GPU_DRIVERS: Optional. When set totrue, Batch fetches the drivers required for the GPU type that you specify in your Compute Engine VM instance template, and Batch installs them on your behalf. For more information, see how to create a job that uses a GPU . -
INSTANCE_TEMPLATE_NAME: the name of an existing Compute Engine VM instance template. Learn how to create and list instance templates .
-
-
Run the following command:
gcloud batch jobs submit example-template-job \ --location us-central1 \ --config hello-world-instance-template.json
API
To create a basic job using the Batch API, use the jobs.create
method
and specify a VM instance template in the allocationPolicy
field.
For example, to create a basic script jobs from a VM instance template, use the following request:
POST
h
tt
ps
:
//batch.googleapis.com/v1/projects/ PROJECT_ID
/locations/us-central1/jobs?job_id=example-script-job
{
"taskGroups"
:
[
{
"taskSpec"
:
{
"runnables"
:
[
{
"script"
:
{
"text"
:
"echo Hello world! This is task ${BATCH_TASK_INDEX}. This job has a total of ${BATCH_TASK_COUNT} tasks."
}
}
],
"computeResource"
:
{
"cpuMilli"
:
2000
,
"memoryMib"
:
16
},
"maxRetryCount"
:
2
,
"maxRunDuration"
:
"3600s"
},
"taskCount"
:
4
,
"parallelism"
:
2
}
],
"allocationPolicy"
:
{
"instances"
:
[
{
"installGpuDrivers"
:
INSTALL_GPU_DRIVERS
,
"instanceTemplate"
:
" INSTANCE_TEMPLATE_NAME
"
}
]
},
"labels"
:
{
"department"
:
"finance"
,
"env"
:
"testing"
},
"logsPolicy"
:
{
"destination"
:
"CLOUD_LOGGING"
}
}
Replace the following:
-
PROJECT_ID: the project ID of your project. -
INSTALL_GPU_DRIVERS: Optional. When set totrue, Batch fetches the drivers required for the GPU type that you specify in your Compute Engine VM instance template, and Batch installs them on your behalf. For more information, see how to create a job that uses a GPU . -
INSTANCE_TEMPLATE_NAME: the name of an existing Compute Engine VM instance template. Learn how to create and list instance templates .
Go
Go
For more information, see the Batch Go API reference documentation .
To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Java
Java
For more information, see the Batch Java API reference documentation .
To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Node.js
Node.js
For more information, see the Batch Node.js API reference documentation .
To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Python
Python
For more information, see the Batch Python API reference documentation .
To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
C++
C++
For more information, see the Batch C++ API reference documentation .
To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
What's next
- If you have issues creating or running a job, see Troubleshooting .
- View jobs and tasks .
- Learn about more job creation options .
