Build and create a Python job in Cloud Run

Learn how to create a simple Cloud Run job, then deploy from source, which automatically packages your code into a container image, uploads the container image to Artifact Registry, and then deploys to Cloud Run. You can use other languages in addition to the ones shown.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Go to project selector

Verify that billing is enabled for your Google Cloud project .

Make sure that you have the following role or roles on the project: Cloud Run Admin, Cloud Run Source Developer, Logs Viewer

Check for the roles

In the Google Cloud console, go to the IAM page.
Go to IAM
Select the project.
In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.
For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.

Grant the roles

In the Google Cloud console, go to the IAM page.
Go to IAM
Select the project.
Click Grant access .
In the New principals field, enter your user identifier. This is typically the email address for a Google Account.
In the Select a role list, select a role.
To grant additional roles, click Add another role and add each additional role.
Click Save .

Install the Google Cloud CLI.

Note:If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update .

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

To initialize the gcloud CLI, run the following command:

gcloud  
init

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Go to project selector

Verify that billing is enabled for your Google Cloud project .

Make sure that you have the following role or roles on the project: Cloud Run Admin, Cloud Run Source Developer, Logs Viewer

Check for the roles

In the Google Cloud console, go to the IAM page.
Go to IAM
Select the project.
In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.
For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.

Grant the roles

In the Google Cloud console, go to the IAM page.
Go to IAM
Select the project.
Click Grant access .
In the New principals field, enter your user identifier. This is typically the email address for a Google Account.
In the Select a role list, select a role.
To grant additional roles, click Add another role and add each additional role.
Click Save .

Install the Google Cloud CLI.

Note:If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update .

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

To initialize the gcloud CLI, run the following command:

gcloud  
init

Enable the Cloud Run Admin API and the Cloud Build API:
```
gcloud  
services  
 enable 
  
run.googleapis.com  
 \ 
  
cloudbuild.googleapis.com
```
After the Cloud Run Admin API is enabled, the Compute Engine default service account is automatically created.
Make sure that you have the Service Account User role granted on the service identity. By default, the service identity is the Compute Engine default service account.
Grant the roles

To grant access on the service identity resource, use the gcloud iam service-accounts add-iam-policy-binding command, replacing the highlighted variables with the appropriate values:
```
  
gcloud  
iam  
service-accounts  
add-iam-policy-binding  
 SERVICE_ACCOUNT_EMAIL 
  
 \ 
  
--member = 
 " PRINCIPAL 
" 
  
 \ 
  
--role = 
 "roles/iam.serviceAccountUser" 
  
```
Replace the following:
- SERVICE_ACCOUNT_EMAIL : the service account email address you are using as the service identity, such as:
  - The Compute Engine default service account: PROJECT_NUMBER -compute@developer.gserviceaccount.com
  - A service account that you created: SERVICE_ACCOUNT_NAME @ PROJECT_ID .iam.gserviceaccount.com
- PRINCIPAL : the user identifier. This is typically the email address for a Google Account.
Grant the Cloud Build service account the following IAM role.
Click to view required roles for the Cloud Build service account

Cloud Build automatically uses the Compute Engine default service account as the default Cloud Build service account to build your source code and Cloud Run resource, unless you override this behavior. For Cloud Build to build your sources, ask your administrator to grant Cloud Run Builder ( roles/run.builder ) to the Compute Engine default service account on your project:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member = serviceAccount: PROJECT_NUMBER -compute@developer.gserviceaccount.com \ --role = roles/run.builder
Replace PROJECT_NUMBER with your Google Cloud project number, and PROJECT_ID with your Google Cloud project ID. For detailed instructions on how to find your project ID, and project number, see Creating and managing projects .

Granting the Cloud Run builder role to the Compute Engine default service account takes a couple of minutes to propagate .

Note:
The iam.automaticIamGrantsForDefaultServiceAccounts organization policy constraint prevents the Editor role from being automatically granted to default service accounts. If you created your organization after May 3, 2024, this constraint is enforced by default.

We strongly recommend that you enforce this constraint to disable the automatic role grant. If you disable the automatic role grant, you must decide which roles to grant to the default service accounts, and then grant these roles yourself.

If the default service account already has the Editor role, we recommend that you replace the Editor role with less permissive roles.To safely modify the service account's roles, use Policy Simulator to see the impact of the change, and then grant and revoke the appropriate roles .
Review Cloud Run pricing or estimate costs with the pricing calculator .

Writing the sample job

To write a job in Python:

Create a new directory named jobs and change directory into it:
```
  mkdir 
 jobs 
 cd 
 jobs 
 
```

Create a main.py file for the actual job code. Copy the following sample lines into it:

  import 
  
 json 
 import 
  
 os 
 import 
  
 random 
 import 
  
 sys 
 import 
  
 time 
 # Retrieve Job-defined env vars 
 TASK_INDEX 
 = 
 os 
 . 
 getenv 
 ( 
 "CLOUD_RUN_TASK_INDEX" 
 , 
 0 
 ) 
 TASK_ATTEMPT 
 = 
 os 
 . 
 getenv 
 ( 
 "CLOUD_RUN_TASK_ATTEMPT" 
 , 
 0 
 ) 
 # Retrieve User-defined env vars 
 SLEEP_MS 
 = 
 os 
 . 
 getenv 
 ( 
 "SLEEP_MS" 
 , 
 0 
 ) 
 FAIL_RATE 
 = 
 os 
 . 
 getenv 
 ( 
 "FAIL_RATE" 
 , 
 0 
 ) 
 # Define main script 
 def 
  
 main 
 ( 
 sleep_ms 
 = 
 0 
 , 
 fail_rate 
 = 
 0 
 ): 
  
 """Program that simulates work using the sleep method and random failures. 
 Args: 
 sleep_ms: number of milliseconds to sleep 
 fail_rate: rate of simulated errors 
 """ 
 print 
 ( 
 f 
 "Starting Task # 
 { 
 TASK_INDEX 
 } 
 , Attempt # 
 { 
 TASK_ATTEMPT 
 } 
 ..." 
 ) 
 # Simulate work by waiting for a specific amount of time 
 time 
 . 
 sleep 
 ( 
 float 
 ( 
 sleep_ms 
 ) 
 / 
 1000 
 ) 
 # Convert to seconds 
 # Simulate errors 
 random_failure 
 ( 
 float 
 ( 
 fail_rate 
 )) 
 print 
 ( 
 f 
 "Completed Task # 
 { 
 TASK_INDEX 
 } 
 ." 
 ) 
 def 
  
 random_failure 
 ( 
 rate 
 ): 
  
 """Throws an error based on fail rate 
 Args: 
 rate: a float between 0 and 1 
 """ 
 if 
 rate 
< 0 
 or 
 rate 
> 1 
 : 
 # Return without retrying the Job Task 
 print 
 ( 
 f 
 "Invalid FAIL_RATE env var value: 
 { 
 rate 
 } 
 . " 
 + 
 "Must be a float between 0 and 1 inclusive." 
 ) 
 return 
 random_failure 
 = 
 random 
 . 
 random 
 () 
 if 
 random_failure 
< rate 
 : 
 raise 
 Exception 
 ( 
 "Task failed." 
 ) 
 # Start script 
 if 
 __name__ 
 == 
 "__main__" 
 : 
 try 
 : 
 main 
 ( 
 SLEEP_MS 
 , 
 FAIL_RATE 
 ) 
 except 
 Exception 
 as 
 err 
 : 
 message 
 = 
 ( 
 f 
 "Task # 
 { 
 TASK_INDEX 
 } 
 , " 
 + 
 f 
 "Attempt # 
 { 
 TASK_ATTEMPT 
 } 
 failed: 
 { 
 str 
 ( 
 err 
 ) 
 } 
 " 
 ) 
 print 
 ( 
 json 
 . 
 dumps 
 ({ 
 "message" 
 : 
 message 
 , 
 "severity" 
 : 
 "ERROR" 
 })) 
 sys 
 . 
 exit 
 ( 
 1 
 ) 
 # Retry Job Task by exiting the process

Cloud Run jobs allows users to specify the number of tasks the job is to execute. This sample code shows how to use the built-in CLOUD_RUN_TASK_INDEX environment variable. Each task represents one running copy of the container. Note that tasks are usually executed in parallel. Using multiple tasks is useful if each task can independently process a subset of your data.

Each task is aware of its index, stored in the CLOUD_RUN_TASK_INDEX environment variable. The built-in CLOUD_RUN_TASK_COUNT environment variable contains the number of tasks supplied at job execution time via the --tasks parameter.

The code shown also shows how to retry tasks, using the built-in CLOUD_RUN_TASK_ATTEMPT environment variable, which contains the number of times this task has been retried, starting at 0 for the first attempt and incrementing by 1 for every successive retry, up to --max-retries .

The code also lets you generate failures as a way to test retries and to generate error logs so you can see what those look like.

Create a text file named Procfile with no file extension, containing the following:
```
  web 
 : 
 python3 
 main 
 . 
 py 
 
```

Your code is complete and ready to be packaged in a container.

Build jobs container, send it to Artifact Registry and deploy to Cloud Run

Important: This quickstart assumes that you have owner or editor roles in the project you are using for the quickstart. Otherwise, refer to the Cloud Run Source Developer role for the required permissions for deploying a Cloud Run resource from source.

This quickstart uses deploy from source, which builds the container, uploads it to Artifact Registry, and deploys the job to Cloud Run:

gcloud  
run  
 jobs 
  
deploy  
job-quickstart  
 \ 
  
--source  
.  
 \ 
  
--tasks  
 50 
  
 \ 
  
--set-env-vars  
 SLEEP_MS 
 = 
 10000 
  
 \ 
  
--set-env-vars  
 FAIL_RATE 
 = 
 0 
.1  
 \ 
  
--max-retries  
 5 
  
 \ 
  
--region  
 REGION 
  
 \ 
  
--project = 
 PROJECT_ID

where PROJECT_ID is your project ID and REGION is your region, for example, europe-west1 . Note that you can change the various parameters to whatever values you want to use for your testing purposes. SLEEP_MS simulates work and FAIL_RATE causes X % of tasks to fail so you can experiment with parallelism and retrying failing tasks.

Execute a job in Cloud Run

To execute the job you just created:

gcloud  
run  
 jobs 
  
execute  
job-quickstart  
--region  
 REGION

Replace REGION with the region you used when you created and deployed the job, for example europe-west1 .

Clean up

To avoid additional charges to your Google Cloud account, delete all the resources you deployed with this quickstart.

Delete your repository

Cloud Run only charges for the time your job executes. However, you might still be charged for storing the container image in Artifact Registry . To delete Artifact Registry repositories, follow the steps in Delete repositories in the Artifact Registry documentation.

Delete your job

Cloud Run jobs only incur cost when a job task is executing. To delete your Cloud Run job, follow one of these steps:

Console

To delete a job:

In the Google Cloud console, go to Cloud Run:

Go to Cloud Run
Locate the job you want to delete in the jobs list, and click its checkbox to select it.
Click Delete. This terminates all the job executions in progress and all running container instances.

gcloud

To delete a job, run the following command:

gcloud  
run  
 jobs 
  
delete  
 JOB_NAME

Replace JOB_NAME with the name of the job.

Delete your test project

Deleting your Google Cloud project stops billing for all resources in that project. To release all Google Cloud resources in your project, follow these steps:

In the Google Cloud console, go to the Manage resources page.
Go to Manage resources
In the project list, select the project that you want to delete, and then click Delete .
In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

For more information on building a container from code source and pushing to a repository, see:

Build and create a Python job in Cloud Run

Before you begin

Check for the roles

Grant the roles

Check for the roles

Grant the roles

Grant the roles

Click to view required roles for the Cloud Build service account

Writing the sample job

Build jobs container, send it to Artifact Registry and deploy to Cloud Run

Execute a job in Cloud Run

Clean up

Delete your repository

Delete your job

Console

gcloud

Delete your test project

What's next