Create and manage batch operation jobs

Command line

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. (Optional) Run a dry run job. Before executing any job, we recommend that you run the job in dry run mode to verify the object selection criteria and check for any errors. The dry run does not modify any objects.

   In your development environment, run the gcloud storage batch-operations jobs create command with the --dry-run flag:

   gcloud storage batch-operations jobs create DRY_RUN_JOB_NAME 
   \
   --bucket= BUCKET_NAME 
    OBJECT_SELECTION_FLAG 
    JOB_TYPE_FLAG 
   \
   --dry-run

   The dry run uses the same parameters as the actual job. For details, see parameter descriptions .

   To view the results of the dry run, see Get storage batch operations job details .

3. After a successful dry run, run the gcloud storage batch-operations jobs create command.

   gcloud storage batch-operations jobs create JOB_NAME 
   \
   --bucket= BUCKET_NAME 
    OBJECT_SELECTION_FLAG 
    JOB_TYPE_FLAG 
   

   Where the parameters are as follows :

   • DRY_RUN_JOB_NAME is the name of the storage batch operations dry run job.

   • JOB_NAME is the name of the storage batch operations job.

   • BUCKET_NAME is the name of the bucket that contains one or more objects you want to process.

   • OBJECT_SELECTION_FLAG is one of the following flags that you need to specify:

     • --included-object-prefixes : Specify one or more object prefixes . For example:

       • To match a single prefix, use: --included-object-prefixes='prefix1' .
       • To match multiple prefixes, use a comma-separated prefix list: --included-object-prefixes='prefix1,prefix2' .
       • To include all objects, use an empty prefix: --included-object-prefixes='' .

     • --manifest-location : Specify the manifest location. For example, gs://bucket_name/path/object_name.csv .

   • JOB_TYPE_FLAG is one of the following flags that you need to specify, depending on the job type .

     • --delete-object : Delete one or more objects.

       • If Object Versioning is enabled for the bucket, current objects transition to a noncurrent state, and noncurrent objects are skipped.

       • If Object Versioning is disabled for the bucket, the delete operation permanently deletes objects and skips noncurrent objects.

     • --enable-permanent-object-deletion : Permanently delete objects. Use this flag along with the --delete-object flag to permanently delete both live and noncurrent objects in a bucket, regardless of the bucket's object versioning configuration.

     • --rewrite-object : Update the customer-managed encryption keys for one or more objects.

     • --put-object-event-based-hold : Enable event-based object holds .

     • --no-put-object-event-based-hold : Disable event-based object holds .

     • --put-object-temporary-hold : Enable temporary object holds .

     • --no-put-object-temporary-hold : Disable temporary object holds .

       The following example shows how to create a job to update the Content-Language metadata to en for all objects listed in manifest.csv .

       gcloud storage batch-operations jobs create my-job \
       --bucket=my-bucket \
       --manifest-location=gs://my-bucket/manifest.csv \
       --put-metadata=Content-Language=en

     • --put-metadata : Update object metadata . Specify the key-value pair for the object metadata you want to modify. You can specify one or more key-value pairs as a list. You can also set object retention configurations using the --put-metadata flag. To do so, specify the retention parameters using the Retain-Until and Retention-Mode fields. For example,

       gcloud storage batch-operations jobs create my-job \
       --bucket=my-bucket \
       --manifest-location=gs://my-bucket/manifest.csv \
       --put-metadata=Retain-Until= RETAIN_UNTIL_TIME 
       , Retention-Mode= RETENTION_MODE 
       

       Where:

       • RETAIN_UNTIL_TIME is the date and time, in RFC 3339 format, until which the object is retained. For example, 2025-10-09T10:30:00Z . To set the retention configuration on an object, you'll need to enable retention on the bucket which contains the object.

       • RETENTION_MODE is the retention mode, either Unlocked or Locked .

         When you send a request to update the RETENTION_MODE and RETAIN_UNTIL_TIME fields, consider the following:

         • To update the object retention configuration, you must provide non-empty values for both RETENTION_MODE and RETAIN_UNTIL_TIME fields; setting only one results in an INVALID_ARGUMENT error.
         • You can extend the RETAIN_UNTIL_TIME value for objects in both Unlocked or Locked modes.
         • The object retention must be in Unlocked mode if you want to do the following:
           • Reduce the RETAIN_UNTIL_TIME value.
           • Remove the retention configuration. To remove the configuration, you'll need to provide empty values for both RETENTION_MODE and RETAIN_UNTIL_TIME fields.
         • If you omit both RETENTION_MODE and RETAIN_UNTIL_TIME fields, the retention configuration remains unchanged.

     • --clear-all-object-custom-contexts : Delete all existing object contexts .

       The following example shows how to create a job to clear all object contexts for objects listed in manifest.csv :

       gcloud storage batch-operations jobs create my-job \
       --bucket=my-bucket \
       --manifest-location=gs://my-bucket/manifest.csv \
       --clear-all-object-custom-contexts

     • --clear-object-custom-contexts : Remove contexts with specific keys. You can also update specific contexts along with removing keys by using both the --clear-object-custom-contexts flag and one of the following flags:

       • --update-object-custom-contexts : Provide a map of key-value pairs.

         The following example shows how to create a job to remove the context with key temp-id and update or insert context with key project-id and cost-center for all objects listed in manifest.csv :

         gcloud storage batch-operations jobs create my-job \
         --bucket=my-bucket \
         --manifest-location=gs://my-bucket/manifest.csv \
         --clear-object-custom-contexts=temp-id \
         --update-object-custom-contexts=project-id=project-A,cost-center=engineering

       • --update-object-custom-contexts-file : Provide the path to a JSON or YAML file with key-value pairs.

         The following example shows how to create a job to process objects defined in manifest.csv . The job does the following:

         • Removes all contexts with the temp-id key.

         • Updates existing contexts with the project-id and cost-center keys defined in the /tmp/context_updates.json file.

         gcloud storage batch-operations jobs create my-job \
         --bucket=my-bucket \
         --manifest-location=gs://my-bucket/manifest.csv \
         --clear-object-custom-contexts=temp-id \
         --update-object-custom-contexts-file=/tmp/context_updates.json

         Where /tmp/context_updates.json contains the following object contexts:

          { 
          "project-id" 
          : 
           
          { 
          "value" 
          : 
           
          "project-A" 
          }, 
          "cost-center" 
          : 
           
          { 
          "value" 
          : 
           
          "engineering" 
          } 
          } 
         

Client libraries

  []( 
 google 
 :: 
 cloud 
 :: 
 storagebatchoperations_v1 
 :: 
 StorageBatchOperationsClient 
  
 client 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 project_id 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 job_id 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 target_bucket_name 
 , 
  
 std 
 :: 
 string 
  
 const 
&  
 object_prefix 
 ) 
  
 { 
  
 auto 
  
 const 
  
 parent 
  
 = 
  
 std 
 :: 
 string 
 { 
 "projects/" 
 } 
  
 + 
  
 project_id 
  
 + 
  
 "/locations/global" 
 ; 
  
 namespace 
  
 sbo 
  
 = 
  
 google 
 :: 
 cloud 
 :: 
 storagebatchoperations 
 :: 
 v1 
 ; 
  
 sbo 
 :: 
 Job 
  
 job 
 ; 
  
 sbo 
 :: 
 BucketList 
 * 
  
 bucket_list 
  
 = 
  
 job 
 . 
 mutable_bucket_list 
 (); 
  
 sbo 
 :: 
 BucketList 
 :: 
 Bucket 
 * 
  
 bucket_config 
  
 = 
  
 bucket_list 
 - 
> add_buckets 
 (); 
  
 bucket_config 
 - 
> set_bucket 
 ( 
 target_bucket_name 
 ); 
  
 sbo 
 :: 
 PrefixList 
 * 
  
 prefix_list_config 
  
 = 
  
 bucket_config 
 - 
> mutable_prefix_list 
 (); 
  
 prefix_list_config 
 - 
> add_included_object_prefixes 
 ( 
 object_prefix 
 ); 
  
 sbo 
 :: 
 DeleteObject 
 * 
  
 delete_object_config 
  
 = 
  
 job 
 . 
 mutable_delete_object 
 (); 
  
 delete_object_config 
 - 
> set_permanent_object_deletion_enabled 
 ( 
 false 
 ); 
  
 auto 
  
 result 
  
 = 
  
 client 
 . 
 CreateJob 
 ( 
 parent 
 , 
  
 job 
 , 
  
 job_id 
 ). 
 get 
 (); 
  
 if 
  
 ( 
 ! 
 result 
 ) 
  
 throw 
  
 result 
 . 
 status 
 (); 
  
 std 
 :: 
 cout 
 << 
 "Created job: " 
 << 
 result 
 - 
> name 
 () 
 << 
 " 
 \n 
 " 
 ; 
 } 
 

  use Google\Cloud\StorageBatchOperations\V1\Client\StorageBatchOperationsClient; 
 use Google\Cloud\StorageBatchOperations\V1\CreateJobRequest; 
 use Google\Cloud\StorageBatchOperations\V1\Job; 
 use Google\Cloud\StorageBatchOperations\V1\BucketList; 
 use Google\Cloud\StorageBatchOperations\V1\BucketList\Bucket; 
 use Google\Cloud\StorageBatchOperations\V1\PrefixList; 
 use Google\Cloud\StorageBatchOperations\V1\DeleteObject; 
 /** 
 * Create a new batch job. 
 * 
 * @param string $projectId Your Google Cloud project ID. 
 *        (e.g. 'my-project-id') 
 * @param string $jobId A unique identifier for this job. 
 *        (e.g. '94d60cc1-2d95-41c5-b6e3-ff66cd3532d5') 
 * @param string $bucketName The name of your Cloud Storage bucket to operate on. 
 *        (e.g. 'my-bucket') 
 * @param string $objectPrefix The prefix of objects to include in the operation. 
 *        (e.g. 'prefix1') 
 */ 
 function create_job(string $projectId, string $jobId, string $bucketName, string $objectPrefix): void 
 { 
 // Create a client. 
 $storageBatchOperationsClient = new StorageBatchOperationsClient(); 
 $parent = $storageBatchOperationsClient->locationName($projectId, 'global'); 
 $prefixListConfig = new PrefixList(['included_object_prefixes' => [$objectPrefix]]); 
 $bucket = new Bucket(['bucket' => $bucketName, 'prefix_list' => $prefixListConfig]); 
 $bucketList = new BucketList(['buckets' => [$bucket]]); 
 $deleteObject = new DeleteObject(['permanent_object_deletion_enabled' => false]); 
 $job = new Job(['bucket_list' => $bucketList, 'delete_object' => $deleteObject]); 
 $request = new CreateJobRequest([ 
 'parent' => $parent, 
 'job_id' => $jobId, 
 'job' => $job, 
 ]); 
 $response = $storageBatchOperationsClient->createJob($request); 
 printf('Created job: %s', $response->getName()); 
 }