Console
    Contact Us Start free

    Relocate buckets

    This page describes the process for relocating buckets from one location to another. For information about bucket relocation, see Bucket relocation .

    Before you begin

    Before you can relocate buckets, complete the following steps:

    1. Configure Storage Intelligence .

    2. Enable soft delete .

    3. Check the quotas and limits to ensure that the new location has sufficient quotas to accommodate the bucket's data.

    4. Determine the bucket relocation type to understand whether write downtime is required.

    5. Remove any existing bucket tags .

    6. If you use inventory reports, save your configurations .

    7. Get required roles, which are described in the following section.

    Get required roles

    To get the permissions that you need to relocate buckets, ask your administrator to grant you the Storage Admin ( roles/storage.admin ) IAM role on the project. For more information about granting roles, see Manage access to projects, folders, and organizations .

    This predefined role contains the permissions required to relocate buckets. To see the exact permissions that are required, expand the Required permissionssection:

    Required permissions

    The following permissions are required to relocate buckets:

    • To relocate a bucket: storage.buckets.relocate
    • To view the status of a bucket relocation operation: storage.bucketOperations.get
    • To view the list of bucket relocation operations for a project: storage.bucketOperations.list
    • To cancel a bucket relocation operation: storage.bucketOperations.cancel
    • To view the metadata of a bucket during the bucket relocation's dry run and incremental data copy phases: storage.buckets.get
    • To get an object in a bucket you want to relocate: storage.objects.get
    • To list the objects in a bucket you want to relocate: storage.objects.list

    You might also be able to get these permissions with custom roles or other predefined roles .

    Relocate buckets

    This section describes the process of relocating Cloud Storage buckets from one location to another. When you relocate a bucket, you initiate the incremental data copy process, monitor the process, and then initiate the final synchronization step. For more information about these steps, see Understand the bucket relocation process .

    Perform a dry run

    To minimize potential issues during the bucket relocation process, we recommend you perform a dry run. A dry run simulates the bucket relocation process without moving data, helping you to catch and resolve issues early on. The dry run checks for the following incompatibilities:

    While a dry run can't identify every possible issue as some issues might only surface during the live migration due to factors such as real-time resource availability, it reduces the risk of facing time-consuming issues during the actual relocation.

    Command line

    Simulate the dry run of bucket relocation:

    gcloud storage buckets relocate gs:// BUCKET_NAME 
--location= LOCATION 
--dry-run

    Where:

    • BUCKET_NAME is the name of the bucket that you want to relocate.

    • LOCATION is the destination location of the bucket.

    After you initiate a dry run, a long-running operation starts. You'll receive an operation ID and a description of the operation. Track the progress and completion of the dry run by getting the details of the long-running operation .

    If the dry run reveals any issues, address them before proceeding with the Initiate the incremental data copy step .

    REST APIs

    JSON API

    1. Have gcloud CLI installed and initialized , which lets you generate an access token for the Authorization header.

    2. Create a JSON file that contains the settings for the bucket, which must include the destinationLocation and validateOnly parameters. See the Buckets: relocate documentation for a complete list of settings. The following are common settings to include:

       { 
  
 "destinationLocation" 
 : 
  
 " DESTINATION_LOCATION 
" 
 , 
  
 "destinationCustomPlacementConfig" 
 : 
  
 { 
  
 "dataLocations" 
 : 
  
 [ 
  
  LOCATIONS 
 
 , 
  
 ... 
  
 ] 
  
 }, 
  
 "validateOnly" 
 : 
  
 "true" 
 }

      Where:

      • DESTINATION_LOCATION is the destination location of the bucket.
      • LOCATIONS is a list of location codes to be used for the configurable dual-region .
      • validateOnly is set to true to perform a dry run.

    3. Use cURL to call the JSON API :

      curl -X POST --data-binary @ JSON_FILE_NAME 
\
 -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 -H "Content-Type: application/json" \
 "https://storage.googleapis.com/storage/v1/b/ BUCKET_NAME 
/relocate"

      Where:

      • JSON_FILE_NAME is the name of the JSON file you created.
      • BUCKET_NAME is the name of the bucket you want to relocate.

      After you initiate a dry run, a long-running operation starts. The dry run succeeds when the following conditions are met:

      • The dry run reports no errors.

      • The operations resource returns a done field value of true .

        {
"kind": "storage#operation",
"name": "projects/_/buckets/bucket/operations/operation_id",
"metadata": {
  "@type": OperationMetadataType*,
  metadata OperationMetadata*
},
"done": "true",
"response": {
  "@type": ResponseResourceType*,
  response ResponseResource*
}
}

      If the dry run reveals any issues, address them before proceeding with the Initiate the incremental data copy step .

    Initiate incremental data copy

    Command line

    Initiate the bucket relocation operation:

    gcloud storage buckets relocate gs:// BUCKET_NAME 
--location= LOCATION

    Where:

    • BUCKET_NAME is the name of the bucket that you want to relocate.

    • LOCATION is the destination location of the bucket.

    REST APIs

    JSON API

    1. Have gcloud CLI installed and initialized , which lets you generate an access token for the Authorization header.

    2. Create a JSON file that contains the settings for the bucket. See the Buckets: relocate documentation for a complete list of settings. The following are common settings to include:

       { 
  
 "destinationLocation" 
 : 
  
 " DESTINATION_LOCATION 
" 
 , 
  
 "destinationCustomPlacementConfig" 
 : 
  
 { 
  
 "dataLocations" 
 : 
  
 [ 
  
  LOCATIONS 
 
 , 
  
 ... 
  
 ] 
  
 }, 
  
 "validateOnly" 
 : 
  
 "false" 
 }

      Where:

      • DESTINATION_LOCATION is the destination location of the bucket.
      • LOCATIONS is a list of location codes to be used for the configurable dual-region .
      • validateOnly is set to false to initiate the incremental data copy step of bucket relocation.

    3. Use cURL to call the JSON API :

      curl -X POST --data-binary @ JSON_FILE_NAME 
\
 -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 -H "Content-Type: application/json" \
 "https://storage.googleapis.com/storage/v1/b/ BUCKET_NAME 
/relocate"

      Where:

      • JSON_FILE_NAME is the name of the JSON file you created.
      • BUCKET_NAME is the name of the bucket you want to relocate.

    Monitor incremental data copy

    The bucket relocation process is a long-running operation that must be monitored to see how it's progressing. You can regularly check the long-running operations list to see the status of the incremental data copy step. For information about how to get the details about a long-running operation, list, or cancel long-running operations, see Use long-running operations in Cloud Storage .

    The following example shows the output generated by an incremental data copy operation:

    done: false
  kind: storage#operation
  metadata:
  '@type': type.googleapis.com/google.storage.control.v2.RelocateBucketMetadata
  commonMetadata:
    createTime: '2024-10-21T04:26:59.666Z
    endTime: '2024-12-29T23:39:53.340Z'
    progressPercent: 99
    requestedCancellation: false
    type: relocate-bucket
    updateTime: '2024-10-21T04:27:03.2892'
  destinationLocation: US-CENTRAL1
  finalizationState: 'READY'
  progress:
    byteProgressPercent: 100
    discoveredBytes: 200
    remainingBytes: 0
    discoveredObjectCount: 10
    remainingObjectCount: 8
    objectProgressPercent: 100
    discoveredSyncCount: 8
    remainingSyncCount: 0
    syncProgressPercent: 100
  relocationState: SYNCING
  sourceLocation: US
  validateOnly: false
  estimatedWriteDowntimeDuration: '7200s'
  writeDowntimeExpireTime: '2024-12-30T10:34:01.786Z'
  name: projects//buckets/my-bucket1/operations/Bar7-1b0khdew@nhenUQRTF_R-Kk4dQ5V1f8fzezkFcPh3XMvlTqJ6xhnqJ1h_QXFIeAirrEqkjgu4zPKSRD6WSSG5UGXil6w
  response:
    '@type': type.googleapis.com/google.storage.control.v2.RelocateBucketResponse
      selfLink: https://storage.googleusercontent.com/storage/v1_ds/b/my-bucket1/operations/Bar7-1b0khdew@nhenUQRTF_R-Kk4dQ5V1f8fzezkFcPh3XMvlTqJ6xhnqJ1h_QXFIeAirrEqkjgu4zPKSRD6WSSG5UGXil6w

    The following table provides information about the key fields in the output generated by the incremental data copy operation:

    Field name
    Description
    Possible values
    done
    Indicates the completion of the bucket relocation operation.
    true , false
    kind
    Indicates that this resource represents a storage operation.
    metadata
    Provides information about the operation.
    metadata.@type
    Indicates the type of operation as bucket relocation.
    metadata.commonMetadata
    Metadata common to all operations.
    metadata.commonMetadata.createTime
    The time the long-running operation was created.
    metadata.commonMetadata.endTime
    The time the long-running operation ended.
    metadata.commonMetadata.progressPercent
    The estimated progress of the long-running operation, in percentage.
    Between 0 to 100 %. A value of -1 means that the progress is unknown or not applicable.
    metadata.commonMetadata.requestedCancellation
    Indicates whether the user has requested cancellation of the long-running operation.
    true , false
    metadata.commonMetadata.type
    Indicates the type of the long-running operation.
    metadata.commonMetadata.updateTime
    The time the long-running operation was last updated.
    metadata.destinationLocation
    The destination location of the bucket.
    metadata.finalizationState
    Indicates the readiness for initiating the final synchronization step .
    • READY : Indicates that you can initiate the final synchronization step. However, we recommend you wait until the value of the progressPercent field reaches 99 .
    • WAITING_ON_SYNC : Indicates that you cannot initiate the final synchronization step.
    • NOT_REQUIRED : Indicates that the final synchronization step is not necessary for this bucket and you can skip it.
    • BLOCKED_ON_ERRORS : Indicates that the finalization step is temporarily paused due to errors. You'll need to resolve the errors to proceed with the step.
    • RUNNING : Indicates that the finalization step is in progress.
    • FINALIZED : Indicates that the finalization step has completed successfully.
    metadata.progress
    Progress details of the relocation operation.
    metadata.progress.byteProgressPercent
    Progress of bytes copied in percentage.
    Between 0 to 100 %. A value of -1 means that the progress is unknown or not applicable.
    metadata.progress.discoveredBytes
    Number of bytes discovered in the source bucket.
    metadata.progress.discoveredObjectCount
    Number of objects discovered in the source bucket.
    metadata.progress.discoveredSyncCount
    Number of object metadata updates discovered in the source bucket.
    metadata.progress.objectProgressPercent
    Progress of objects copied in percentage.
    Between 0 to 100 %. A value of -1 means that the progress is unknown or not applicable.
    metadata.progress.remainingBytes
    Number of bytes remaining to be copied from the source bucket to the destination bucket.
    metadata.progress.remainingObjectCount
    Number of objects remaining to be copied from the source bucket to the destination bucket.
    metadata.progress.remainingSyncCount
    Number of remaining object metadata updates to be synced.
    metadata.progress.syncProgressPercent
    Progress of object metadata updates to be synced in percentage.
    Between 0 to 100 %. A value of -1 means that the progress is unknown or not applicable.
    metadata.relocationState
    Overall state of the bucket relocation operation.
    • SYNCING : Indicates that the incremental data copy step is actively copying objects from the source bucket to the destination bucket.
    • FINALIZING : Indicates that the finalization step has been initiated.
    • FAILED : Indicates that the incremental data copy step encountered an error and did not complete successfully.
    • SUCCEEDED : Indicates that the incremental data copy step has completed successfully.
    • CANCELLED : Indicates that the incremental data copy step was canceled.
    metadata.sourceLocation
    The source location of the bucket.
    metadata.validateOnly
    Indicates if a dry run of the bucket relocation has been initiated.
    true , false
    metadata.estimatedWriteDowntimeDuration
    The estimated write downtime duration; populated once the finalizationState is READY .
    Minimum value is 7200s .
    metadata.writeDowntimeExpireTime
    The time the write downtime expires.
    name
    The unique identifier for this relocation operation.
    Format: projects/_/buckets/ bucket-name /operations/ operation-id
    response
    The response of the operation.
    response.@type
    The type of the response.
    selfLink
    A link to this operation.

    If you encounter issues when interacting with other Cloud Storage features, see Limitations .

    Initiate the final synchronization step

    The final synchronization step involves a period where you cannot perform write operations on the bucket. We recommend that you schedule the final synchronization step at a time that minimizes disruption to your applications.

    Before you proceed, confirm that the bucket is fully prepared by checking the finalizationState value in the output of the incremental data copy step. The finalizationState value must be READY to proceed.

    If you initiate the final synchronization step prematurely, the command returns an error message The relocate bucket operation is not ready to advance to finalization running state but the relocation process continues.

    We recommend that you wait until the progressPercent value is 99 before initiating the final synchronization step.

    Command line

    Initiate the final synchronization step of the bucket relocation operation once the finalizationState value is READY :

    gcloud storage buckets relocate --finalize --operation=projects/_/buckets/ BUCKET_NAME 
/operations/ OPERATION_ID

    Where:

    • BUCKET_NAME is the name of the bucket that you want to relocate.
    • OPERATION_ID is the ID of the long-running operation, which is returned in the response of methods you call. For example, the following response is returned from calling gcloud storage operations list and the long-running operation ID is AbCJYd8jKT1n-Ciw1LCNXIcubwvij_TdqO-ZFjuF2YntK0r74 .
    `name: projects/_/buckets/my-bucket/operations/AbCJYd8jKT1n-Ciw1LCNXIcubwvij_TdqO-ZFjuF2YntK0r74`

    Set the ttl flag to have greater control over the relocation process. For example:

    gcloud storage buckets relocate --finalize --ttl TTL_DURATION 
--operation=projects/_/buckets/ BUCKET_NAME 
/operations/ OPERATION_ID

    Where:

    TTL_DURATION is the Time to live (TTL) for the write downtime phase during a relocation process. It is expressed as a string, such as 12h for 12 hours. The TTL_DURATION determines the maximum allowed duration for the write downtime phase. If the write downtime exceeds this limit, the relocation process automatically reverts to the incremental copy step, and write operations to the bucket are re-enabled. The value must be within the range of 6h (6 hours) to 48h (48 hours). If not specified, the default value is 12h (12 hours).

    REST APIs

    JSON API

    1. Have gcloud CLI installed and initialized , which lets you generate an access token for the Authorization header.

    2. Create a JSON file that contains the settings for bucket relocation. See the Buckets: advanceRelocateBucket documentation for a complete list of settings. The following are common settings to include:

       { 
  
 "expireTime" 
 : 
  
 " EXPIRE_TIME 
" 
 , 
  
 "ttl" 
 : 
  
 " TTL_DURATION 
" 
 }

      Where:

      • EXPIRE_TIME is the time the write downtime expires.
      • TTL_DURATION is the Time to live (TTL) for the write downtime phase during a relocation process. It is expressed as a string, such as 12h for 12 hours. The TTL_DURATION determines the maximum allowed duration for the write downtime phase. If the write downtime exceeds this limit, the relocation process automatically reverts to the incremental copy step, and write operations to the bucket are re-enabled. The value must be within the range of 6h (6 hours) to 48h (48 hours). If not specified, the default value is 12h (12 hours).

    3. Use cURL to call the JSON API :

      curl -X POST --data-binary @ JSON_FILE_NAME 
\
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://storage.googleapis.com/storage/v1/b/bucket/ BUCKET_NAME 
/operations/ OPERATION_ID 
/advanceRelocateBucket"

      Where:

      • JSON_FILE_NAME is the name of the JSON file you created.
      • BUCKET_NAME is the name of the bucket you want to relocate.
      • OPERATION_ID is the ID of the long-running operation, which is returned in the response of methods you call. For example, the following response is returned from calling Operations: list and the long-running operation ID is AbCJYd8jKT1n-Ciw1LCNXIcubwvij_TdqO-ZFjuF2YntK0r74 .

    Validate the bucket relocation process

    After initiating a relocation, verify its successful completion. This section provides guidance on confirming the successful transfer of data.

    Validate the success of the relocation process using the following methods:

    • Poll long-running operations: Bucket relocation is a long-running operation. You can poll the long-running operation using the operation id to monitor the operation's progress and confirm its successful completion by verifying the success state. This involves periodically querying the operation's status until it reaches a terminal state. For information about monitoring long-running operations, see Use long-running operations in Cloud Storage .

    • Analyze Cloud Audit Logs entries: Cloud Audit Logs provides a detailed record of events and operations in your Google Cloud environment. You can analyze the Cloud Audit Logs entries associated with the relocation to validate its success. Analyze the logs for any errors, warnings, or unexpected behavior that might indicate issues during the transfer. For information about viewing Cloud Audit Logs logs, see Viewing audit logs .

      The following log entries help you to determine if your move is a success or a failure:

      • Successful relocation: Relocate bucket succeeded. All existing objects are now in the new placement configuration.

      • Failed relocation: Relocate bucket has failed. Bucket location remains unchanged.

      Using Pub/Sub notifications, you can also set up alerts that notify when the specific success or failure event appears in the logs. For information about setting up Pub/Sub notifications, see Configure Pub/Sub notifications for Cloud Storage .

    Complete the post bucket relocation tasks

    After you have successfully relocated your bucket, complete the following steps:

    1. Optional: Restore any tag-based access controls on your bucket.
    2. Existing inventory report configurations are not preserved during the relocation process and you'll need to manually recreate them. For information about creating an inventory report configuration, see Create an inventory report configuration .
    3. Update your infrastructure as code configurations such as Terraform and Google Kubernetes Engine configuration connector to specify the bucket's new location.
    4. Regional endpoints are tied to specific locations, and you'll need to modify your application code to reflect the new endpoint.

    How to handle failed bucket relocation operations

    Consider the following factors before handling failed bucket relocation operations:

    • A failed bucket relocation might leave obsolete resources, such as temporary files or incomplete data copies, at the destination. You must wait 7 to 14 days before initiating another bucket relocation to the same destination. You can initiate a bucket relocation to a different destination immediately.

    • If the destination location is not the optimal location for your data, you might want to roll back the relocation. However, you cannot initiate a relocation immediately. A waiting period of up to 14 days is required before you can initiate the relocation process again. This restriction is in place to ensure stability and prevent data conflicts.

    What's next
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: