Recreate and update a Dataproc on GKE virtual cluster

gcloud

1. Set environment variables:

   CLUSTER= existing Dataproc on GKE  cluster name 
   \
     REGION=  region 
    
   

2. Export the existing Dataproc on GKE cluster configuration to a YAML file.

   gcloud dataproc clusters export $CLUSTER \
       --region=$REGION > "${CLUSTER}-config.yaml"

3. Update the configuration.

   1. Remove the kubernetesNamespace field. Removing this field is necessary to avoid a namespace conflict when you create the updated cluster.

      Sample sed command to remove the kubernetesNamespace field:

      sed -E "s/kubernetesNamespace: .+$//g" ${CLUSTER}-config.yaml

   2. Make additional changes to update Dataproc on GKE virtual cluster configuration settings, such as changing the Spark componentVersion .

4. Delete the existing Dataproc on GKE virtual cluster if you will create a cluster that has the same name as the cluster it is updating (if you are replacing the original cluster).

5. Wait for the previous delete operation to finish, and then import the updated cluster configuration to create a new Dataproc on GKE virtual cluster with the updated config settings.

   gcloud dataproc clusters import $CLUSTER \
       --region=$REGION \
       --source="${CLUSTER}-config.yaml"

API

1. Set environment variables:

   CLUSTER= existing Dataproc on GKE  cluster name 
   \
     REGION=  region 
    
   

2. Export the existing Dataproc on GKE cluster configuration to a YAML file.

   curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)"  "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}?alt=json" > "${CLUSTER}-config.json"

3. Update the configuration.

   1. Remove the kubernetesNamespace field. Removal of this field is necessary to avoid a namespace conflict when you create the updated cluster.

      Sample jq command to remove kubernetesNamespace field:

      jq 'del(.virtualClusterConfig.kubernetesClusterConfig.kubernetesNamespace)'

   2. Make additional changes to update Dataproc on GKE virtual cluster configuration settings, such as changing the Spark componentVersion .

4. Delete the existing Dataproc on GKE virtual cluster if you will create a cluster that has the same name as the cluster it is updating (if you are replacing the original cluster).

   curl -X DELETE -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}"

5. Wait for the previous delete operation to finish, and then import the updated cluster configuration to create a new Dataproc on GKE virtual cluster with the updated settings.

   curl -i -X POST  -H "Authorization: Bearer $(gcloud auth print-access-token)"  -H "Content-Type: application/json; charset=utf-8" -d "@${CLUSTER}-config.json" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters?alt=json"

Console

The Google Cloud console does not support recreating a Dataproc on GKE virtual cluster by importing an existing cluster's configuration.