Create an RDMA-enabled HPC Slurm cluster with H4D instances

Launch Cloud Shell

In this tutorial, you use Cloud Shell which is a shell environment for managing resources hosted on Google Cloud.

Cloud Shell comes preinstalled with the Google Cloud CLI . gcloud CLI provides the primary command-line interface for Google Cloud. To launch Cloud Shell:

1. Go to the Google Cloud console.

   Google Cloud console

2. From the upper-right corner of the console, click the Activate Cloud Shellbutton:

A Cloud Shell session starts and displays a command-line prompt. You use this shell to run gcloud and Cluster Toolkit commands.

Set environment variables

In Cloud Shell, set the environment variables described in the following sections to use for the remainder of the tutorial. These environment variables set placeholder values for the following tasks:

• Configures your project with the relevant values to access your reserved H4D instances.

• Sets up a Cloud Storage bucket to store Cluster Toolkit modules.

Reservation capacity variables

export H4D_RESERVATION_PROJECT_ID= H4D_RESERVATION_PROJECT_ID 
export H4D_RESERVATION_NAME= H4D_RESERVATION_NAME 
export H4D_DEPLOYMENT_NAME= H4D_DEPLOYMENT_NAME 
export H4D_REGION= H4D_REGION 
export H4D_ZONE= H4D_ZONE 
export H4D_DEPLOYMENT_FILE_NAME= H4D_DEPLOYMENT_FILE_NAME 

Replace the following:

• H4D_RESERVATION_PROJECT_ID - the Google Cloud project ID that was granted the H4D machine type reservation block.
• H4D_RESERVATION_NAME - the name of your VM reservation block, found in your project. For example, h4d-highmem-exr .
• H4D_DEPLOYMENT_NAME - a unique name for your Slurm cluster deployment. For example, h4d-hpc-slurm-cluster-deployment .
• H4D_REGION - the region that is running the reserved H4D machine reservation block. For example, us-central1 .
• H4D_ZONE - the zone that contains the reserved machines. This string must contain both the region and zone. For example, us-central1-a .
• H4D_DEPLOYMENT_FILE_NAME - a unique name for your Slurm blueprint YAML file. If you run through this tutorial more than once, choose a unique deployment name each time.

Storage capacity variables

Create the environment variables for your Cloud Storage bucket.

Cluster Toolkit uses blueprints to define and deploy clusters of VMs. A blueprint defines one or more Terraform modules to provision Cloud infrastructure. This bucket is used to store these blueprints.

export GOOGLE_CLOUD_BUCKET_NAME= BUCKET_NAME 
export GOOGLE_CLOUD_BUCKET_LOCATION= BUCKET_LOCATION 

Replace the following:

• BUCKET_NAME - the name that you want to use for your Cloud Storage bucket that meets the bucket naming requirements .
• BUCKET_LOCATION - any Google Cloud region of your choice, where the bucket will be hosted. For example, us-central1 .

Switch to the project with the reserved H4D capacity block

Run the following command to ensure that you are in the Google Cloud project that has the approved reservation block for the H4D instances.

gcloud config set project ${H4D_RESERVATION_PROJECT_ID}

Create a Cloud Storage bucket

Create the bucket to store your Terraform modules. From Cloud Shell, using your environment variables, run the following command:

A best practice when working with Terraform is to store the state remotely in a version-enabled file. On Google Cloud, you can create a Cloud Storage bucket that has versioning enabled.

gcloud storage buckets create gs://${GOOGLE_CLOUD_BUCKET_NAME} \
    --project=${H4D_RESERVATION_PROJECT_ID} \
    --default-storage-class=STANDARD \
    --location=${GOOGLE_CLOUD_BUCKET_LOCATION} \
    --uniform-bucket-level-access

gcloud storage buckets update gs://${GOOGLE_CLOUD_BUCKET_NAME} --versioning

Set up the Cluster Toolkit

To create a Slurm cluster in a Google Cloud project, you can use Cluster Toolkit to handle deploying and provisioning the cluster. Cluster Toolkit is open-source software offered by Google Cloud to simplify the process of deploying workloads on Google Cloud.

Use the following steps to set up Cluster Toolkit.

Clone the Cluster Toolkit GitHub repository

1. In Cloud Shell, clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/cluster-toolkit.git

2. Go to the main working directory:

   cd cluster-toolkit/

Build the Cluster Toolkit binary

1. In Cloud Shell, build the Cluster Toolkit binary from source by running the following command:

   make

2. To verify the build, run the following command:

   To deploy an HPC-optimized Slurm cluster that uses H4D instances, you must use version v1.70.0 or later of the Cluster Toolkit.

   ./gcluster --version

   After building the binary, you are now ready to deploy clusters to run your jobs or workloads.

Create a deployment file

1. In the Cluster Toolkit directory, create your Slurm deployment YAML file.

   nano ${H4D_DEPLOYMENT_FILE_NAME}.yaml

2. Paste the following content into the YAML file.

   ---
   terraform_backend_defaults:
     type: gcs
     configuration:
       bucket: GOOGLE_CLOUD_BUCKET_NAME 
   vars:
     deployment_name: H4D_DEPLOYMENT_FILE_NAME 
   project_id: H4D_RESERVATION_PROJECT_ID 
   region: H4D_REGION 
   zone: H4D_ZONE 
   

3. To save and exit the file, press Ctrl+O > Enter > Ctrl+X.

Provision the Slurm cluster

To provision the Slurm cluster, run the following deployment command. This command provisions the Slurm cluster with the H4D Cluster Toolkit blueprint .

In Cloud Shell, start the cluster creation.

./gcluster deploy -d ${H4D_DEPLOYMENT_FILE_NAME}.yaml examples/hpc-slurm-h4d/hpc-slurm-h4d.yaml --auto-approve

Connect to the cluster

After deploying, connect to the Google Cloud console to view your cluster.

1. Go to the Compute Engine> VM instancespage in the Google Cloud console.

   Go to VM instances

2. Locate the login node, which might have a name like example-deployment-login-123 .

3. Click SSHto connect.