Using the Compute Engine persistent disk CSI Driver

Google Kubernetes Engine (GKE) provides a simple way for you to automatically deploy and manage the Compute Engine persistent disk Container Storage Interface (CSI) Driver in your clusters. The Compute Engine persistent disk CSI Driver is always enabled in Autopilot clusters and can't be disabled or edited. In Standard clusters, you must enable the Compute Engine persistent disk CSI Driver.

The Compute Engine persistent disk CSI Driver version is tied to GKE version numbers. The Compute Engine persistent disk CSI Driver version is typically the latest driver available at the time that the GKE version is released. The drivers update automatically when the cluster is upgraded to the latest GKE patch.

Benefits

Using the Compute Engine persistent disk CSI Driver provides the following benefits:

• It enables the automatic deployment and management of the persistent disk driver without having to manually set it up.
• You can use customer-managed encryption keys (CMEKs). These keys are used to encrypt the data encryption keys that encrypt your data. To learn more about CMEK on GKE, see Using CMEK .
• You can use volume snapshots with the Compute Engine persistent disk CSI Driver. Volume snapshots let you create a copy of your volume at a specific point in time. You can use this copy to bring a volume back to a prior state or to provision a new volume.
• You can use volume cloning with the Compute Engine persistent disk CSI Driver in clusters running GKE version 1.22 and later. Volume cloning lets you create a duplicate of your volume at a specific point in time, provisioned with all of the data of from the source volume.
• Bug fixes and feature updates are rolled out independently from minor Kubernetes releases. This release schedule typically results in a faster release cadence.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update .

Enable the Compute Engine persistent disk CSI Driver

To enable the Compute Engine persistent disk CSI Driver in existing Standard clusters, use the Google Cloud CLI or the Google Cloud console.

To enable the driver on an existing cluster, complete the following steps:

 gcloud  
container  
clusters  
update  
 CLUSTER-NAME 
  
 \ 
  
--update-addons = 
 GcePersistentDiskCsiDriver 
 = 
ENABLED 

After you have enabled the Compute Engine persistent disk CSI Driver, you can use the driver in Kubernetes volumes by using the driver and provisioner name: pd.csi.storage.gke.io .

Disable the Compute Engine persistent disk CSI Driver

You can disable the Compute Engine persistent disk CSI Driver for Standard clusters by using Google Cloud CLI or the Google Cloud console.

If you disable the driver, then any Pods currently using PersistentVolumes owned by the driver do not terminate. Any new Pods that try to use those PersistentVolumes also fail to start.

To disable the driver on an existing Standard cluster, complete the following steps:

 gcloud  
container  
clusters  
update  
 CLUSTER-NAME 
  
 \ 
  
--update-addons = 
 GcePersistentDiskCsiDriver 
 = 
DISABLED 

Use the Compute Engine persistent disk CSI Driver for Linux clusters

The following sections describe the typical process for using a Kubernetes volume backed by a CSI driver in GKE. These sections are specific to clusters using Linux.

Create a StorageClass

After you enable the Compute Engine persistent disk CSI Driver, GKE automatically installs the following StorageClasses :

• standard-rwo , using balanced persistent disk
• premium-rwo , using SSD persistent disk

For Autopilot clusters, the default StorageClass is standard-rwo , which uses the Compute Engine persistent disk CSI Driver. For Standard clusters, the default StorageClass uses the Kubernetes in-tree gcePersistentDisk volume plugin.

You can find the name of your installed StorageClasses by running the following command:

 kubectl  
get  
sc 

You can also install a different StorageClass that uses the Compute Engine persistent disk CSI Driver by adding pd.csi.storage.gke.io in the provisioner field.

For example, you could create a StorageClass using the following file named pd-example-class.yaml :

  apiVersion 
 : 
  
 storage.k8s.io/v1 
 kind 
 : 
  
 StorageClass 
 metadata 
 : 
  
 name 
 : 
  
 pd-example 
 provisioner 
 : 
  
 pd.csi.storage.gke.io 
 volumeBindingMode 
 : 
  
 WaitForFirstConsumer 
 allowVolumeExpansion 
 : 
  
 true 
 parameters 
 : 
  
 type 
 : 
  
 pd-balanced 
 

You can specify the following Persistent Disk types in the type parameter:

• pd-balanced
• pd-ssd
• pd-standard
• pd-extreme (supported on GKE version 1.26 and later)

If using pd-standard or pd-extreme , see Unsupported machine types for additional usage restrictions.

When you use the pd-extreme option, you must also add the provisioned-iops-on-create field to your manifest. This field must be set to the same value as the provisioned IOPS value that you specified when you created your persistent disk.

  apiVersion 
 : 
  
 storage.k8s.io/v1 
 kind 
 : 
  
 StorageClass 
 metadata 
 : 
  
 name 
 : 
  
 pd-extreme-example 
 provisioner 
 : 
  
 pd.csi.storage.gke.io 
 volumeBindingMode 
 : 
  
 WaitForFirstConsumer 
 allowVolumeExpansion 
 : 
  
 true 
 parameters 
 : 
  
 type 
 : 
  
 pd-extreme 
  
 provisioned-iops-on-create 
 : 
  
 '10000' 
 

After creating the pd-example-class.yaml file, run the following command:

 kubectl  
create  
-f  
pd-example-class.yaml 

Create a PersistentVolumeClaim

You can create a PersistentVolumeClaim that references the Compute Engine persistent disk CSI Driver's StorageClass.

The following file, named pvc-example.yaml , uses the pre-installed storage class standard-rwo :

  kind 
 : 
  
 PersistentVolumeClaim 
 apiVersion 
 : 
  
 v1 
 metadata 
 : 
  
 name 
 : 
  
 podpvc 
 spec 
 : 
  
 accessModes 
 : 
  
 - 
  
 ReadWriteOnce 
  
 storageClassName 
 : 
  
 standard-rwo 
  
 resources 
 : 
  
 requests 
 : 
  
 storage 
 : 
  
 6Gi 
 

After creating the PersistentVolumeClaim manifest, run the following command:

 kubectl  
create  
-f  
pvc-example.yaml 

In the pre-installed StorageClass ( standard-rwo ), volumeBindingMode is set to WaitForFirstConsumer . When volumeBindingMode is set to WaitForFirstConsumer , the PersistentVolume is not provisioned until a Pod referencing the PersistentVolumeClaim is scheduled. If volumeBindingMode in the StorageClass is set to Immediate (or it's omitted), a persistent-disk-backed PersistentVolume is provisioned after the PersistentVolumeClaim is created.

Create a Pod that consumes the volume

When using Pods with PersistentVolumes, we recommend that you use a workload controller (such as a Deployment or StatefulSet). While you would not typically use a standalone Pod, the following example uses one for simplicity.

The following example consumes the volume that you created in the previous section:

  apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Pod 
 metadata 
 : 
  
 name 
 : 
  
 web-server 
 spec 
 : 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 web-server 
  
 image 
 : 
  
 nginx 
  
 volumeMounts 
 : 
  
 - 
  
 mountPath 
 : 
  
 /var/lib/www/html 
  
 name 
 : 
  
 mypvc 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 mypvc 
  
 persistentVolumeClaim 
 : 
  
 claimName 
 : 
  
 podpvc 
  
 readOnly 
 : 
  
 false 
 

Use the Compute Engine persistent disk CSI Driver for Windows clusters

The following sections describe the typical process for using a Kubernetes volume backed by a CSI driver in GKE. These sections are specific to clusters using Windows.

Ensure that the:

• Cluster version is 1.19.7-gke.2000, 1.20.2-gke.2000, or later.
• Node versions is 1.18.12-gke.1203, 1.19.6-gke.800, or later.

Create a StorageClass

Creating a StorageClass for Windows is very similar to Linux. You should be aware that the StorageClass installed by default will not work for Windows because the file system type is different. Compute Engine persistent disk CSI Driver for Windows requires NTFS as the file system type.

For example, you could create a StorageClass using the following file named pd- windows-class.yaml . Make sure to add csi.storage.k8s.io/fstype: NTFS to the parameters list:

  apiVersion 
 : 
  
 storage.k8s.io/v1 
 kind 
 : 
  
 StorageClass 
 metadata 
 : 
  
 name 
 : 
  
 pd-sc-windows 
 provisioner 
 : 
  
 pd.csi.storage.gke.io 
 volumeBindingMode 
 : 
  
 WaitForFirstConsumer 
 allowVolumeExpansion 
 : 
  
 true 
 parameters 
 : 
  
 type 
 : 
  
 pd-balanced 
  
 csi.storage.k8s.io/fstype 
 : 
  
 NTFS 
 

Create a PersistentVolumeClaim

After creating a StorageClass for Windows, you can now create a PersistentVolumeClaim that references that StorageClass:

  kind 
 : 
  
 PersistentVolumeClaim 
 apiVersion 
 : 
  
 v1 
 metadata 
 : 
  
 name 
 : 
  
 podpvc-windows 
 spec 
 : 
  
 accessModes 
 : 
  
 - 
  
 ReadWriteOnce 
  
 storageClassName 
 : 
  
 pd-sc-windows 
  
 resources 
 : 
  
 requests 
 : 
  
 storage 
 : 
  
 6Gi 
 

Create a Pod that consumes the volume

The following example consumes the volume that you created in the previous task:

  apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Pod 
 metadata 
 : 
  
 name 
 : 
  
 web-server 
 spec 
 : 
  
 nodeSelector 
 : 
  
 kubernetes.io/os 
 : 
  
 windows 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 iis-server 
  
 image 
 : 
  
 mcr.microsoft.com/windows/servercore/iis 
  
 ports 
 : 
  
 - 
  
 containerPort 
 : 
  
 80 
  
 volumeMounts 
 : 
  
 - 
  
 mountPath 
 : 
  
 /var/lib/www/html 
  
 name 
 : 
  
 mypvc 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 mypvc 
  
 persistentVolumeClaim 
 : 
  
 claimName 
 : 
  
 podpvc-windows 
  
 readOnly 
 : 
  
 false 
 

Use the Compute Engine persistent disk CSI Driver with non-default filesystem types

The default filesystem type for Compute Engine persistent disks in GKE is ext4 . You can also use the xfs storage type as long as your node image supports it. See Storage driver support for a list of supported drivers by node image.

The following example shows you how to use xfs as the default filesystem type instead of ext4 with the Compute Engine persistent disk CSI Driver.

Create a StorageClass

1. Save the following manifest as a YAML file named pd-xfs-class.yaml :

     apiVersion 
    : 
     
    storage.k8s.io/v1 
    kind 
    : 
     
    StorageClass 
    metadata 
    : 
     
    name 
    : 
     
    xfs-class 
    provisioner 
    : 
     
    pd.csi.storage.gke.io 
    parameters 
    : 
     
    type 
    : 
     
    pd-balanced 
     
    csi.storage.k8s.io/fstype 
    : 
     
    xfs 
    volumeBindingMode 
    : 
     
    WaitForFirstConsumer 
    
   

2. Apply the manifest:

    kubectl  
   apply  
   -f  
   pd-xfs-class.yaml 
   

Create a PersistentVolumeClaim

1. Save the following manifest as pd-xfs-pvc.yaml :

     apiVersion 
    : 
     
    v1 
    kind 
    : 
     
    PersistentVolumeClaim 
    metadata 
    : 
     
    name 
    : 
     
    xfs-pvc 
    spec 
    : 
     
    storageClassName 
    : 
     
    xfs-class 
     
    accessModes 
    : 
     
    - 
     
    ReadWriteOnce 
     
    resources 
    : 
     
    requests 
    : 
     
    storage 
    : 
     
    10Gi 
    
   

2. Apply the manifest:

    kubectl  
   apply  
   -f  
   pd-xfs-pvc.yaml 
   

Create a Pod that consumes the volume

1. Save the following manifest as pd-xfs-pod.yaml :

     apiVersion 
    : 
     
    v1 
    kind 
    : 
     
    Pod 
    metadata 
    : 
     
    name 
    : 
     
    pd-xfs-pod 
    spec 
    : 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    cloud-sdk 
     
    image 
    : 
     
    google/cloud-sdk:slim 
     
    args 
    : 
     
    [ 
    "sleep" 
    , 
    "3600" 
    ] 
     
    volumeMounts 
    : 
     
    - 
     
    mountPath 
    : 
     
    /xfs 
     
    name 
    : 
     
    xfs-volume 
     
    volumes 
    : 
     
    - 
     
    name 
    : 
     
    xfs-volume 
     
    persistentVolumeClaim 
    : 
     
    claimName 
    : 
     
    xfs-pvc 
    
   

2. Apply the manifest:

    kubectl  
   apply  
   -f  
   pd-xfs-pod.yaml 
   

Verify that the volume was mounted correctly

1. Open a shell session in the Pod:

    kubectl  
    exec 
     
   -it  
   pd-xfs-pod  
   --  
   /bin/bash 
   

2. Look for xfs partitions:

    df  
   -aTh  
   --type = 
   xfs 
   

   The output should be similar to the following:

    Filesystem     Type  Size  Used Avail Use% Mounted on
   /dev/sdb       xfs    30G   63M   30G   1% /xfs 
   

View logs for Compute Engine persistent disk CSI Driver

You can use Cloud Logging to view events that relate to the Compute Engine persistent disk CSI Driver. Logs can help you troubleshoot issues.

For more information about Cloud Logging, see Viewing your GKE logs .

To view logs for the Compute Engine persistent disk CSI Driver, complete the following steps:

1. Go to the Cloud Loggingpage in the Google Cloud console.

   Go to Cloud Logging

2. Run the following query:

      
   resource.type = 
    "k8s_container" 
     
   resource.labels.project_id = 
    " PROJECT_ID 
   " 
     
   resource.labels.location = 
    " LOCATION 
   " 
     
   resource.labels.cluster_name = 
    " CLUSTER_NAME 
   " 
     
   resource.labels.namespace_name = 
    "kube-system" 
     
   resource.labels.container_name = 
    "gce-pd-driver" 
    
   

   Replace the following:

   • PROJECT_ID : the name of your project.
   • LOCATION : the Compute Engine region or zone of the cluster.
   • CLUSTER_NAME : the name of your cluster.

Known issues

Unsupported machine types

If you are using the C3 series machine family, the pd-standard persistent disk type is not supported.

If you attempt to run a Pod on a machine, and the Pod uses an unsupported persistent disk type, you will see a warning message like the following emitted on the Pod:

 AttachVolume.Attach failed for volume "pvc-d7397693-5097-4a70-9df0-b10204611053" : rpc error: code = Internal desc = unknown Attach error: failed when waiting for zonal op: operation operation-1681408439910-5f93b68c8803d-6606e4ed-b96be2e7 failed (UNSUPPORTED_OPERATION): [pd-standard] features are not compatible for creating instance. 

If your cluster has multiple node pools with different machine families, you can use node taints and node affinity to limit where workloads can be scheduled. For example, you can use this approach to restrict a workload using pd-standard from running on an unsupported machine family.

If you are using the pd-extreme persistent disk type, you need to ensure that your disk is attached to a VM instance with a suitable machine shape. To learn more, refer to Machine shape support .

What's next

• Learn how to use volume expansion .
• Learn how to use volume snapshots .
• Learn how to use volume cloning .
• Learn how to create regional persistent disks .
• Read more about the driver on GitHub .