Deploy a stateful workload with Filestore

Configure the managed file storage with Filestore using CSI

GKE provides a way to automatically deploy and manage the Kubernetes Filestore CSI driver in your clusters. Using Filestore CSI allows you to dynamically create or delete Filestore instances and use them in Kubernetes workloads with a StorageClass or a Deployment .

You can create a new Filestore instance by creating a PVC that dynamically provisions a Filestore instance and the PV, or access pre-provisioned Filestore instances in Kubernetes workloads.

  apiVersion 
 : 
  
 storage.k8s.io/v1 
 kind 
 : 
  
 StorageClass 
 metadata 
 : 
  
 name 
 : 
  
 filestore-sc 
 provisioner 
 : 
  
 filestore.csi.storage.gke.io 
 volumeBindingMode 
 : 
  
 Immediate 
 allowVolumeExpansion 
 : 
  
 true 
 parameters 
 : 
  
 tier 
 : 
  
 standard 
  
 network 
 : 
  
 default 
 

  apiVersion 
 : 
  
 storage.k8s.io/v1 
 kind 
 : 
  
 StorageClass 
 metadata 
 : 
  
 name 
 : 
  
 filestore-sc 
 provisioner 
 : 
  
 filestore.csi.storage.gke.io 
 volumeBindingMode 
 : 
  
 Immediate 
 allowVolumeExpansion 
 : 
  
 true 
 

  apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 PersistentVolume 
 metadata 
 : 
  
 name 
 : 
  
 fileserver 
  
 annotations 
 : 
  
 pv.kubernetes.io/provisioned-by 
 : 
  
 filestore.csi.storage.gke.io 
 spec 
 : 
  
 storageClassName 
 : 
  
 filestore-sc 
  
 capacity 
 : 
  
 storage 
 : 
  
 1Ti 
  
 accessModes 
 : 
  
 - 
  
 ReadWriteMany 
  
 persistentVolumeReclaimPolicy 
 : 
  
 Delete 
  
 volumeMode 
 : 
  
 Filesystem 
  
 csi 
 : 
  
 driver 
 : 
  
 filestore.csi.storage.gke.io 
  
 # Modify this to use the zone, filestore instance and share name. 
   
 volumeHandle 
 : 
  
 "modeInstance/<LOCATION>/<INSTANCE_NAME>/<FILE_SHARE_NAME>" 
  
 volumeAttributes 
 : 
   
 ip 
 : 
  
< IP_ADDRESS 
>  
 # Modify this to Pre-provisioned Filestore instance IP 
   
 volume 
 : 
  
< FILE_SHARE_NAME 
>  
 # Modify this to Pre-provisioned Filestore instance share name 
 

Use a PersistentVolumeClaim to access the volume

The following pvc.yaml manifest references the Filestore CSI driver's StorageClass named filestore-sc .

In order to have multiple Pods reading and writing to the volume, the accessMode is set to ReadWriteMany .

  kind 
 : 
  
 PersistentVolumeClaim 
 apiVersion 
 : 
  
 v1 
 metadata 
 : 
  
 name 
 : 
  
 fileserver 
 spec 
 : 
  
 accessModes 
 : 
  
 - 
  
 ReadWriteMany 
  
 storageClassName 
 : 
  
 filestore-sc 
  
 resources 
 : 
  
 requests 
 : 
  
 storage 
 : 
  
 1Ti 
 

1. Deploy the PVC:

    kubectl  
   create  
   -f  
   pvc.yaml 
   

2. Verify that the PVC is created:

    kubectl  
   get  
   pvc 
   

   The output is similar to the following:

    NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
   fileserver   Bound    pvc-aadc7546-78dd-4f12-a909-7f02aaedf0c3   1Ti        RWX            filestore-sc        92m 
   

3. Verify that the newly created Filestore instance is ready:

    gcloud  
   filestore  
   instances  
   list 
   

   The output is similar to the following:

    INSTANCE_NAME: pvc-5bc55493-9e58-4ca5-8cd2-0739e0a7b68c
   LOCATION: northamerica-northeast2-a
   TIER: STANDARD
   CAPACITY_GB: 1024
   FILE_SHARE_NAME: vol1
   IP_ADDRESS: 10.29.174.90
   STATE: READY
   CREATE_TIME: 2022-06-24T18:29:19 
   

Create a reader and a writer Pod

In this section, you create a reader Pod and a writer Pod. This tutorial uses Kubernetes Deployments to create the Pods. A Deployment is a Kubernetes API object that lets you run multiple replicas of Pods that are distributed among the nodes in a cluster..

Create the reader Pod

The reader Pod will read the file that is being written by the writers Pods. The reader Pods will see what time and which writer Pod replica wrote to the file.

  apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 Deployment 
 metadata 
 : 
  
 name 
 : 
  
 reader 
 spec 
 : 
  
 replicas 
 : 
  
 1 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 app 
 : 
  
 reader 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 app 
 : 
  
 reader 
  
 spec 
 : 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 nginx 
  
 image 
 : 
  
 nginx:stable-alpine 
  
 ports 
 : 
  
 - 
  
 containerPort 
 : 
  
 80 
  
 volumeMounts 
 : 
  
 - 
  
 name 
 : 
  
 fileserver 
   
 mountPath 
 : 
  
 /usr/share/nginx/html 
  
 # the shared directory 
  
 readOnly 
 : 
  
 true 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 fileserver 
  
 persistentVolumeClaim 
 : 
  
 claimName 
 : 
  
 fileserver 
 

The reader Pod will read from the path /usr/share/nginx/html which is shared between all the Pods.

1. Deploy the reader Pod:

    kubectl  
   apply  
   -f  
   reader-fs.yaml 
   

2. Verify that the reader replicas are running by querying the list of Pods:

    kubectl  
   get  
   pods 
   

   The output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
   reader-66b8fff8fd-jb9p4   1/1     Running   0          3m30s 
   

Create the writer Pod

The writer Pod will periodically write to a shared file that other writer and reader Pods can access. The writer Pod records its presence by writing its host name to the shared file.

The image used for the writer Pod is a custom image of Alpine Linux, which is used for utilities and production applications. It includes a script indexInfo.html that will obtain the metadata of the most recent writer, and keep count of all the unique writers and total writes.

  apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 Deployment 
 metadata 
 : 
  
 name 
 : 
  
 writer 
 spec 
 : 
   
 replicas 
 : 
  
 2 
  
 # start with 2 replicas 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 app 
 : 
  
 writer 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 app 
 : 
  
 writer 
  
 spec 
 : 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 content 
  
 image 
 : 
  
 us-docker.pkg.dev/google-samples/containers/gke/stateful-workload:latest 
  
 volumeMounts 
 : 
  
 - 
  
 name 
 : 
  
 fileserver 
   
 mountPath 
 : 
  
 /html 
  
 # the shared directory 
  
 command 
 : 
  
 [ 
 "/bin/sh" 
 , 
  
 "-c" 
 ] 
  
 args 
 : 
   
 - 
  
 cp /htmlTemp/indexInfo.html /html/index.html; 
   
 while true; do 
   
 echo "<b> Date :</b> <text>$(date)</text> <b> Writer :</b> <text2> ${HOSTNAME} </text2> <br>  " >> /html/indexData.html; 
   
 sleep 30; 
  
   
 done 
  
 volumes 
 : 
  
 - 
  
 name 
 : 
  
 fileserver 
  
 persistentVolumeClaim 
 : 
  
 claimName 
 : 
  
 fileserver 
 

For this tutorial, the writer Pod writes every 30 seconds to the path /html/index.html . Modify the sleep number value to have a different write frequency.

1. Deploy the writer Pod:

    kubectl  
   apply  
   -f  
   writer-fs.yaml 
   

2. Verify that the writer Pods are running by querying the list of Pods:

    kubectl  
   get  
   pods 
   

   The output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
   reader-66b8fff8fd-jb9p4   1/1     Running   0          3m30s
   writer-855565fbc6-8gh2k   1/1     Running   0          2m31s
   writer-855565fbc6-lls4r   1/1     Running   0          2m31s 
   

Expose and access the reader workload to a Service Load Balancer

To expose a workload outside the cluster, create a Service of type LoadBalancer . This type of Service creates an external load balancer with an IP address reachable through the internet.

1. Create a Service of type LoadBalancer named reader-lb :

    kubectl  
   create  
   -f  
   loadbalancer.yaml 
   

2. Watch the deployment to see that GKE assigns an EXTERNAL-IP for reader-lb Service:

    kubectl  
   get  
   svc  
   --watch 
   

   When the Service is ready, the EXTERNAL-IP column displays the public IP address of the load balancer:

    NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)        AGE
     kubernetes   ClusterIP      10.8.128.1    <none>          443/TCP        2d21h
     reader-lb    LoadBalancer   10.8.131.79   34.71.232.122   80:32672/TCP   2d20h 
   

3. Press Ctrl+Cto terminate the watch process.

4. Use a web browser to navigate to the EXTERNAL-IP assigned to the load balancer. The page refreshes every 30 seconds. The more writers Pods and shorter the frequency, the more entries it will show.

To see more details about the load balancer service, see loadbalancer.yaml .

Scale up the writer

Because the PV accessMode was set to ReadWriteMany , GKE can scale up the number of Pods so that more writer Pods can write to this shared volume (or more readers can read to read them).

1. Scale up the writer to five replicas:

    kubectl  
   scale  
   deployment  
   writer  
   --replicas = 
    5 
    
   

   The output is similar to the following:

    deployment.extensions/writer scaled 
   

2. Verify the number of running replicas:

    kubectl  
   get  
   pods 
   

   The output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
   reader-66b8fff8fd-jb9p4   1/1     Running   0          11m
   writer-855565fbc6-8dfkj   1/1     Running   0          4m
   writer-855565fbc6-8gh2k   1/1     Running   0          10m
   writer-855565fbc6-gv5rs   1/1     Running   0          4m
   writer-855565fbc6-lls4r   1/1     Running   0          10m
   writer-855565fbc6-tqwxc   1/1     Running   0          4m 
   

3. Use a web browser to navigate again to the EXTERNAL-IP assigned to the load balancer.

At this point, you configured and scaled your cluster to support five stateful writer Pods. Where multiple writer Pods are writing to the same file simultaneously. The reader Pods can also be easily scaled up.

Optional: Access data from the writer Pod

This section demonstrates how to use a command-line interface to access a reader or writer Pod. You can see the shared component that the writer is writing to and the reader is reading from.

1. Obtain the writer Pod name:

    kubectl  
   get  
   pods 
   

   The output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
   writer-5465d65b46-7hxv4   1/1     Running   0          20d 
   

   Note the hostname of a writer Pod (Example: writer-5465d65b46-7hxv4 ).

2. Run the following command to access the writer Pod:

    kubectl  
    exec 
     
   -it  
    WRITER_HOSTNAME 
     
   --  
   /bin/sh 
   

3. See the shared component in the file indexData.html :

     cd 
     
   /html
   cat  
   indexData.html 
   

4. Clear the indexData.html file:

     echo 
     
    '' 
    > 
   indexData.html 
   

   Refresh the web browser hosting the EXTERNAL-IP address to see the change.

5. Exit the environment:

     exit