Resizing persistent volumes in EKS/AKS/GKE
7d1
•May 1, 2024
(Source/Credits: https://dev.to/7d1/resizing-persistent-volumes-in-eks-aks-gke-3oa7)
Persistent volumes (PVs) in Kubernetes is the way to abstract out block volumes used by pods in the...
Persistent volumes (PVs) in Kubernetes is the way to abstract out block volumes used by pods in the cluster. The way pods request PVs is by creating Persistent Volume Claims (PVCs), which work like resource demand documents. Storage Class is responsible for actual implementation of how the PVs are actually handled, it can be provided by ceph volumes, or in the case of cloud providers, by EBS volumes in AWS, Managed Disks in Azure and Disks in GCP.
This post is a note made to myself, when I was checking how does the PV resize works in managed kuberentes services. Feel free to replicate the commands and see what happens by yourself.
Amazon Elastic Kubernetes Service
Let's first create the cluster fast with eksctl. Mind it takes a significant time.
eksctl create cluster -n darek -r eu-central-1
After a while you should see some nodes running.
```
➜ ~ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-26-190.eu-central-1.compute.internal Ready
```
Let's see the storage class.
yaml
➜ ~ kubectl get sc gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs"}
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2019-08-24T09:23:35Z"
name: gp2
resourceVersion: "281"
selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
uid: d903b89e-c650-11e9-968c-0273bdd9611a
parameters:
fsType: ext4
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
To resize volumes, storage class needs to have the allowVolumeExpansion set to true. Fortunatelly, it can be patched.
k patch sc gp2 -p '{"allowVolumeExpansion": true}'
Now, create a basic PVC of size 10Gi and storageclass set to gp2. We will us the same yaml for AKS and GKE later too, save it as pvc.yml.
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-volume-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp2
Create the PVC.
kubectl apply -f pvc.yml
The PVC should eventually create a PV of size 10, which is reflected in a real EBS volume created in the AWS. The name of the EBS is encoded in Source/VolumeID. You can get it by doing describe operation.
➜ kubectl describe pv/pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Name: pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Labels: failure-domain.beta.kubernetes.io/region=eu-central-1
failure-domain.beta.kubernetes.io/zone=eu-central-1b
Annotations: kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp2
Status: Bound
Claim: default/test-volume-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/zone in [eu-central-1b]
failure-domain.beta.kubernetes.io/region in [eu-central-1]
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: aws://eu-central-1b/vol-056a0bc0115292add
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
The above command shows the EBS volume name to be vol-056a0bc0115292add.
Now let's resize the PVC. Change the storage size in pvc.yml to 12Gi and do kubectl apply again.
➜ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-0262b546-c65a-11e9-8f74-06e2726fa54c 12Gi RWO Delete Bound default/test-volume-pvc gp2 19h
Ok so the PV shows 12Gi. Let's check the size of the actual EBS volume with aws cli.
aws ec2 describe-volumes --volume-ids vol-056a0bc0115292add --query 'Volumes[0].Size'
It should now show 12.
Have in mind that in Amazon you can do resize operation on EBS only once every 6h. So next resize won't work, unless you wait for it.
You can delete the clutser by doing eksctl delete cluster -n darek.
Azure AKS
Let's now check AKS. Create a cluster in AKS. The below snippet creates a resource group darek and a cluster called darekEKS.
az group create --name darek --location westeurope
az aks create --resource-group darek --name darekEKS --node-count 2 --enable-addons monitoring --generate-ssh-keys
az aks get-credentials --resource-group darek --name darekEKS
Storage class in AKS is called default, it will provision azure managed disk.
``` ➜ kubectl get sc NAME PROVISIONER AGE default (default) kubernetes.io/azure-disk 13m managed-premium kubernetes.io/azure-disk 13m
```
Let's see the whole class, similarly as in EKS the resize is not enbled.
➜ kubectl get storageclasses.storage.k8s.io default -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"cachingmode":"ReadOnly","kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
storageclass.beta.kubernetes.io/is-default-class: "true"
creationTimestamp: "2019-08-24T09:31:20Z"
labels:
kubernetes.io/cluster-service: "true"
name: default
resourceVersion: "459"
selfLink: /apis/storage.k8s.io/v1/storageclasses/default
uid: ee21d5ae-c651-11e9-9f0b-aac03b918570
parameters:
cachingmode: ReadOnly
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
Let's patch it: kubectl patch sc default -p '{"allowVolumeExpansion": true}'
And create the PVC, remember to change the storageClassName to default.
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-volume-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: default
Is PVC and PV created?
``` ➜ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test-volume-pvc Bound pvc-eb332309-c654-11e9-a0fd-36761452f1dd 10Gi RWO default 26s
➜ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-eb332309-c654-11e9-a0fd-36761452f1dd 10Gi RWO Delete Bound default/test-volume-pvc default 41s ```
The name of the Azure disk will be seen in Source/DiskURI.
```
➜ ~ kubectl describe pv
Name: pvc-eb332309-c654-11e9-a0fd-36761452f1dd
Labels:
```
Now change the disk size to 12 and see. We can check the disk size with the following command.
```
az disk show --ids /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd --query 'diskSizeGb' 12 ```
Ok works in AKS too. What is quite weird, when you do kubectl get pv it shows the new size but the PVC still shows 10Gi.
Clean resources you have created: az group delete -n darek -y
Google Kubernetes Engine
I created a zonal cluster using the CLI in my project called turnkey-cooler-31343.
gcloud config set project turnkey-cooler-31343
gcloud container clusters create darek --zone europe-north1-a --cluster-version "1.13.7-gke.19" --machine-type n1-standard-1 --num-nodes 2
And generate a kubeconfig:
gcloud container clusters get-credentials darek --zone europe-north1-a --project turnkey-cooler-31343
In GKE the default storageclass is called standard.
NAME PROVISIONER AGE
standard (default) kubernetes.io/gce-pd 3m44s
Just as in AKS and EKS the auto expand was not enabled, let's patch the storage class.
kubectl patch sc standard -p '{"allowVolumeExpansion": true}'
And create a pvc using our PVC yaml. Remember to change storageClassName to standard.
kubectl apply -f pvc.yml
And check the disk:
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 10Gi RWO Delete Bound default/test-volume-pvc standard 8m30s
Now check the name of the disk in GCP:
kubectl get pv/pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 -o jsonpath='{.spec.gcePersistentDisk.pdName}'
gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8
And the size of the disk.
➜ gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '10'
Let's now resize the disk, change the pvc.yml or patch the PVC.
➜ gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '12'
Ok all fine but as in AKS and EKS the PVC still shows 10Gi.
Clean the infra: gcloud container clusters delete darek --zone europe-north1-a.
Comments section