In the previous posts we discussed the following:
- Part1 - Prerequisites
- Part2 - Configure NSX
- Part3 - Edge Cluster
- Part4 - Tier-0 Gateway and BGP peering
- Part5 - Tier-1 Gateway and Segments
- Part6 - Create tags, storage policy, and content library
- Part7 - Enable workload management
- Part8 - Create namespace and deploy Tanzu Kubernetes Cluster
- Part9 - Monitoring
- Part10 - Upgrade Tanzu Kubernetes Cluster
- Part11 - Troubleshooting TKC
- Part12 - Deploy application on TKC and access it
- Part13 - Export WCP admin kubeconfig
This
article is about using kubestr to test storage options of Tanzu Kubernetes Cluster (TKC). Following are the steps to install kubestr on MAC:
- wget https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_MacOS_amd64.tar.gz
- tar -xvf kubestr_0.4.31_MacOS_amd64.tar.gz
- chmod +x kubestr
- mv kubestr /usr/local/bin
Now, lets do kubestr help.
% kubestr help
kubestr is a tool that will scan your k8s cluster
and validate that the storage systems in place as well as run
performance tests.
Usage:
kubestr [flags]
kubestr [command]
Available Commands:
browse Browse the contents of a CSI PVC via file browser
csicheck Runs the CSI snapshot restore check
fio Runs an fio test
help Help about any command
Flags:
-h, --help help for kubestr
-e, --outfile string The file where test results will be written
-o, --output string Options(json)
Use "kubestr [command] --help" for more information about a command.
I am going to use the following TKC for testing.
% KUBECONFIG=gc.kubeconfig kubectl get nodes
NAME STATUS ROLES AGE VERSION
gc-control-plane-pwngg Ready control-plane,master 103d v1.20.9+vmware.1
gc-workers-wrknn-f675446b6-cz766 Ready <none> 103d v1.20.9+vmware.1
gc-workers-wrknn-f675446b6-f6zqs Ready <none> 103d v1.20.9+vmware.1
gc-workers-wrknn-f675446b6-rsf6n Ready <none> 103d v1.20.9+vmware.1
Let's run kubestr against the cluster now.
% KUBECONFIG=gc.kubeconfig kubestr
**************************************
_ ___ _ ___ ___ ___ _____ ___
| |/ / | | | _ ) __/ __|_ _| _ \
| ' <| |_| | _ \ _|\__ \ | | | /
|_|\_\\___/|___/___|___/ |_| |_|_\
Explore your Kubernetes storage options
**************************************
Kubernetes Version Check:
Valid kubernetes version (v1.20.9+vmware.1) - OK
RBAC Check:
Kubernetes RBAC is enabled - OK
Aggregated Layer Check:
The Kubernetes Aggregated Layer is enabled - OK
W0130 14:17:16.937556 87541 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
Available Storage Provisioners:
csi.vsphere.xxxx.com:
Can't find the CSI snapshot group api version.
This is a CSI driver!
(The following info may not be up to date. Please check with the provider for more information.)
Provider: vSphere
Website: https://github.com/kubernetes-sigs/vsphere-csi-driver
Description: A Container Storage Interface (CSI) Driver for VMware vSphere
Additional Features: Raw Block,<br/><br/>Expansion (Block Volume),<br/><br/>Topology Aware (Block Volume)
Storage Classes:
* sc2-01-vc16c01-wcp-mgmt
To perform a FIO test, run-
./kubestr fio -s <storage class>
You can run storage tests using kubestr and it uses FIO for generating IOs. For example this is how you can run a basic storage test.
% KUBECONFIG=gc.kubeconfig kubestr fio -s sc2-01-vc16c01-wcp-mgmt -z 10G
PVC created kubestr-fio-pvc-zvdhr
Pod created kubestr-fio-pod-kdbs5
Running FIO test (default-fio) on StorageClass (sc2-01-vc16c01-wcp-mgmt) with a PVC of Size (10G)
Elapsed time- 29.290421119s
FIO test results:
FIO version - fio-3.20
Global options - ioengine=libaio verify=0 direct=1 gtod_reduce=1
JobName: read_iops
blocksize=4K filesize=2G iodepth=64 rw=randread
read:
IOPS=3987.150391 BW(KiB/s)=15965
iops: min=3680 max=4274 avg=3992.034424
bw(KiB/s): min=14720 max=17096 avg=15968.827148
JobName: write_iops
blocksize=4K filesize=2G iodepth=64 rw=randwrite
write:
IOPS=3562.628906 BW(KiB/s)=14267
iops: min=3237 max=3750 avg=3565.896484
bw(KiB/s): min=12950 max=15000 avg=14264.862305
JobName: read_bw
blocksize=128K filesize=2G iodepth=64 rw=randread
read:
IOPS=2988.549316 BW(KiB/s)=383071
iops: min=2756 max=3252 avg=2992.344727
bw(KiB/s): min=352830 max=416256 avg=383056.187500
JobName: write_bw
blocksize=128k filesize=2G iodepth=64 rw=randwrite
write:
IOPS=2754.796143 BW(KiB/s)=353151
iops: min=2480 max=2992 avg=2759.586182
bw(KiB/s): min=317440 max=382976 avg=353242.781250
Disk stats (read/write):
sdd: ios=117160/105647 merge=0/1210 ticks=2100090/2039676 in_queue=4139076, util=99.608589%
- OK
As you can see, a PVC of 10G, a FIO pod will be created, and this will be used for the FIO test. Once the test is complete, the PVC and FIO pod will be deleted automatically.
I hope it was useful. Cheers!
