vSAN performance benchmarking

In this article, I will explain briefly on performance benchmarking considerations, factors affecting performance, and some of the best practices. We do performance benchmarking to understand the capabilities and bottlenecks of a system. When I say system it could be a storage system, CPU, GPU, network switch, etc. Now let's consider a VMware vSAN cluster infrastructure. It includes multiple components and each of these contributes to the performance. In this case, the vSAN cluster is the solution under test. We will have to conduct performance benchmarking to understand the storage performance behavior of the cluster. When I say storage behavior it includes the IOPS, latency, and throughput that the cluster can produce under varying loads.

The goal of benchmarking

• Identify bottlenecks
• Hardware bottleneck
• Software bottleneck
• Application bottleneck
• Compare tradeoffs
• Manage expectations
• Make decisions

Usually in a real-world scenario, benchmarking will be done once the cluster is deployed/ ready and before starting to host production workload on top of it. As these benchmark values define the performance maximums it will be helpful to decide on when to scale or upgrade the cluster before it hits a bottleneck.

Fundamental factors of vSAN performance

Server hardware

• Compatibility as per vSAN HCL

Host

• Number of hosts in the cluster
• Power settings
• CPU - number of cores and frequency 

Storage

• Hybrid or All-flash
• NVMe, SAS, or SATA
• Number of disk groups per host
• Storage controller configuration
• Compatibility of hardware devices as per vSAN HCL

Network

• 10/ 25/ 40 GbE
• MTU 
• LAG

SPBM policy

• FTT (Failures To Tolerate)
• FTM (Mirroring/ Erasure coding)
• Thin or Thick provision

Security

• Encryption
• Checksum

Other

• Stripe width
• Flash read cache reservation
• IOPS limit for object

All of the above factors will affect performance. So you should know the benefits and tradeoffs. 

Benchmarking methodology

Image credit: VMware

Storage benchmarking tools

IO load generation tools

• FIO
• Diskspd
• IOmeter
• HCIBench

Application-specific tools

• HammerDB (MSSQL, Oracle)
• Jetstress (MS Exchange)
• SLOB (Oracle)
• DBGen (MSSQL, Oracle)

Best practices

• Understand the production performance metrics.
• Test what you plan to deploy.
• Workload modeling.
• Plan for use case testing.
• Choose an appropriate size for benchmarking
• Choose the right tool.
• Pre-allocate blocks while testing.
• Test for a longer time duration.
• Deploy multiple VMs with multiple VMDKs.

References

Best Practices for HCI Performance Benchmarking (HCI1891BU)

A Guide to vSAN Performance: A VM-Centric Approach (HCI2185BU)

Optimize vSAN performance using vRealize Operations and Reinforcement Learning (HCI1650BU)

Benchmarking hyper-converged vSphere environment using HCIBench

In this article I will explain briefly about using HCIBench for storage benchmarking vSphere deployments. I recently had a chance to try HCIBench on a ScaleIO cluster running on a 3 node ESXi 6.0 cluster. Latest version of the tool can be downloaded from here.

I conducted the tests with HCIBench version 1.6.5.1.  Installation steps are very straight forward. You can install it on one of the ESXi nodes using "Deploy OVF Template" option.

*Browse and select the template.
*Provide a name and select location.
*Select a resource to host HCIBench.
*Review details and click Next.
*Accept license agreements.
*Select a datastore to store HCIBench files.
*Select networks (as shown below).

Note:
Public network is the network through which you can access management web GUI of HCIBench.
Private network is the network where HCIBench test VMs will be deployed.

*Provide management IP details (I am using static IP)  and root password for HCIBench  as shown below.

*Click Next and Finish.

*Once the deployment is complete, you can power on the HCIBench VM.

*Web GUI can be accessed by: <IP address>:8080

*Provide root credentials to access the configuration page.

In the configuration page fill the necessary details as given below.

*vCenter IP.

*vCenter username and password.

*Datacenter name.

*Cluster name.

*Network name (the network on which HCIBench test VMs will be deployed).

Note:

The network on to which HCIBench test VMs will be deployed should have a DHCP server to provide IPs to the test VMs. If you do not have a DHCP server available in that network, select "Set Static IP for Vdbench VMs" as shown below. In this case HCIBench will provide IPs to the test VMs through the private network that you have chosen in the earlier installation step. Here, I don't have a DHCP server in VMNet-SIO network. So I connected eth1 of HCIBench to VMNet-SIO (private network) and checked the option "Set Static IP for Vdbench VMs".

*Provide datastore names.

*Provide ESXi host username and password.

*Total number of VMs.

*Number of data disks per VM.

*Size of each data disk.

Note:

Make sure to keep a ratio between the total number of datastores, number of ESXi nodes and the number of test VMs. Here I have 3 ESXi nodes and 3 datastores in the cluster. So if I deploy 9 VMs for the test, each ESXi node will host 3 test VMs, one on each datastore. 

*Give a name for the test.

*Select the test parameters.

*You can use the Generate button to customize the parameters (shown in next screenshot).

*Input parameters as per your requirement and submit.

*If you are doing this the very first time, you will find the below option at the end of the page to upload Vdbench ZIP file. You can upload the file if you have one, or you can download it from the Oracle website using the Download button.

*After providing necessary inputs, you can save the configuration and validate. If the validation passes, you can click on test to begin the benchmarking test.

*Once the test is complete, click on the Result button and it will take you to the directory where result files are stored. 

Hope the article was useful to you. Happy benchmarking. Cheers!