VMFS vs VVOL

Let's start with a quick comparison.

VMFS	VVOL
LUN centric approach Pre-provisioning of LUNs Use of multiple datastores for different performance capabilities Management difficulties as a single VM may span across multiple datastores	VM centric vSphere is now aware of array capabilities through VASA provider No pre-provisioning One VVOL datastore can represent the whole array Storage policies can be applied per vmdk level Some of the vSphere operations are offloaded to array using VAAI (full cloning, snapshots) VVOL snapshots are faster (different from traditional snapshots with redo logs)

Note: The explanations below regarding SAN is based on Dell EMC Unity (hybrid array) 

VMFS environment

A traditional VMFS datastore setup is given below. Say, you have two VMs. One with 3 virtual disks and the second one with 2 virtual disks. And your array has 4 storage pools with specific capabilities. Based on those capabilities the pools are classified into 4 service levels (Platinum, Gold, Silver and Bronze). In this case you need to provision 4 datastores/ LUNs from the respective pool to meet requirements of the two virtual machines.

VMFS

You can clearly see that the first VM is spanned across 3 datastores and the second VM is spanned across 2 datastores. If your environment has hundreds and thousands of VMs management becomes too complicated. In this case, as the datastores are properly named the vSphere admin can easily identify the service level/ capability of a specific datastore. But if naming convention is not followed, then vSphere admin has to contact the storage admin to know about the capabilities of that specific datastore/ LUN. Again lots of communication needed, making it a complex process!

VVOL environment

Now let’s have a look at the VVOL environment which is shown the figure below. You are having the same array, but instead of provisioning datastores/ LUNs from the respective pools, here we are creating one vvol datastore. The array has 4 storage pools, each having specific capabilities like drive type, RAID level, tiering policy, FAST Cache ON/ OFF etc. and are classified into 4 service levels as Platinum, Gold, Silver, and Bronze. So each pool has its own capability profile. You can provide any name, but here I just gave the same name as the service level of each pool.

Pool_01   -> Service level: Platinum         -> Capability profile: Platinum

Pool_02   -> Service level: Silver               -> Capability profile: Silver
Pool_03   -> Service level: Bronze            -> Capability profile: Bronze
Pool_04   -> Service level: Gold                -> Capability profile: Gold

Note: vvol datastores cannot be created without capability profile

Next steps are:

• Create vvol datastore in the SAN
• Provide a name for the vvol datastore
• Add capability profiles that need to be part of the vvol datastore
• In this case, all 4 capability profiles are part of the vvol datastore
• You can also limit the amount of space that will be used from each capability profile by the vvol datastore
• Once it’s done, vvol datastore is created in the SAN

VVOL

At this point, you can go ahead and configure host access to this vvol datastore. Now you have to let your Vsphere environment know about the capabilities of the storage array. That is done by registering the new storage provider on your vCenter server making use of VASA (VMware vStorage API for Storage Awareness) provider. Once registered the vSphere environment will communicate with the VASA provider through the array management interface (OOB network) which forms a control path. Next step is creating a vvol datastore on the ESXI host which you provided access earlier. For each vvol datastore created on the storage array, two protocol endpoints (PEs) will be automatically generated to communicate with an ESXI host forming a data path. If you create another vvol datastore on the array and provide access to same ESXI host, two more PEs will be created. PEs act like a target. And on the ESXI side, if you look at the storage devices, you can see 2 proxy LUNs which connects to the respective PEs.

Now you have to create VM storage policies based on service levels. Here you have 4 service levels, so you have to create 4 storage policies. Assign storage policy per vmdk basis as per requirements.

Eg: Storage_Policy_Gold -> VMDK 02 (second VM) -> it will be placed in Pool_04 automatically

So in this case, instead of having 4 VMFS datastores we needed only 1 VVOL datastore which has all the required capabilities. This means you don't need to provision more LUNs from the SAN. Storage management becomes easy with just one VVOL. There is more granularity with SPBM at each VMDK level. With VVOLs the SAN is aware of all the VMs and its corresponding files hosted on it. This makes space reclamation very easy and straight. The moment a VM or a VMDK is deleted, that space will be immediately made free as SAN is having the complete insight of virtual machines stored in it. Data mobility between different storage pools based on its service level becomes effortless as it is handled directly by the SAN internally based on SPBM. All together VVOL simplifies storage/ LUN management.

Hope it was useful. Cheers!

References:

Working with Cisco Nexus 9K switches using PowerShell

In this article, I will explain briefly about how to work with Cisco NX-API CLI through PowerShell. Before getting into the steps and sample PowerShell code, lets first familiarize with NX-API Developer Sandbox. You can access the sandbox by entering the management IP of your Cisco switch in a web browser. Provide user name and password. Once successfully logged in, you will get a page as shown below. The developer sandbox will convert the normal CLI commands into JSON request and will also provide result/ response in JSON format.

NX-API Developer Sandbox

In the above screenshot, I am using "show hostname" command. The message format is set to JSON and command type is set to cli_show to get a response in JSON. Once done, hit POST. You can see the corresponding request and response in JSON format.

Now, let's try to do the same using PowerShell.

#Step1:

<# 

To fix the connection issues <The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.>

#> 

add-type @"

    using System.Net;

    using System.Security.Cryptography.X509Certificates;

    public class TrustAllCertsPolicy : ICertificatePolicy {

    public bool CheckValidationResult(

        ServicePoint srvPoint, X509Certificate certificate,

        WebRequest request, int certificateProblem) {

        return true;

        }

    }

"@

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 -bor [System.Net.SecurityProtocolType]::Tls11

#Step2:

$creds = Get-Credential

#Step3:

$request = @'

{

    "ins_api": {

          "version": "1.0",

          "type": "cli_show",

          "chunk": "0",

          "sid": "1",

          "input": "show hostname",

          "output_format": "json"

    }

}

'@

#Step4:

$result = Invoke-RestMethod -Uri "http://<IP of switch>/ins" -Method post -Credential $creds -ContentType "application/json" -Body $request

Output:

Hope it was useful. Cheers!

Reference:

https://inbound.kelsercorp.com/blog/cisco-nexus-9000-automation

Deleting inaccessible objects from vSAN cluster

Scenario:

Resolution:

• Login to vSAN VCSA as root
• rvc
• Provide a username that has administrator privileges @localhost as shown below
• Change directory to localhost

• Now browse through and set the directory to the respective cluster (in my case cluster name is "Cluster-Rack7")

• vsan.check_state -r <path to your cluster>

• You can purge vswp objects using: vsan.purge_inaccessible_vswp_objects <path to cluster>

• In this case its not a vswp object. So we have to find the owner node of the object and delete it forcefully from there.
• vsan.cmmds_find -u <UUID of object> <path to cluster> to find owner of the object

• SSH into the owner node and delete the inaccessible object file forcefully using /usr/lib/vmware/osfs/bin/objtool delete -u <UUID of inaccessible object> -f -v 10

• Perform vSAN health check to verify status

Hope it was useful. Cheers!

References:

https://www.thinkcharles.net/blog/2018/2/16/removing-inaccessible-objects-in-vsan

https://www.steffr.ch/vmware-vsan-deletepurge-inaccessible-objects/

How to manually add multiple NICs to the vROps 7.x appliance

This article explains how to add multiple network interfaces to a vROps 7.0/ 7.5 appliance. Recently we had a scenario where the vROps appliance needs access to different networks which are isolated/ not routed with the primary network management interface of vROps. In my case the vROps instance needed access to 3 different networks.

Initially while installing the vROps there will be only one interface (eth0) and its the default interface for vROps appliance.  

For configuring additional interfaces follow the steps below:

• Add a network card and connect to the respective port group by editing VM settings
• Login to vROps with root creds
• cd /etc/sysconfig/network
• Create an entry for the new interface eth1 (as it will not be present!)
• vi ifcfg-eth1
• Provide all necessary IP details and save

• Reboot the appliance
• Verify details

• Similarly, follow the above steps if you require more interfaces 

Note: I am not sure whether this is officially supported by vROps, but it works! Haven't found any VMware documentations related to this.

Inactive or missing VMware VMFS datastore

Today I came across a situation where one of the shared VMFS datastores in a 4 node ESXi 6.5 cluster was found missing/ inactive after a planned reboot. This post is about the steps I followed to resolve this issue/ re-mount the inactive shared datastore.

On a ESXi node, to list the datastores that are available to mount: esxcfg-volume –l
To mount the available datastore: esxcfg-volume –M <UUID>

Sample screenshot is given below:

Hope it was useful. Cheers!

Reference:
https://community.spiceworks.com/topic/2108624-missing-datastore-after-upgrade-from-esxi-6-0-to-6-5

Real time VMware VM resource monitoring using PowerShell

This post is about monitoring resource usage of a list of virtual machines hosted on VMware ESXi clusters using PowerCLI. Output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/vmware_vm_monitor

Sample screenshot of output:

Notes:

VMware guidance: CPU Ready time and Co-Stop values per core greater than 5% and 3% respectively could  be a performance concern.

Hope this will be useful for resource monitoring as well as right sizing of VMs. Cheers!

References:

• http://docs.hol.vmware.com/HOL-2012/HOL-INF-04_EN/HOL-INF-04-m1/lessons/Part_1_of_4__Performance_101_-_vSphere_Performance_Basics_-_CPU__C__.html
• http://virtualization.solutions/2018/02/04/performance-monitoring-cpu/

Real time VMware datastore performance monitoring using PowerShell

I had a scenario where we had to monitor the real time performance statistics of multiple shared VMFS datastores which are part of a multi-node VMware ESXi cluster. So this post is about the short script which uses VMware PowerCLI to get read/ write IOPS and latency details of these shared datastores across all nodes in the cluster. The output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/datastore_perfmon

Sample screenshot of output:

Hope it was useful. Cheers!

Reference:

https://www.rudimartinsen.com/2017/07/12/vsphere-performance-data-part-3-get-stat/