Deleting inaccessible objects from vSAN cluster

Scenario:

Resolution:

• Login to vSAN VCSA as root
• rvc
• Provide a username that has administrator privileges @localhost as shown below
• Change directory to localhost

• Now browse through and set the directory to the respective cluster (in my case cluster name is "Cluster-Rack7")

• vsan.check_state -r <path to your cluster>

• You can purge vswp objects using: vsan.purge_inaccessible_vswp_objects <path to cluster>

• In this case its not a vswp object. So we have to find the owner node of the object and delete it forcefully from there.
• vsan.cmmds_find -u <UUID of object> <path to cluster> to find owner of the object

• SSH into the owner node and delete the inaccessible object file forcefully using /usr/lib/vmware/osfs/bin/objtool delete -u <UUID of inaccessible object> -f -v 10

• Perform vSAN health check to verify status

Hope it was useful. Cheers!

References:

https://www.thinkcharles.net/blog/2018/2/16/removing-inaccessible-objects-in-vsan

https://www.steffr.ch/vmware-vsan-deletepurge-inaccessible-objects/

How to manually add multiple NICs to the vROps 7.x appliance

This article explains how to add multiple network interfaces to a vROps 7.0/ 7.5 appliance. Recently we had a scenario where the vROps appliance needs access to different networks which are isolated/ not routed with the primary network management interface of vROps. In my case the vROps instance needed access to 3 different networks.

Initially while installing the vROps there will be only one interface (eth0) and its the default interface for vROps appliance.  

For configuring additional interfaces follow the steps below:

• Add a network card and connect to the respective port group by editing VM settings
• Login to vROps with root creds
• cd /etc/sysconfig/network
• Create an entry for the new interface eth1 (as it will not be present!)
• vi ifcfg-eth1
• Provide all necessary IP details and save

• Reboot the appliance
• Verify details

• Similarly, follow the above steps if you require more interfaces 

Note: I am not sure whether this is officially supported by vROps, but it works! Haven't found any VMware documentations related to this.

Inactive or missing VMware VMFS datastore

Today I came across a situation where one of the shared VMFS datastores in a 4 node ESXi 6.5 cluster was found missing/ inactive after a planned reboot. This post is about the steps I followed to resolve this issue/ re-mount the inactive shared datastore.

On a ESXi node, to list the datastores that are available to mount: esxcfg-volume –l
To mount the available datastore: esxcfg-volume –M <UUID>

Sample screenshot is given below:

Hope it was useful. Cheers!

Reference:
https://community.spiceworks.com/topic/2108624-missing-datastore-after-upgrade-from-esxi-6-0-to-6-5

Real time VMware VM resource monitoring using PowerShell

This post is about monitoring resource usage of a list of virtual machines hosted on VMware ESXi clusters using PowerCLI. Output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/vmware_vm_monitor

Sample screenshot of output:

Notes:

VMware guidance: CPU Ready time and Co-Stop values per core greater than 5% and 3% respectively could  be a performance concern.

Hope this will be useful for resource monitoring as well as right sizing of VMs. Cheers!

References:

• http://docs.hol.vmware.com/HOL-2012/HOL-INF-04_EN/HOL-INF-04-m1/lessons/Part_1_of_4__Performance_101_-_vSphere_Performance_Basics_-_CPU__C__.html
• http://virtualization.solutions/2018/02/04/performance-monitoring-cpu/

Real time VMware datastore performance monitoring using PowerShell

I had a scenario where we had to monitor the real time performance statistics of multiple shared VMFS datastores which are part of a multi-node VMware ESXi cluster. So this post is about the short script which uses VMware PowerCLI to get read/ write IOPS and latency details of these shared datastores across all nodes in the cluster. The output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/datastore_perfmon

Sample screenshot of output:

Hope it was useful. Cheers!

Reference:

https://www.rudimartinsen.com/2017/07/12/vsphere-performance-data-part-3-get-stat/

Working with iDRAC9 Redfish API using PowerShell - Part 2

In this article I will explain briefly about the JSON response from iDRAC and how you can navigate through the Redfish API tree structure to get all the required information. Now, lets have a look at the URIs. 

Query the computer system collection:

$result1 = Invoke-RestMethod -Uri "https://$($idrac_ip)/redfish/v1/Systems" -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json'

Response: 

You can see one member with URI /redfish/v1/Systems/System.Embedded.1

Below is a sample screen shot of JSON output when you try to query the above listed member system. 

You can get some of the basic information straight away from the above JSON response. And these are organized in hierarchy where you can drill down to each object and get the required details. Below diagram shows basic iDRAC Redfish API tree structure.

Example: You can get details/ health status of  storage controller as shown below.

Query:

$result2 = Invoke-RestMethod -Uri "https://$($idrac_ip)/redfish/v1/Systems/System.Embedded.1/Storage /Controllers/NonRAID.Integrated.1-1" -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json'

Response:

Hope it was useful. Cheers!

References:

Working with iDRAC9 Redfish API using PowerShell - Part 1

iDRAC9 Redfish API reference guide

github.com/dell/iDRAC-Redfish-Scripting

Working with iDRAC9 Redfish API using PowerShell - Part 1

Redfish is a industry standard protocol and specification defined by Distributed Management Task Force (DMTF) for performing systems/ IT infrastructure management actions using RESTful methodology. It is a next generation systems management interface standard which is simple, secure, scalable. Redfish uses JSON data format and transports payload over HTTPS. Initial releases of Redfish focused primarily on systems management and was targeted to be a replacement for IPMI over LAN protocol. Now the capabilities have been extended over the past few years providing a rich set of features and support for network, memory, storage devices etc. Over time the scope of Redfish is being expanded to fit more use cases as the forum is working with several partner organizations. Promoters of this standard include several companies like Broadcom, Cisco, Dell, HP, VMware, Intel, Microsoft etc.

Now, lets have a look at how to connect to iDRAC Redfish API using PowerShell. Redfish provides two authentication methods. Basic authentication and Session-based authentication. Here I will explain basic authentication using username and password for each Redfish API request to iDRAC.

#To fix the connection issues to iDRAC REST API

add-type @"

    using System.Net;

    using System.Security.Cryptography.X509Certificates;

    public class TrustAllCertsPolicy : ICertificatePolicy {

    public bool CheckValidationResult(

        ServicePoint srvPoint, X509Certificate certificate,

        WebRequest request, int certificateProblem) {

        return true;

        }

    }

"@

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 -bor [System.Net.SecurityProtocolType]::Tls11

#Get iDRAC creds

$Credentials = Get-Credential -Message "Enter iDRAC Creds"

#URI to get basic system info

$u1 = "https://192.168.10.11/redfish/v1/Systems/System.Embedded.1"

#Using Invoke-RestMethod

$result1 = Invoke-RestMethod -Uri $u1 -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json' -Headers @{"Accept"="application/json"}

Output:

Hope it was useful. Cheers!

References:
iDRAC9 Redfish API reference guide
github.com/dell/iDRAC-Redfish-Scripting