Anatomy of Hyper-V cluster debug log

• Get-ClusterLog dumps the events to a text file
• Location: C:\Windows\Clsuter\Reports\Cluster.log
• It captures last 72 hours log
• Cluster log is in GMT (because of geographically spanned multi-site clusters)
• Usage: Get-ClusterLog -timespan (which gives last "x" minutes logs)
• You can also set the levels of logs
• Set-ClusterLog -Level 3 (level 3 is default)
• It can be from level 0 to level 5 (increasing level of logging has performance impact)
• Level 5 will provide the highest level of detail
• Log format:
  [ProcessID] [ThreadID] [Date/Time] [INFO/WARN/ERR/DBG] [RescouceType] [ResourceName] [Description]

Troubleshooting Live Migration issues on Hyper-V

1. Check whether enough resources (CPU, RAM) are available at the destination host
2. Make sure all nodes in the cluster follow same naming standard for vSwitches
3. Check NUMA spanning is enabled or not. If NUMA spanning is disabled, VM must fit entirely within a single physical NUMA node or the VM will not start or be restored or migrated
4. Constrained delegation should be configured for all servers in the cluster if you are using Kerberos authentication protocol for live migration
5. Check live migration setting is enabled on Hyper-V settings
6. Verify Hyper-V-High-Availability logs in event viewer
7. Finally check cluster debug log (Get-Clusterlog -timespan) in C:\Windows\Cluster\Reports\Cluster.log 

How Live Migration works on Hyper-V

1.Live migration setup

• Source host  creates TCP connection with destination host
• VM configuration data is transferred to destination host
• A skeleton VM is setup at destination host
• Physical memory is allocated to that VM

2.Memory pages are transferred from the source to destination host

• In-state memory (working set) of the VM will be transferred first
• Default page size is 4 KB
• All utilized pages will be copied to destination
• Modified pages are tracked by source and marked as being modified
• Several iteration of copy process will take place

3.Remaining modified pages will be transferred to destination host

• VM is then registered and the device state is transferred
• Less modified pages implies fast migration
• Total working set is copied to destination

4.Move storage handle from source to destination

• Till this step VM at destination host is not online

5.Once control of storage is transferred, VM will be online and resumed at destination

• Now the VM is completely migrated and running on destination host

6.Network cleanup

• Message is sent to physical network switch causes it to relearn MAC address of migrated VM 

Nutanix Certifications

For all those who are enthusiastic to learn the Nutanix Administration course and to be a certified NPP (Nutanix Platform Professional), I strongly recommend to visit their education portal http://nuschool.nutanix.com and enroll yourself. I've completed the course and cleared NPP Certification Exam 4.5 last Monday. Its a decent self paced course which may take around 8-10 hours.

The exam has 50 objective questions and you need a minimum of 80% to pass. If you are not successful at the first attempt, don't worry, you have two more chances. In my first attempt I was able to score 3523/ 5000 only. And on my second attempt I scored 4500/ 5000. 

Once you pass the NPP, you can apply for the next level of certification which is NSS (Nutanix Support Specialist) and then to the ultimate level NPX (Nutanix Platform Expert).

Reference : http://www.nutanix.com/services/education/certifications/

Hyper-V on Nutanix

This article explains briefly about Hyper-V on Nutanix virtual computing platform. The below figure shows a 'N' node Nutanix architecture, where each node is an independent server unit with a hypervisor, processor, memory and local storage (combination of SSD and HDD). Along with this there is a CVM (controller VM) through which storage resources are accessed.

'N' node Hyper-V over Nutanix architecture

Local storage from all the nodes are combined together to form a virtualized and unified storage pool by the Nutanix Distributed File System (NDFS). This can be considered as an advanced NAS which delivers a unified storage pool across the cluster and having features like striping, replication, error detection, failover, automatic recovery etc. From this pool shared storage resources are presented to the hypervisors using containers.



NDFS - Logical view of storage pool and containers

As mentioned above, each Hyper-V node has its own CVM which is shown below.

Hyper-V node with a CVM

PRISM console

View of storage pool in PRISM

View of containers in PRISM

Containers are mounted to Hyper-V as SMB 3.0 based file shares where virtual machine files are stored. This is shown below.

SMB 3.0 share path to store VHDs and VM configuration files

Once share path is given properly as mentioned above, you can create virtual machines on your Hyper-V server.

Zoning and LUN masking

Zoning and LUN masking are used to isolate SAN traffic and to restrict access to storage devices. For example you might manage different zones separately for testing and production environment, so that they will not interfere. If you want to restrict certain hosts from accessing the storage devices then you have to setup zoning. This is generally done at FC switch level. Zoning are of two types : soft zoning and hard zoning.

Soft zoning is based on WWN name of the device and hard zoning is configured at FC switch port level. Soft zoning offers a greater range of flexibility. That means even if you move a device from one port to another on the FC switch, it will have the same access rights as the restriction is based on WWN name of the device. But the down side of this is using WWN spoofing you can gain access to zones that you aren't supposed to see. In case of hard zoning at switch port level, you will get a tighter access control but with less flexibility compared to soft zoning. Now if you change the device from one port to another as we done before, it won't be able to see its partner. In this case you can't spoof a physical port unless you are standing in the same room at the switch.

Once zoning is done you can further restrict access to SAN LUNs by using LUN masking. This will prevent certain devices from seeing specific LUNs hosted in the storage device. LUN masking is done at storage controller level or OS level of the storage device. It is recommended to use zoning and LUN masking together for securing storage traffic.

Installing Nutanix CE on nested ESXI 5.5 without using SSD

This is a great way to test/ learn Nutanix using the community edition (CE) which is totally free with Nutanix software based controller VM (CVM) and Acropolis hypervisor. This article will help you install Nutanix CE on a nested ESXI 5.5 without using SSD drives.

-Download links :

Version 2015.06.08: http://download.nutanix.com/ce/2015.06.08-beta/ce-2015.06.08-beta.img.gz

Version 2015.07.16: http://download.nutanix.com/ce/2015.07.16/ce-2015.07.16-beta.img.gz

-Extract the downloaded *.gz file and you will get a *.img file. Rename that file to ce-flat.vmdk.


Files

-Now create a descriptor file as given below :

# Disk DescriptorFile

version=4

encoding="UTF-8"

CID=a63adc2a

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 14540800 VMFS "ce-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "905"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "2e046b033cecaa929776efb0a63adc2a"

ddb.uuid = "60 00 C2 9b 69 2f c9 76-74 c4 07 9e 10 87 3b f9"

ddb.virtualHWVersion = "10" 

-Copy the above content, paste it in a text file and save it as ce.vmdk.

-Create a VM on ESXI 5.5


VM properties

-RAM (min) 16 GB
-CPU (min) 4 (here I used 2 virtual sockets * 6 cores per socket)
-Network adapter E1000
-VM hardware version 8
-Guest OS selected as Linux CentOS 4/5/6 (64-bit)
-Initially the VM is created without a hard disk
-Enable CPU/ MMU virtualization

CPU/ MMU Virtualization

-Upload ce.vmdk and ce-flat.vmdk to the datastore

Datastore





-Add hard disk (ce.vmdk) to the VM

 
-Add 2 more hard disks to the VM of size 500 GB each. So now we have total 3 hard disks.

Hard disk 1 - ce.vmdk - SCSI (0:0)

Hard disk 2 - 500 GB - SCSI (0:1) (this drive will be emulated as SSD drive)

Hard disk 3 - 500 GB - SCSI (0:2)

-Download *.vmx file from the datastore and add line vhv.enable = "TRUE" to it and upload it back to the datastore. This is to enable nested hypervisor installation over ESXI 5.5.

-To emulate SSD, edit VM settings - options - advanced - general - configuration parameters - add row and add name scsi0:1.virtualSSD with value 1

-Power up the VM and Nutanix will start the installation wizard

-Login as root and password : nutanix/4u

-vi /home/install/phx_iso/phoenix/sysUtil.py

-Click insert and then edit SSD_rdIOPS_thresh = 50 and SSD_wrIOPS_thresh = 50

-And save it (press Esc button then :wq! to save the file) and reboot

-Login in with 'install' and hit enter (no password)

-Click proceed

-The installer will now verify all prerequisites and then you can enter the following details

-Host IP, subnet mask, gateway

-CVM IP, subnet mask, gateway

-Select create single-node cluster and enter DNS IP (8.8.8.8)

-Scroll down through the license agreement and select accept and click proceed

-Set promiscuous  mode in vSwitch to Accept

-Once the installation is complete, you can access the PRISM dashboard with the CVM IP address and NEXT account credentials



PRISM dashboard