Monday, April 30, 2018

Infrastructure testing using Pester - Part 3

In this article, I will explain briefly about how to use Pester to validate your switching infrastructure/ switch configurations. If your switches have incorrect configurations, you will experience several problems like network disconnections, high latency, low throughput, etc. And all these will contribute towards network performance issues. In a hyper-converged infrastructure, incorrect switch configurations will affect both compute and storage performance. So it is very important to make sure your switches are configured in the right way according to best practice recommendations.

Using Pester tests, you can define the expected configuration rules and execute it against your existing switches to verify everything is configured correctly or not.

Here in this example, I will show how to verify the below.
  • Networking OS version (here I am using Dell EMC S5048F-ON switch)
  • The interfaces are Up (given a range)
  • A given set of VLANs are present and Up

Prerequisite PowerShell modules:
  • Pester - Version 4.3.1
  • Posh-SSH - Version 2.0.2

Note: I am using Powershell 5.1.14393.0

#Collect input 
#Provide interface range to verify status
[int]$Start_port = Read-host "Enter starting switch port number"
[int]$End_port = Read-host "Enter ending switch port number"
#Provide VLANs to verify status
[int[]]$vlans = 20,23
$check_vlans = @{}

#New SSH session to the Switch 
$sw_creds = Get-Credential -Message "Enter switch creds"

Write-Host "Creating new SSH session to Switch."
$SWssh = New-SSHSession -ComputerName 192.168.10.4 -Credential $sw_creds -Force -ConnectionTimeout 300
Write-Host "Collecting configuration details from Switch. This will take few seconds."
Start-Sleep -s 3

Write-Host "Collecting VLAN details from Switch. This will take few seconds."
for ($j=0; $j -lt $vlans.Count; $j++) {
    Write-Host "Collecting details of VLAN $($vlans[$j])"
    $cmd_vlan = "show interfaces vlan $($vlans[$j])"
    $check_vlan = invoke-sshcommand -Command $cmd_vlan -SSHSession $SWssh
    Start-Sleep -s 3
    $check_vlans[$j] = $check_vlan.Output
}

#Collecting networking OS info 
$Networking_OS = Invoke-SSHCommand -SSHSession $SWssh -Command "show system"
Start-Sleep -s 3

#Collecting interface status details
$interface_cmd =  "show interfaces twentyFiveGigE 1/$Start_port-1/$End_port"
$interface_status = Invoke-SSHCommand -SSHSession $SWssh -Command $interface_cmd

Write-Host "Configuration verification started.`n"

Describe "System basic checks" {
    Context "Check networking OS version" {
        It "Should be Dell EMC Networking OS Version : 9.12(1.0)" {
            ($Networking_OS.Output) -match 'Dell EMC Networking OS Version : 9\.12\(1\.0\)\s\s$' | Should be $true
        }
    }
}

$Global:i=1

Describe "Interface checks" {
    for ($i=$Start_port; $i -le $End_port; $i++{
        Context "Interface should be UP" {
            It "Interface 1/$i should be UP" {
                 $Global:c1 = "twentyFiveGigE 1/$i is up, line protocol is up"
                 $res = ($interface_status.Output) -match $c1
                 $res | should be $true
            }
        }
    }
}

Describe "VLAN checks" {
    for ($j=0; $j -lt $vlans.Count; $j++) {
        Context "Check VLAN $($vlans[$j])" {
             It "Should contain VLAN $($vlans[$j])" {
                  $check = ($check_vlans[$j]) -match '% Error: No such interface'
                  $check | should be $null
                  Write-host $check
             }
             It "VLAN $($vlans[$j]) should be UP" {
                  $tt = "Vlan $($vlans[$j]) is up, line protocol is up"
                  $t3 = ($check_vlans[$j]) -match $tt
                  $t3 | should be $true
             }
        }
    }
}

Remove-SSHSession -SSHSession $SWssh

#Sample output

You can write custom Pester tests according to your switching infrastructure configuration, where you can verify port channels, VLAN membership of switch ports, VLT configuration etc. Hope this was helpful. Cheers!

Tuesday, March 20, 2018

Working with ScaleIO REST API using PowerShell - Part 3

As you all know disk drives are the most important and basic building blocks of a storage system. In order to ensure data availability, the disk drives in your storage system should always stay healthy. Unhealthy drives not only cause risk to your data but also contribute to degraded performance. In this article, I will show how to query and get disk health-related information in a ScaleIO cluster using REST API and PowerShell. 

Please refer my previous posts for API authentication, token generation and 'Invoke-RestMethod'.


#Query all devices in the ScaleIO cluster
$all_devices = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/types/Device/instances " -Method Get -Headers $ScaleIOAuthHeaders)  

#Select necessary properties and sort
$all_devices = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/types/Device/instances " -Method Get -Headers $ScaleIOAuthHeaders) | select sdsId, storagePoolId, name, deviceCurrentPathName, errorState, deviceState, ssdEndOfLifeState, temperatureState, aggregatedState  | sort sdsId | ft -AutoSize


#Sample output

Hope this was helpful. Please refer to ScaleIO 2.5 API reference guide for more details. Cheers!

Sunday, March 11, 2018

PowerShell quick start guide

This post is for all those who would like to kickstart and learn PowerShell from the very basic level. When I started learning PowerShell I wrote few articles so that it will be helpful for folks who are looking forward to 'how to start and learn it' and I can also refer to it whenever I need. Here in this post, I am just putting all of them together in proper order so that anyone can easily make use of it.

PowerShell 101 blog series

Friday, March 9, 2018

Infrastructure testing using Pester - Part 2

In my previous post I explained briefly about the basics of using Pester to test and validate your infrastructure. This article is about how to create/ output custom messages on Pester test failures. Let's have a look at the below example.

#example_test.ps1
Describe "Verify drive D" {
    Context "Check file system type and AUS" {
        It "Test 1: Should be NTFS" {
            $drive_stat = fsutil fsinfo statistics D:
            ($drive_stat[0]) -match 'File System Type :\s+NTFS' | Should Be $true
        }
        It "Test 2: Should have 4K AUS" {
            $AUS_stat = fsutil fsinfo refsinfo D:
            ($AUS_stat[8]) -match 'Bytes Per Cluster :\s+4096' | Should Be $true
        }
    }
}

Describe "Verify Hyper-V vSwitch" {
    Context "Check for Corp vSwitch, its type and connected NIC" {
       
        $check = Get-VMSwitch | where name -eq Corp

        It "Test 3: Corp vSwitch should be present" {
            ($check.Name) | Should -BeExactly "Corp"
        }
        It "Test 4: Corp vSwitch type should be Internal" {
            ($check.SwitchType) | Should -BeExactly "Internal"  
        }
    }
}


I get the following output, when I run it against one of my test machine.


The above screenshot shows, Test 1 and Test 4 is failed. Eventhough you get the error message, some users/ admins prefer to have a custom message when a test fails. So that it will be very easy to troubleshoot for the one who is validating the infrastructure. To save custom messages for each test, I am using a hash table, and if a test is found to be failed, it will pick the respective custom message from the hash table and displays it and the end of the test. Here I created a new ps1 file as below.

#invoke_test.ps1
$test = Invoke-Pester .\example_test.ps1 -PassThru

$failmsg_table = @{
    "Test 1: Should be NTFS"                        = "Test 1: Failed because the filesystem of drive D is not NTFS"
    "Test 2: Should have 4K AUS"                    = "Test 2: Failed because the AUS of drive D is not 4K"
    "Test 3: Corp vSwitch should be present"        = "Test 3: Failed becuase Corp vSwitch is not present"
    "Test 4: Corp vSwitch type should be Internal"  = "Test 4: Failed because Corp vSwitch type is not Internal"
}

for ($i=0; $i -le $test.TotalCount; $i++){
    if ($test.TestResult[$i].Result -eq "Failed") {
        write-output $failmsg_table[$($test.TestResult[$i].Name)]
    }
}



When I execute "invoke_test.ps1", it invokes "example_test.ps1" and the output is saved to variable "test". The hash table has the list of tests and the corresponding custom message. If any of the "TestResult" is Failed, the corresponding custom message will be displayed. Sample output screenshot is given below.


 Hope this was useful. Cheers!

Wednesday, March 7, 2018

Working with ScaleIO REST API using PowerShell - Part 2

In my previous post, I explained how to connect and authenticate with ScaleIO REST API, and use Invoke-RestMethod. In this article, I will explain how to query selected statistics/ properties with POST method. 

First, you have to define the set of properties that you would like to query, and then pass it to "invoke-restmethod". You can get the list of available properties of the respective object from the ScaleIO REST API Reference Guide. 

Example 1: Query and collect MDM cluster information
Here I am collecting the below-mentioned properties of "MDMCluster" object. 

#selected properties to query

$param1 = @'
{
    "properties":["name", "clusterMode", "master", "clusterState", "virtualIps"]
}
'@

#invoke-restmethod with respective uri

$MDM_cluster_stats = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/instances/System/queryMdmCluster " -Body $param1 -ContentType "application/json" -Headers $ScaleIOAuthHeaders -Method Post)

#organize output in a hash table

$mdm_props = @{

    'Cluster name'      = ($MDM_cluster_stats).name
    'Mode'              = ($MDM_cluster_stats).clusterMode
    'Cluster state'     = ($MDM_cluster_stats).clusterState
    'Master MDM IP'     = ($MDM_cluster_stats.master).managementIPs[0]
    'Cluster VIP01'     = ($MDM_cluster_stats).virtualIps[0]
    'Cluster VIP02'     = ($MDM_cluster_stats).virtualIps[1]
}

Write-Output $mdm_props

#sample output screenshot given below


Example 2: Query and collect overall ScaleIO system capacity details
Here I am collecting some properties of "System Statistics" object. 

#selected properties to query

$param2 = @'
{
    "properties":["maxCapacityInKb", "capacityInUseInKb", "spareCapacityInKb", "failedCapacityInKb", "degradedFailedCapacityInKb"]
}
'@

#invoke-restmethod with respective uri

$system_overall_stats = (Invoke-RestMethod -uri "https://192.168.11.15:443/api/types/System/instances/action/querySelectedStatistics " -Body $param2 -ContentType "application/json" -Headers $ScaleIOAuthHeaders -Method Post)

#organize output in a hash table


$system_capacity_props = @{

    "System max capacity (TB)"                = (($system_overall_stats.maxCapacityInKb)/1024/1024/1024)
    "System capacity in use (TB)"             = (($system_overall_stats.capacityInUseInKb)/1024/1024/1024)
    "System spare capacity (TB)"              = (($system_overall_stats.spareCapacityInKb)/1024/1024/1024)
    "System failed capacity (TB)"             = (($system_overall_stats.failedCapacityInKb)/1024/1024/1024)
    "System degraded failed capacity (TB)"    = (($system_overall_stats.degradedFailedCapacityInKb)/1024/1024/1024)
}

Write-Output $system_capacity_props

#sample output screenshot given below


Hope this was useful. Please refer Dell EMC ScaleIO Ready Node Version 2.5 API Reference Guide for more details.