Validate your Kubernetes cluster using Sonobuoy

Sonobuoy is a diagnostic tool that helps to validate the state of a Kubernetes cluster by running a set of tests in an accessible and non-destructive manner. By default, Sonobuoy runs the Kubernetes conformance tests. The conformance testing ensures that a cluster is properly configured and that its behavior conforms to official Kubernetes specifications. It also helps ensure that a Kubernetes cluster meets the minimal set of features. They are a subset of end-to-end (e2e) tests that should pass on any Kubernetes cluster. 

A conformance-passing cluster provides the guarantee that your Kubernetes is properly configured as per best practices. There are around 275 tests that need to be passed for qualifying Kubernetes conformance.

Install Sonobuoy

wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.51.0/sonobuoy_0.51.0_linux_amd64.tar.gz
tar -xvf sonobuoy_0.51.0_linux_amd64.tar.gz

Note: I am installing Sonobuoy on CentOS Linux release 7.9.2009 (Core).

Help

/root/sonobuoy --help

Run Sonobuoy
/root/sonobuoy run --wait

Note: e2e test takes around 60-90 minutes to complete.

Sonobuoy Objects

kubectl get all -n sonobuoy

kubectl get pods -n sonobuoy -o wide

Sonobuoy Status

/root/sonobuoy status

/root/sonobuoy status --json
/root/sonobuoy status --json | jq

Note: If you are getting this while using jq "bash: jq: command not found..." , follow this blog to install jq.

Inspect Logs
/root/sonobuoy logs

Sonobuoy Results

results=$(/root/sonobuoy retrieve)

/root/sonobuoy results $results
/root/sonobuoy results <tar ball file>

See passed/ failed tests

/root/sonobuoy results <tar ball file> --mode=detailed | jq 'select(.status=="passed")' /root/sonobuoy results <tar ball file> --mode=detailed | jq 'select(.status=="failed")'

List the conformance tests
/root/sonobuoy results <tar ball file> --mode=detailed| jq 'select(.name | contains("[Conformance]"))'

Cleanup
/root/sonobuoy delete --wait

References

https://github.com/vmware-tanzu/sonobuoy
https://sonobuoy.io/docs/v0.51.0/

Infrastructure testing using Pester - Part 3

In this article, I will explain briefly about how to use Pester to validate your switching infrastructure/ switch configurations. If your switches have incorrect configurations, you will experience several problems like network disconnections, high latency, low throughput, etc. And all these will contribute towards network performance issues. In a hyper-converged infrastructure, incorrect switch configurations will affect both compute and storage performance. So it is very important to make sure your switches are configured in the right way according to best practice recommendations.

Using Pester tests, you can define the expected configuration rules and execute it against your existing switches to verify everything is configured correctly or not.

Here in this example, I will show how to verify the below.

• Networking OS version (here I am using Dell EMC S5048F-ON switch)
• The interfaces are Up (given a range)
• A given set of VLANs are present and Up

Prerequisite PowerShell modules:

• Pester - Version 4.3.1
• Posh-SSH - Version 2.0.2

Note: I am using Powershell 5.1.14393.0

#Collect input 

#Provide interface range to verify status

[int]$Start_port = Read-host "Enter starting switch port number"

[int]$End_port = Read-host "Enter ending switch port number"

#Provide VLANs to verify status

[int[]]$vlans = 20,23

$check_vlans = @{}

#New SSH session to the Switch 

$sw_creds = Get-Credential -Message "Enter switch creds"

Write-Host "Creating new SSH session to Switch."

$SWssh = New-SSHSession -ComputerName 192.168.10.4 -Credential $sw_creds -Force -ConnectionTimeout 300

Write-Host "Collecting configuration details from Switch. This will take few seconds."

Start-Sleep -s 3

Write-Host "Collecting VLAN details from Switch. This will take few seconds."

for ($j=0; $j -lt $vlans.Count; $j++) {

    Write-Host "Collecting details of VLAN $($vlans[$j])"

    $cmd_vlan = "show interfaces vlan $($vlans[$j])"

    $check_vlan = invoke-sshcommand -Command $cmd_vlan -SSHSession $SWssh

    Start-Sleep -s 3

    $check_vlans[$j] = $check_vlan.Output

}

#Collecting networking OS info 

$Networking_OS = Invoke-SSHCommand -SSHSession $SWssh -Command "show system"

Start-Sleep -s 3

#Collecting interface status details

$interface_cmd =  "show interfaces twentyFiveGigE 1/$Start_port-1/$End_port"

$interface_status = Invoke-SSHCommand -SSHSession $SWssh -Command $interface_cmd

Write-Host "Configuration verification started.`n"

Describe "System basic checks" {

    Context "Check networking OS version" {

        It "Should be Dell EMC Networking OS Version : 9.12(1.0)" {

            ($Networking_OS.Output) -match 'Dell EMC Networking OS Version : 9\.12\(1\.0\)\s\s$' | Should be $true

        }

    }

}

$Global:i=1

Describe "Interface checks" {

    for ($i=$Start_port; $i -le $End_port; $i++) {

        Context "Interface should be UP" {

            It "Interface 1/$i should be UP" {

                 $Global:c1 = "twentyFiveGigE 1/$i is up, line protocol is up"

                 $res = ($interface_status.Output) -match $c1

                 $res | should be $true

            }

        }

    }

}

Describe "VLAN checks" {

    for ($j=0; $j -lt $vlans.Count; $j++) {

        Context "Check VLAN $($vlans[$j])" {

             It "Should contain VLAN $($vlans[$j])" {

                  $check = ($check_vlans[$j]) -match '% Error: No such interface'

                  $check | should be $null

                  Write-host $check

             }

             It "VLAN $($vlans[$j]) should be UP" {

                  $tt = "Vlan $($vlans[$j]) is up, line protocol is up"

                  $t3 = ($check_vlans[$j]) -match $tt

                  $t3 | should be $true

             }

        }

    }

}

Remove-SSHSession -SSHSession $SWssh

#Sample output

You can write custom Pester tests according to your switching infrastructure configuration, where you can verify port channels, VLAN membership of switch ports, VLT configuration etc. Hope this was helpful. Cheers!

Infrastructure testing using Pester - Part 2

In my previous post I explained briefly about the basics of using Pester to test and validate your infrastructure. This article is about how to create/ output custom messages on Pester test failures. Let's have a look at the below example.

#example_test.ps1

Describe "Verify drive D" {

    Context "Check file system type and AUS" {

        It "Test 1: Should be NTFS" {

            $drive_stat = fsutil fsinfo statistics D:

            ($drive_stat[0]) -match 'File System Type :\s+NTFS' | Should Be $true

        }

        It "Test 2: Should have 4K AUS" {

            $AUS_stat = fsutil fsinfo refsinfo D:

            ($AUS_stat[8]) -match 'Bytes Per Cluster :\s+4096' | Should Be $true

        }

    }

}

Describe "Verify Hyper-V vSwitch" {

    Context "Check for Corp vSwitch, its type and connected NIC" {

       

        $check = Get-VMSwitch | where name -eq Corp

        It "Test 3: Corp vSwitch should be present" {

            ($check.Name) | Should -BeExactly "Corp"

        }

        It "Test 4: Corp vSwitch type should be Internal" {

            ($check.SwitchType) | Should -BeExactly "Internal"  

        }

    }

}

I get the following output, when I run it against one of my test machine.

The above screenshot shows, Test 1 and Test 4 is failed. Eventhough you get the error message, some users/ admins prefer to have a custom message when a test fails. So that it will be very easy to troubleshoot for the one who is validating the infrastructure. To save custom messages for each test, I am using a hash table, and if a test is found to be failed, it will pick the respective custom message from the hash table and displays it and the end of the test. Here I created a new ps1 file as below.

#invoke_test.ps1

$test = Invoke-Pester .\example_test.ps1 -PassThru

$failmsg_table = @{

    "Test 1: Should be NTFS"                        = "Test 1: Failed because the filesystem of drive D is not NTFS"

    "Test 2: Should have 4K AUS"                    = "Test 2: Failed because the AUS of drive D is not 4K"

    "Test 3: Corp vSwitch should be present"        = "Test 3: Failed becuase Corp vSwitch is not present"

    "Test 4: Corp vSwitch type should be Internal"  = "Test 4: Failed because Corp vSwitch type is not Internal"

}

for ($i=0; $i -le $test.TotalCount; $i++){

    if ($test.TestResult[$i].Result -eq "Failed") {

        write-output $failmsg_table[$($test.TestResult[$i].Name)]

    }

}

When I execute "invoke_test.ps1", it invokes "example_test.ps1" and the output is saved to variable "test". The hash table has the list of tests and the corresponding custom message. If any of the "TestResult" is Failed, the corresponding custom message will be displayed. Sample output screenshot is given below.

 Hope this was useful. Cheers!