VMware PowerCLI 101 - part9 - Working with NSX-T

Note I am using the following versions:

PSVersion: 7.1.3
VMware PowerCLI: 12.3.0.17860403

Connect-NsxtServer -Server 192.168.41.8

Get-Module "VMware.VimAutomation.Nsx*" -ListAvailable
Get-Command -Module "VMware.VimAutomation.Nsxt"

Get-NsxtService | measure

Get-NsxtService | more

Get-NsxtService com.vmware.nsx.cluster

$t1 = Get-NsxtService com.vmware.nsx.cluster
$t1 | Get-Member
$t1.get()

$t1 = Get-NsxtService com.vmware.nsx.cluster.status

$t1.get()

$t1.get().mgmt_cluster_status

$t1.get().control_cluster_status

$t1 = Get-NsxtService com.vmware.nsx.capacity.usage

$t1.get().capacity_usage | select usage_type, display_name, current_usage_count, max_supported_count, current_usage_percentage,severity | ft

$t1 = Get-NsxtService com.vmware.nsx.alarms

$t1.list().results | select feature_name, event_type, summary, severity, status | ft

Hope it was useful. Cheers!

References

https://blogs.vmware.com/PowerCLI/2017/11/getting-started-nsx-t-powercli.html

Docker 101 - Part5 - Push container image to docker hub repository

vSphere with Tanzu using NSX-T - Part9 - Monitoring

In the previous posts we discussed the following:

Part1 - Prerequisites

Part2 - Configure NSX

Part3 - Edge Cluster

Part4 - Tier-0 Gateway and BGP peering

Part5 - Tier-1 Gateway and Segments

Part6 - Create tags, storage policy, and content library

Part7 - Enable workload management

Part8 - Create namespace and deploy Tanzu Kubernetes Cluster

In this article, I will explain some of the popular tools used for monitoring Kubernetes clusters that provides insight into different objects in K8s, status, metrics, logs, and so on.

• Lens
• Octant
• Prometheus and Grafana
• vROps and Kubernetes Management Pack
• Kubebox

-Lens-

Download the Lens binary file from: https://k8slens.dev/

I am installing it on a Windows server. Once the installation is complete, the first thing you have to do is to provide the Kube config file details so that Lens can connect to the Kubernetes cluster and start monitoring it.

Add Cluster

Click File - Add Cluster

You can either browse and select the Kube config file or you can paste the content of your Kube config file as text. I am just pasting it as text.

 

Once you have pasted your Kube config file contents, make sure to select the context, and then click Add cluster.

Deploy Prometheus stack

If you aren't seeing CPU and memory metrics, you will need to install the Prometheus stack on your K8s cluster. And Lens has a feature that deploys the Prometheus stack on your K8s cluster with the click of a button!

Select the cluster icon and click Settings.

Scroll all the way to the end, and under Features, you will find an Install button. In my case, I've already installed it, that's why it's showing the Uninstall button.

Once you click the Install button, Lens will go ahead and install the Prometheus stack on the selected K8s cluster. After few minutes, you should be able to see all the metrics.

You can see a namespace called "lens-metrics" and under that, the Prometheus stack components are deployed.

Following are the service objects that are created as part of the Prometheus stack deployment.

And, here is the PVC that is attached to the Prometheus pod.

Terminal access

Click on Terminal to get access directly to the K8s cluster.

Pod metrics, SSH to the pod, and container logs

Scaling

 

Note: In a production environment, it is always a best practice to apply configuration changes to your K8s cluster objects through a version control system.

You can also see the Service Accounts, Roles, Role Bindings, and PSPs under the Access Control tab. For more details see https://docs.k8slens.dev/main/.

-Octant-

https://vineethac.blogspot.com/2020/08/visualize-your-kubernetes-clusters-and.html

-Prometheus and Grafana-

https://vineethac.blogspot.com/2020/09/monitoring-tanzu-kubernetes-cluster.html

-vROps and Kubernetes Management Pack-

https://blogs.vmware.com/management/2020/12/announcing-the-vrealize-operations-management-pack-for-kubernetes-1-5-1.html

https://rudimartinsen.com/2021/03/07/vrops-kubernetes-mgmt-pack/

https://www.brockpeterson.com/post/vrops-management-pack-for-kubernetes

-Kubebox-

curl -Lo kubebox https://github.com/astefanutti/kubebox/releases/download/v0.9.0/kubebox-linux && chmod +x kubebox

Select namespace

Select Pod

This will show the selected pod metrics and logs.

Note: Kubebox relies on cAdvisor to retrieve the resource usage metrics. It’s recommended to use the provided cadvisor.yaml file, that’s tested to work with Kubebox. 

kubectl apply -f https://raw.github.com/astefanutti/kubebox/master/cadvisor.yaml

Kubebox: https://github.com/astefanutti/kubebox

Hope it was useful. Cheers!

Validate your Kubernetes cluster using Sonobuoy

Sonobuoy is a diagnostic tool that helps to validate the state of a Kubernetes cluster by running a set of tests in an accessible and non-destructive manner. By default, Sonobuoy runs the Kubernetes conformance tests. The conformance testing ensures that a cluster is properly configured and that its behavior conforms to official Kubernetes specifications. It also helps ensure that a Kubernetes cluster meets the minimal set of features. They are a subset of end-to-end (e2e) tests that should pass on any Kubernetes cluster. 

A conformance-passing cluster provides the guarantee that your Kubernetes is properly configured as per best practices. There are around 275 tests that need to be passed for qualifying Kubernetes conformance.

Install Sonobuoy

wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.51.0/sonobuoy_0.51.0_linux_amd64.tar.gz
tar -xvf sonobuoy_0.51.0_linux_amd64.tar.gz

Note: I am installing Sonobuoy on CentOS Linux release 7.9.2009 (Core).

Help

/root/sonobuoy --help

Run Sonobuoy
/root/sonobuoy run --wait

Note: e2e test takes around 60-90 minutes to complete.

Sonobuoy Objects

kubectl get all -n sonobuoy

kubectl get pods -n sonobuoy -o wide

Sonobuoy Status

/root/sonobuoy status

/root/sonobuoy status --json
/root/sonobuoy status --json | jq

Note: If you are getting this while using jq "bash: jq: command not found..." , follow this blog to install jq.

Inspect Logs
/root/sonobuoy logs

Sonobuoy Results

results=$(/root/sonobuoy retrieve)

/root/sonobuoy results $results
/root/sonobuoy results <tar ball file>

See passed/ failed tests

/root/sonobuoy results <tar ball file> --mode=detailed | jq 'select(.status=="passed")' /root/sonobuoy results <tar ball file> --mode=detailed | jq 'select(.status=="failed")'

List the conformance tests
/root/sonobuoy results <tar ball file> --mode=detailed| jq 'select(.name | contains("[Conformance]"))'

Cleanup
/root/sonobuoy delete --wait

References

https://github.com/vmware-tanzu/sonobuoy
https://sonobuoy.io/docs/v0.51.0/

Index

Generative AI and LLMs

• Ollama
  
  
• Part1 - Deploy Ollama on Kubernetes
• Part2 - Prompt LLMs using Ollama, LangChain, and Python
• Part3 - Web UI for Ollama to interact with LLMs
• Part4 - Vision assistant using LLaVA
  
  
• Hugging Face
  
  
• Part1 - Getting started with Hugging Face
• Part2 - Code generation with Code Llama Instruct
• Part3 - Inference with Code Llama using LangChain

Kubernetes

• Kubernetes mini project
• Storage performance benchmarking of Tanzu Kubernetes Clusters
• Benchmarking Kubernetes infrastructure using K-Bench
• Validate your Kubernetes cluster using Sonobuoy
• vSphere with Tanzu using NSX-T Blog Series
  
  
• Part1 - Prerequisites
• Part2 - Configure NSX
• Part3 - Edge Cluster
• Part4 - Tier-0 Gateway and BGP peering
• Part5 - Tier-1 Gateway and Segments
• Part6 - Create tags, storage policy, and content library
• Part7 - Enable workload management
• Part8 - Create namespace and deploy Tanzu Kubernetes Cluster
• Part9 - Monitoring
• Part10 - Upgrade Tanzu Kubernetes Cluster
• Part11 - Troubleshooting Tanzu Kubernetes Cluster 
• Part12 - Deploy application on TKC and access it
• Part13 - Export WCP admin kubeconfig 
• Part14 - Testing TKC storage using kubestr
• Part15 - Working with etcd on TKC with one control plane
• Part16 - Troubleshooting content library related issues
• Part17 - Troubleshooting TKC stuck at updating phase
• Part18 - Troubleshooting vSphere pods with ProviderFailed status
• Part19 - Troubleshooting TKC stuck at creating phase  
• Part20 - Safely deleting NotReady nodes from a TKC
• Part21 - Pointers while upgrading the stack
• Part22 - Working with NGINX Ingress Controller
• Part23 - Supervisor cluster certificates expiry
  
• Part24 - Kubernetes component certs in TKC
• Part25 - Spherelet
• Part26 - Jumpbox kubectl plugin to SSH to TKC node
• Part27 - nullfinalizer kubectl plugin
• Part28 - Create a custom VM Class
• Part29 - Logging using Loki stack
  
  
• Tanzu Kubernetes Grid (TKG) on vSphere 6.7 U3

• Part1 - Install
• Part2 - Deploy, and manage multiple Kubernetes workload clusters
• Part3 - Deploy FIO application with persistent storage
  
  
• Kubernetes 101 Blog Series

• Part1 - Create K8s cluster with kubeadm
• Part2 - Basic operations
• Part3 - Install kubectl on Windows
• Part4 - Kubectl autocomplete and alias
  

• Visualize your Kubernetes clusters and workloads using Octant

vRealize Operations (vROps)

• vRealize Operations 7.5 Blog Series
  
  
• Part1 - Installation
• Part2 - Configure vCenter adapter and dashboards overview
• Part3 - Rightsizing
• Part4 - High availability
• Part5 - Alerting
• Part6 - Adding new symptoms and alert definitions
• Part7 - vSAN monitoring and troubleshooting
• Part8 - Super Metrics
• Part9 - Dashboard sharing
• Part10 - Adding Remote Collector Node
  
  
• Dell EMC PowerFlex Management Pack for vROps 8.x Blog Series
  
  
• Part1: Install
• Part2: Configure
• Part3: Dashboards
• Part4: Resource kinds and relationships
• Part5: Collection interval
• Part6: Create custom alerts
• Part7: Create custom reports

PowerShell

• PowerShell 101
  
  
• Part 1: Microsoft PowerShell Help System
• Part 2: Objects, properties, and methods in PowerShell
• Part 3: PowerShell Pipeline and object filtering
• Part 4: Avoid disasters in PowerShell
• Part 5: PowerShell Remoting
  
  
• VMware PowerCLI Blog Series
  
  
• Part1 - Installing the module and working with stand-alone ESXi host
• Part2 - Working with vCenter server
• Part3 - Basic VM operations
• Part4 - Snapshots
• Part5 - Real time storage IOPS and latency
• Part6 - vSphere networking
• Part7 - Working with vROps
• Part8 - Working with vSAN
• Part9 - Working with NSX-T
  
  
• Working with iDRAC9 using PowerShell
  
  
• Part 1
• Part 2
• Part 3
• Part 4
  
  
• Working with ScaleIO/ VxFlex OS/ PowerFlex REST API using PowerShell
  
  
• Part 1
• Part 2
• Part 3
• Part 4

vSphere with Tanzu using NSX-T - Part8 - Create namespace and deploy Tanzu Kubernetes Cluster

In the previous posts we discussed the following:

vSphere with Tanzu using NSX-T - Part1 - Prerequisites

vSphere with Tanzu using NSX-T - Part2 - Configure NSX

vSphere with Tanzu using NSX-T - Part3 - Edge Cluster

vSphere with Tanzu using NSX-T - Part4 - Tier-0 Gateway and BGP peering

vSphere with Tanzu using NSX-T - Part5 - Tier-1 Gateway and Segments

vSphere with Tanzu using NSX-T - Part6 - Create tags, storage policy, and content library

vSphere with Tanzu using NSX-T - Part7 - Enable workload management

Now that we have enabled workload management, the next step is to create namespaces on the supervisor cluster, set resource quotas as per requirements, and then the vSphere administrator can provide access to developers to these namespaces, and they can either deploy Tanzu Kubernetes clusters or VMs or vSphere pods. 

• Create namespace.

• Select the cluster and provide a name for the namespace.

• Now the namespace is created successfully. Before handing over this namespace to the developer, you can set permissions, assign storage policies, and set resource limits.

Let's have a look at the NSX-T components that are instantiated when we created a new namespace.

• A new segment is now created for the newly created namespace. This segment is connected to the T1 Gateway of the supervisor cluster.

• A SNAT rule is also now in place on the supervisor cluster T1 Gateway. This helps the Kubernetes objects residing in the namespace to reach the external network/ internet. It uses the egress range 192.168.72.0/24 that we provided during the workload management configuration for address translation.

We can now assign a storage policy to this newly created namespace.

• Click on Add Storage and select the storage policy. In my case, I am using Tanzu Storage Policy which uses a vsanDatastore.

Let's apply some capacity and usage limits for this namespace. Click edit limits and provide the values.

Let's set user permissions to this newly created namespace. Click add permissions.

Now we are ready to hand over this new namespace to the dev user (John).

Under the first tile, you can see copy link, you can provide this link to the dev user. And he can open it in a web browser to access the CLI tools to connect to the newly created namespace.

Download and install the CLI tools. In my case, CLI tools are installed on a CentOS 7.x VM. You can also see the user John has connected to the newly created namespace using the CLI.

The user can now verify the resource limits of the namespace using kubectl.

You can see the following limits:

• cpu-limit: 21.818
• memory-limit: 131072Mi
• storage: 500Gi

Storage is limited at 500 GB and memory at 128 GB which is very straightforward. We (vSphere admin) had set the CPU limits to 48 GHz. And here what you see is cpu-limit of this namespace is limited to 21.818 CPU cores. Just to give some more background on this calculation, the ESXi host that I am using for this study has 20 physical cores, and the total CPU capacity of a host is 44 GHz. I have 4 such ESXi hosts in the cluster. Now, the computing power of one physical core is (44/ 20) = 2.2 GHz. So, in order to limit the CPU to 48 GHz, the number of cpu core should be limited to (48/ 2.2) = 21.818.  

Apply the following cluster definition yaml file to create a Tanzu Kubernetes cluster under the ns-01-dev-john namespace.

apiVersion: run.tanzu.vmware.com/v1alpha1

kind: TanzuKubernetesCluster

metadata:

 name: tkg-cluster-01

 namespace: ns-01-dev-john

spec:

 topology:

   controlPlane:

     count: 3

     class: guaranteed-medium

     storageClass: tanzu-storage-policy

   workers:

     count: 3

     class: guaranteed-xlarge

     storageClass: tanzu-storage-policy

 distribution:

   version: v1.18.15

 settings:

  network:

   services:

    cidrBlocks: ["198.32.1.0/12"]

   pods:

    cidrBlocks: ["192.1.1.0/16"]

   cni:

    name: calico

  storage:

   defaultClass: tanzu-storage-policy

Login to the Tanzu Kubernetes cluster directly using CLI and verify.

You can see corresponding VMs in the Center UI.

Now, let's have a look at the NSX-T side.

• A Tier-1 Gateway is now available with a segment linked to it.

• You can see a server load balancer with one virtual server that provides access to KubeAPI (6443) of the Tanzu Kubernetes cluster that we just deployed.

• You can also find a SNAT rule. This helps the Tanzu Kubernetes cluster objects to reach the external network/ internet. It uses the egress range 192.168.72.0/24 that we provided during the workload management configuration for address translation.

Note: This architecture is explained on the basis of vSphere 7 U1. In the newer versions there are changes. With vSphere 7 U1c the architecture changed from a per-TKG cluster Tier 1 Gateway model to a per-Supervisor namespace Tier 1 Gateway model. For more details, feel free to refer the blog series published by Harikrishnan T @hari5611.

In the next part we will discuss monitoring aspects of vSphere with Tanzu environment and Tanzu Kubernetes clusters. I hope this was useful. Cheers!