Understanding NUMA: Its Impact on VM Performance in ESXi

VMware ESXi hosts use Non-Uniform Memory Access (NUMA) architecture to optimize CPU and memory locality. Each NUMA node consists of a subset of CPUs and memory. Accessing local memory within the same NUMA node is significantly faster than remote memory access. Misaligned NUMA configurations can lead to latency spikes, increased CPU Ready Time, and degraded VM performance.

Key symptoms

The common symptoms for Virtual Machines (VMs) on ESXi that have a misconfigured or misaligned Non-Uniform Memory Access (NUMA) configuration primarily manifest as performance degradation and latency. The main issue caused by NUMA misalignment is that the VM's vCPUs end up frequently having to access memory that belongs to a different physical NUMA node on the ESXi host (known as Remote Access), which is significantly slower than accessing local memory.

The resulting symptoms for the VM include:

• Overall Slowness and Unresponsiveness: Services and applications running inside the guest OS may respond slowly or intermittently. The entire VM can feel sluggish.

• High CPU Ready Time (%RDY): This is the most critical ESXi-level metric. CPU Ready Time represents the percentage of time a VM was ready to run but could not be scheduled on a physical CPU. High %RDY times (often above 5% or 10%) can indicate that the VM's vCPUs are struggling to get scheduled efficiently, which happens when they are spread across multiple NUMA nodes (NUMA spanning).

• Excessive Remote Memory Access: When a VM consumes more vCPUs or memory than is available on a single physical NUMA node, a portion of its memory traffic becomes "remote." You can check this using the esxtop utility on the ESXi host.

Common misconfigurations

Misalignment often occurs when the VM's vCPU and memory settings exceed the resources of a single physical NUMA node on the host. Common causes include:

• Over-Sized VM: Allocating more vCPUs than the physical cores available in a single physical NUMA node or allocating more memory than the physical memory on a single NUMA node.

• Hot-Add Features: Enabling CPU Hot-Add or Memory Hot-Add can disable vNUMA (Virtual NUMA) for the VM, preventing the VMkernel from presenting an optimized NUMA topology to the guest OS.

• Incorrect Cores per Socket Setting: While vSphere 6.5 and later are smarter about vNUMA, configuring the Cores per Socket value manually in a way that doesn't align with the host's physical NUMA topology can still lead to poor scheduling and memory placement, particularly when licensing dictates a low number of virtual sockets.

• Setting VM Limits: Setting a memory limit on a VM that is lower than its configured memory can force the VMkernel to allocate the remaining memory from a remote NUMA node.

Check NUMA assignments in ESXi

• SSH into the ESXi node.
• Issue the esxtop command and press m for memory view, then press f to enable the fields, G to enable NUMA information.
• You should be able to view the NUMA related information like NRMEM, NLMEM, and N%L.
• NRMEM (MB): NUMA Remote MEMory
• This is the current amount of a VM's memory (in MB) that is physically located on a remote NUMA node relative to where the VM's vCPUs are currently running.
• High NRMEM indicates NUMA locality issues, meaning the vCPUs must cross the high-speed interconnect (like Intel's QPI/UPI or AMD's Infinity Fabric) to access some of their data, which results in slower performance.
• NLMEM (MB): NUMA Local MEMory
• This is the current amount of a VM's memory (in MB) that is physically located on the local NUMA node, meaning it's on the same physical node as the vCPUs accessing it.
• The ESXi NUMA scheduler's goal is to maximize NLMEM to ensure fast memory access.
• N%L: NUMA % Locality
• This is the percentage of the VM's total memory that resides on the local NUMA node.
• A value close to 100% is ideal, indicating excellent memory locality. If this value drops below 80%, the VM may experience poor NUMA locality and potential performance issues due to slower remote memory access.
• Issue the esxtop command and press v to see the virtual machine screen.
• From the virtual machine screen note down the GID of the VM under consideration, and press q to exit the screen.
• Now issue the sched-stats -t numa-clients command. This will list down NUMA details of the VM. Check the groupID column to match the GID of the VM.
• For example, the GID of the VM I am looking at is 7886858. This is a 112 CPU VM which is running on an 8-socket physical host.

• You can see the VM is spread/ placed under NUMA nodes 0, 1, 2, and 3.
• The remoteMem is 0, for each of these NUMA nodes, which means they are accessing all the local memory of the NUMA node.
• To view physical NUMA details of the ESXI you can use sched-stats -t numa-pnode command. You can see this server has 8 NUMA nodes.

• To view the NUMA latency, you can use the sched-stats -t numa-latency command.

Verify NUMA node details at guest OS

Windows

• Easiest way is to go to Task Manager - Performance - CPU
• Right click on the CPU utilization graph and select Change graph to - NUMA nodes
• If there only one NUMA node, you may notice the option as greyed out.
• To get detailed info you can consider using the sysinternals utility coreinfo64.

Linux

• To view NUMA related details from the Linux guest OS layer, you can use the following commands:

lscpu | grep -i NUMA

dmesg | grep -i NUMA

Remediation

The most common remediation steps for fixing Non-Uniform Memory Access (NUMA) related performance issues in ESXi VMs revolve around right-sizing the VM to align its resources with the physical NUMA boundaries of the host.

The primary goal is to minimize Remote Memory Access (NRMEM) and maximize Local Memory Access (N%L). The vast majority of NUMA issues stem from a VM's resource allocation crossing a physical NUMA node boundary.

• Right-Size VMs: Keep vCPU count within physical cores of a single NUMA node.
• Evenly Divide Resources: For monster/ wide VMs, ensure the total vCPUs are configured such that they are evenly divisible by the number of physical NUMA nodes they span.
• Example: If a VM needs 16 vCPUs on a host with 12-core NUMA nodes, configure the vCPUs to be a multiple of a NUMA node count (e.g., 2 sockets $\times$ 8 cores per socket to create 2 vNUMA nodes, aligning with 2 pNUMA nodes).
• Cores per Socket Setting (Important for older vSphere/Licensing): While vSphere 6.5 and later automatically present an optimal vNUMA topology, you should still configure the Cores per Socket setting on the VM to create a vNUMA structure that aligns with the physical NUMA boundaries of the host. This helps the guest OS make better scheduling decisions.
• Disable VM CPU/ Memory Hot-Add: Plan capacity upfront.

NUMA awareness is critical for troubleshooting and optimizing VM performance on ESXi. Misconfigured NUMA placements can severely impact latency-sensitive workloads like databases and analytics. Regular checks at both the hypervisor and guest OS layers ensure memory locality, reduce latency, and improve efficiency.

References

• 64 Cores per NUMA Node Limit in Microsoft SQL Server: Recommendation for Efficiently Allocating Logical CPUs to SQL Server VMs on VMware vSphere - VMware Cloud Foundation (VCF) Blog 
• Architecting Microsoft SQL Server on VMware vSphere | VMware 
• Virtual Machine vCPU and vNUMA Rightsizing - Guidelines - VROOM! Performance Blog 
• Setting corespersocket can affect guest OS topologies 
• Performance Best Practices for VMware vSphere 7.0, Update 3 
• Home - Flings (broadcom.com)  (Virtual Machine Compute Optimizer)
• vSphere 7 Cores per Socket and Virtual NUMA - frankdenneman.nl 
• How many NUMA Nodes do I have? – SQLpassion 
• Coreinfo - Sysinternals | Microsoft Learn

Hope it was useful. Cheers!

Troubleshooting ESXi PSOD: A Quick Guide for SREs

When an ESXi host hits a Purple Screen of Death (PSOD), it’s more than just a crash - it’s a signal that something critical needs attention. Here’s how to handle it effectively.

What happens during a PSOD?

• The ESXi server displays a purple diagnostic screen.
• You’ll see alerts/ incidents for host connectivity, FC/ Ethernet link down, and related alarms.
• The console screen confirms the purple screen.

Immediate actions

• Capture screenshots of the PSOD from the console screen.
• Check server hardware health via the out-of-band management interface like iDRAC/ RMC/ BMC.
• Observe if the host is stuck or rebooting repeatedly.
• If ESXi reloads successfully, immediately place the node in Maintenance Mode via vCenter.
• If it keeps crashing, try to capture all PSOD instances.

Collecting logs

• Generate a support bundle from vCenter once the host is online.
• Collect server hardware logs.

Engage support

• Broadcom/ VMware: Share PSOD screenshots and ESXi support bundle for RCA.
• Hardware vendor: Attach server hardware logs, screenshots, and context for analysis.

Analyze crash dumps

• Look for these keywords in the core dump logs: BlueScreen, Backtrace, Exception
• In the ESXi support bundle you will find the crash dump logs under /var/core directory.
• Analyzing the core dump files should help you find the root cause of the PSOD event. It could be due to some hardware issue, bugs in ESXi hypervisor, faults in device firmware or drivers, etc.
• You may notice many vmkernel-zdump files, and to quickly filter out all the BlueScreen events, you can use the following PowerShell code snippet.

------------------------------------------------------------------

param(
    [string]$directoryPath,
    [string[]]$keywords
)

# Function to search for keywords in files
function Search-Files {
    param (
        [string]$path,
        [string[]]$keywords
    )

    # Get all files in the directory and subdirectories
    $files = Get-ChildItem -Path $path -Recurse -File

    # Loop through each file
    foreach ($file in $files) {
        # Read the content of the file
        $content = Get-Content -Path $file.FullName

        # Loop through each line in the file content
        foreach ($line in $content) {
            # Check if the line contains any of the keywords (case-insensitive)
            foreach ($keyword in $keywords) {
                if ($line -match "(?i)$keyword") {
                    # Print the file name and the matching line
                    Write-Output "File: $($file.FullName)"
                    Write-Output "Line: $line"
                    Add-Content -path out.txt -value $line
                    break
                }
            }
        }
    }
}

# Call the function with the provided parameters
Search-Files -path $directoryPath -keywords $keywords

------------------------------------------------------------------

• Save the above code snippet to a .ps1 file (example: find.ps1) and you can run it as follows:

> .\find.ps1 -directoryPath "C:\esx-esxi1.xre.com-2025-04-18--13.05-2106842\var\core" -keywords "bluescreen"
> .\find.ps1 -directoryPath "C:\esx-esxi1.xre.com-2025-04-18--13.05-2106842\var\core" -keywords "bluescreen", "#PF Exception"

• All the log lines that include the given keyword or keywords will be saved to out.txt file.
• A sample output of the above-mentioned code snippet against an ESXi core dump is given below.

• Once you identify the root cause of the PSOD event, you can start working towards the resolution which may involve replacing a faulty hardware component, updating firmware/ driver/ ESXi, etc.

References

• Interpreting an ESXi host purple diagnostic screen 
• Extracting the log file after an ESXi host fails with a purple screen error

Hope it was useful. Cheers!

VMware PowerCLI 101 - part12 - Get uptime and boot time of ESXi nodes and VMs

You can use the following PowerCLI commandlets to get the uptime and boot time of ESXi hosts and VMs.

• Get ESXi boot time.

> Get-Cluster | Get-VMHost | Select Name, @{Name="Boot time";Expression={$_.ExtensionData.Runtime.BootTime}}

Name                                  Boot time
----                                  ---------
esxi1.wxxxxx.com                      2/27/2025 10:13:45 AM
esxi2.wxxxxx.com                      6/18/2025 2:39:38 PM

• Get ESXi uptime.

> Get-Cluster | Get-VMHost | Select Name, @{Name="Uptime (Days)";Expression={(Get-Date) - $_.ExtensionData.Runtime.BootTime | Select-Object -ExpandProperty Days}}

Name                                  Uptime (Days)
----                                  -------------
esxi1.wxxxxx.com                      116
esxi2.wxxxxx.com                      5

• Get VM boot time.

> Get-Cluster | Get-VM | Select Name, @{Name="Boot time";Expression={$_.ExtensionData.Runtime.BootTime}}

Name                                      Boot time
----                                      ---------
test-vineeth
rhel84-vineeth
VMware vCenter Server                     2/25/2025 11:28:13 AM
vCLS-50a1b507-5a1d-5e00-8410-7599fc72e5b5 6/18/2025 2:51:59  PM

• Get VM uptime.

> Get-Cluster | Get-VM | Select Name, @{Name="Uptime (Days)";Expression={(Get-Date) - $_.ExtensionData.Runtime.BootTime | Select-Object -ExpandProperty Days}}

Name                                      Uptime (Days)
----                                      -------------
test-vineeth
rhel84-vineeth
VMware vCenter Server                     118
vCLS-50a1b507-5a1d-5e00-8410-7599fc72e5b5 5

Hope it was useful. Cheers!

VMware PowerCLI 101 - part11 - Get vCLS cluster state

vSphere Cluster Services (vCLS) is a feature introduced in vSphere 7.0 Update 1 to ensure the availability of cluster services like DRS and HA, even if the vCenter Server becomes unavailable. It provides a distributed control plane for clustering services, improving scalability and resilience.

> Get-Cluster | select Name, @{N="VcsHealthStatus";E={$PSItem.ExtensionData.SummaryEx.VcsHealthStatus}}  
  
Name                    VcsHealthStatus  
----                    ---------------  
Cluster-01              healthy  
Cluster-02              healthy

Hope it was useful. Cheers!

VMware PowerCLI 101 - part10 - Verify vSphere cluster state

To verify the most common selected status attributes of the cluster:

Get-Cluster | Get-VMHost | Select Name,@{N='HAState';E={$_.ExtensionData.Runtime.DasHostState.State}},ConnectionState,PowerState,@{N='OverallStatus';E={$_.ExtensionData.OverallStatus}},@{N='ConfigStatus';E={$_.ExtensionData.ConfigStatus}},@{N='InMaintenanceMode';E={$_.ExtensionData.Runtime.InMaintenanceMode}},@{N='RebootRequired';E={$_.ExtensionData.Summary.RebootRequired}},@{N='BootTime';E={$_.ExtensionData.Runtime.BootTime}} | ft

> Get-Cluster | Get-VMHost | Select Name,@{N='HAState';E={$_.ExtensionData.Runtime.DasHostState.State}},ConnectionState,PowerState,@{N='OverallStatus';E={$_.ExtensionData.OverallStatus}},@{N='ConfigStatus';E={$_.ExtensionData.ConfigStatus}},@{N='InMaintenanceMode';E={$_.ExtensionData.Runtime.InMaintenanceMode}},@{N='RebootRequired';E={$_.ExtensionData.Summary.RebootRequired}},@{N='BootTime';E={$_.ExtensionData.Runtime.BootTime}} | ft

Name      HAState           ConnectionState PowerState OverallStatus ConfigStatus InMaintenanceMode RebootRequired BootTime
----      -------           --------------- ---------- ------------- ------------ ----------------- -------------- --------
10.90.1.4 connectedToMaster       Connected  PoweredOn        yellow       yellow             False          False 8/27/2024 7:31:10 AM
10.90.1.5 master                  Connected  PoweredOn        yellow       yellow             False          False 9/6/2024 9:08:09 PM

Hope it was useful. Cheers!

A decade of tech - My professional journey so far

Laying the Groundwork

My professional career commenced in February 2014, as a Trainee IT Services Engineer at Alamy Images. During my initial days, I was tasked with daily maintenance activities such as running tape backups, setting up Active Directory user accounts, mailboxes, and desktops for new employees. I also handled general IT support, troubleshooting various user issues within the organization.

After a few months, I had the opportunity to set up a lab infrastructure project using old decommissioned servers as part of a continuous learning initiative. This hands-on experience involved racking, stacking, and cabling physical servers, installing and configuring ESXi and Hyper-V hypervisors, FreeNAS storage servers, and deploying highly available clusters. Additionally, I gained exposure to configuring L2 network switches. This project significantly contributed to building my IT infrastructure foundation.

A year later, I was promoted to Junior IT Services Engineer, where I focused on virtualization projects. I spearheaded the migration of over 20 Dev/ Test/ UAT virtual machines from VMware to a Hyper-V cluster, enhancing system flexibility and cost-efficiency. I deployed a high-availability Hyper-V failover cluster in production and contributed to the planning and execution of a iSCSI storage server migration project.

Beyond virtualization, I worked on network infrastructure by a seamless L2 switch replacement and upgrade project with minimal operational disruption. Furthermore, I assisted in capacity planning initiatives for optimized resource utilization for both physical and virtual environments. These experiences refined my technical skills and problem-solving abilities. During this time, I developed a passion for infrastructure management and optimization, shaping my future career path.

From Junior IT Services Engineer to Storage Solutions Engineer

In January 2017, I transitioned to a Systems Development Engineer role at Dell EMC, specializing in Solutions Engineering. This marked a significant career shift as I immersed myself in the world of storage and virtualization solutions integration/development.

My daily responsibilities encompassed the installation and testing of various components, progressing from integration to validating system reliability and performance at scale. I designed and deployed multiple PowerFlex software-defined storage clusters for customer demos and proof-of-concepts, showcasing the product's performance and auto rebuild capabilities. A notable achievement was automating the storage performance benchmarking using PowerShell, FIO, and ELK stack, reducing process time from weeks to days.

I led the engineering efforts for developing a vROps management pack for PowerFlex, ensuring seamless integration and visibility. Additionally, I mastered vSphere Virtual Volumes (vVols), successfully executing integration projects between Dell storage solutions and VMware environments.

To streamline operations, I created a PowerShell module for managing PowerFlex using REST APIs and developed Ansible playbook for automated deployment of Kubernetes cluster with PowerFlex CSI driver. My expertise extended beyond systems engineering and automation as I authored and published whitepapers on disaster recovery using VMware SRM and hardware lifecycle management with Dell OME.

This period solidified my reputation as a virtualization and storage solutions expert, providing me with a deep understanding of storage architecture, performance optimization, and automation. I developed a passion for building scalable and reliable hyperconverged solutions.

From Storage Solutions Engineer to Site Reliability Engineer

In July 2021, I transitioned to a Site Reliability Engineer (SRE) role at VMware, focusing on ensuring the reliability and scalability of Kubernetes-as-a-Service project based on the vSphere with Tanzu platform.

Managing a vast infrastructure of Kubernetes clusters, I honed my skills in incident response, GitOps pipelines, automation, and monitoring. I played a crucial role in maintaining platform availability, collaborating closely with multiple internal teams and stakeholders to resolve issues and enhance service delivery. My proficiency in Python and PowerShell was instrumental in automating tasks and building custom monitoring solutions. During this time, I prepared diligently, practiced extensively, and successfully qualified for the CKA exam.

Beyond core SRE responsibilities, I explored emerging technologies. I successfully deployed and evaluated open-source language models on Kubernetes using Python, Ollama, and LangChain. In addition, I contributed to developing custom metrics for the Kubernetes-as-a-Service platform using Python, Prometheus, Grafana, and Helm.

This role deepened my expertise and ability to bridge the gap between development and operations, fostering a culture of reliability and efficiency. It has been an exciting journey of learning and growth, positioning me as a versatile IT professional with a strong foundation in both infrastructure and cloud-native technologies.

Gratitude

"This journey has been immensely fulfilling, made possible by the support and encouragement of exceptional organisations, inspiring managers, talented colleagues, friends, and family. I am truly grateful for the opportunities to learn, grow, and contribute meaningfully to driving success and making a positive impact."

The journey continues...

vSphere with Tanzu using NSX-T - Part35 - Monitoring supervisor cluster health with Python and vCenter APIs

vSphere with Tanzu Supervisor cluster is a Kubernetes platform that simplifies the deployment, management, and scaling of Kubernetes clusters. Monitoring the health of your WCP/ Supervisor clusters is crucial to ensure the smooth running of your Tanzu Kubernetes Clusters (TKCs) and applications. In this blog post, we'll explore how to use Python and vCenter APIs to verify the health of your Supervisor clusters.

You can access the Python script from my GitHub repository: https://github.com/vineethac/VMware/tree/main/vSphere_with_Tanzu/wcp_cluster_health

This script connects to the vCenter server, retrieves the cluster summary, and checks the Tanzu Supervisor cluster configuration info and prints the status of the cluster. By using this Python script, you can easily monitor the health of your Tanzu Supervisor clusters through vCenter APIs.

Hope it was useful. Cheers!

vSphere with Tanzu using NSX-T - Part33 - Troubleshooting intermittent connection timeouts to apiserver and workloads

In the realm of managing Tanzu Kubernetes clusters (TKCs), we have encountered several challenges that hindered the smooth functioning of our applications. In this blog post, we will discuss three such cases and the workarounds we employed to resolve them.

Case 1: TKC Control Plane Node Connectivity Issues

Symptoms:

• TKC apiserver connection timeouts when attempting to connect using the kubeconfig.
• Traffic was not flowing to two of the control plane nodes.
• NSX-T web UI LB VS stats indicated this issue.

Case 2: TKC Worker Node Connectivity Issues

Symptoms:

• Workload (example: PostgreSQL cluster) connection timeouts.
• Traffic was not flowing to two of the worker nodes in the TKC.
• NSX-T web UI LB VS stats indicated this issue.

Case 3: Load Balancer Connectivity Issues

Symptoms:

• Connection timeouts when attempting to connect to a PostgreSQL workload through the load balancer VS IP.
• This issue was observed only when creating new services of type LoadBalancer in the TKC.
• We noticed datapath mempool usage for the edge nodes was above the threshold value.

Resolution/ work around

• Find the T1 router that is attached to the TKC which has connectivity issues. 
• In an Active - Standby HA configuration, you will see that there will be one Edge node that will be Active and another one in Standby status. 
• First place the Standby Edge node in NSX MM, reboot it, and then exit it from NSX MM. 
• Now, place the Active Edge node in NSX MM, there will be a slight network disruption during this failover, once it is in NSX MM, reboot it, and then exit NSX MM. 
• This should resolved the issue.

In conclusion, these cases illustrate the importance of verifying NSX-T components when managing Tanzu Kubernetes clusters. By identifying the root cause of the issues and employing effective workarounds, we were able to restore functionality and maintain the health of our applications. Stay tuned for more insights and best practices in managing Kubernetes clusters.

Hope it was useful. Cheers!

vSphere with Tanzu using NSX-T - Part32 - Troubleshooting BGP related issues

This article provides basic guidance on troubleshooting BGP related issues.

Sample diagram showing connectivity between Edge Nodes and TOR switches

Verify Tier-0 Gateway status on NSX-T

• Status of T0 should be Success.

• Check the interfaces of T0 to identify which all edge nodes are part of it.

• Check the status of Edge Transport Nodes.

• As you can see from the T0 interfaces, Edge01/02/03/04 are part of it and in those edge nodes you should be able to see the SR_TIER0 component. Next step is to login to those Edge nodes that are part of T0 and verify BGP summary.
  
  

Verify BGP on all Edge nodes that are part of T0 Gateway  

• SSH into the edge node as admin user.

• get logical-router

• Look for SERVICE_ROUTER_TIER0.

sc2-01-nsxt04-r08edge02> get logical-router
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                        Ports   Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      4       22/5000
e6d02207-c51e-4cf8-81a6-44afec5ad277   2      84653  DR-t1-domain-c1034:1de3adfa-0ee   DISTRIBUTED_ROUTER_TIER1    5       9/50000
a590f1da-2d79-4749-8153-7b174d23b069   32     85271  DR-t1-domain-c1034:1de3adfa-0ee   DISTRIBUTED_ROUTER_TIER1    5       5/50000
758d9736-6781-4b3a-906f-3d1b03f0924d   33     88016  DR-t1-domain-c1034:1de3adfa-0ee   DISTRIBUTED_ROUTER_TIER1    4       1/50000
5e7bfe98-0b5e-4620-90b1-204634e99127   37     3      SR-sc2-01-nsxt04-tr               SERVICE_ROUTER_TIER0        6       5/50000

• vrf <SERVICE_ROUTER_TIER0 VRF>

• get bgp neighbor summary

• Note: If everything is working fine State should show Estab.

sc2-01-nsxt04-r08edge02> vrf 37
sc2-01-nsxt04-r08edge02(tier0_sr[37])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.184.248.2  Local AS: 4259971071

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.184.248.239                      4259970544  Estab 05w1d22h     NC  12641393 12610093 2      568
10.184.248.240                      4259970544  Estab 05w1d23h     NC  12640337 11580431 2      566

• You should be able to ping to the BGP neighbor IP. If you are unable to ping to neighbor IPs, then there is an issue.

sc2-01-nsxt04-r08edge02(tier0_sr[37])> ping 10.184.248.239
PING 10.184.248.239 (10.184.248.239): 56 data bytes
64 bytes from 10.184.248.239: icmp_seq=0 ttl=255 time=1.788 ms
^C
--- 10.184.248.239 ping statistics ---
2 packets transmitted, 1 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 1.788/1.788/1.788/0.000 ms

sc2-01-nsxt04-r08edge02(tier0_sr[37])> ping 10.184.248.240
PING 10.184.248.240 (10.184.248.240): 56 data bytes
64 bytes from 10.184.248.240: icmp_seq=0 ttl=255 time=1.925 ms
64 bytes from 10.184.248.240: icmp_seq=1 ttl=255 time=1.251 ms
^C
--- 10.184.248.240 ping statistics ---
3 packets transmitted, 2 packets received, 33.3% packet loss
round-trip min/avg/max/stddev = 1.251/1.588/1.925/0.337 ms

• Get interfaces | more

sc2-01-nsxt04-r08edge02> vrf 37
sc2-01-nsxt04-r08edge02(tier0_sr[37])> get interfaces | more
Fri Aug 19 2022 UTC 11:07:18.042
Logical Router
UUID                                   VRF    LR-ID  Name                              Type
5e7bfe98-0b5e-4620-90b1-204634e99127   37     3      SR-sc2-01-nsxt04-tr               SERVICE_ROUTER_TIER0
Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)
    Interface     : dd83554d-47c0-5a4e-9fbe-3abb1239a071
    Ifuid         : 335
    Mode          : cpu
    Port-type     : cpu
    Enable-mcast  : false

    Interface     : 008b2b15-17d1-4cc8-9d94-d9c4c2d0eb3a
    Ifuid         : 1000
    Name          : tr-interconnect-edge02
    Fwd-mode      : IPV4_AND_IPV6
    Internal name : uplink-1000
    Mode          : lif
    Port-type     : uplink
    IP/Mask       : 10.184.248.2/24
    MAC           : 02:00:70:51:9d:79
    VLAN          : 1611

Verify BGP on Cisco TOR switches

• SSH to TOR switch.

• show ip bgp summary

❯ ssh -o PubkeyAuthentication=no netadmin@sc2-01-r08lswa.xxxxxxxx.com
User Access Verification
(netadmin@sc2-01-r08lswa.xxxxxxxx.com) Password:

Cisco Nexus Operating System (NX-OS) Software

sc2-01-r08lswa# show ip bgp summary
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.184.17.248, local AS number 65001.65008
BGP table version is 520374, IPv4 Unicast config peers 10, capable peers 8
5150 network entries and 11372 paths using 2003240 bytes of memory
BGP attribute entries [110/18920], BGP AS path entries [69/1430]
BGP community entries [0/0], BGP clusterlist entries [0/0]
11356 received paths for inbound soft reconfiguration
11356 identical, 0 modified, 0 filtered received paths using 0 bytes

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.184.10.14    4 65011.65000
                        47979514 10570342   520374    0    0     5w1d 4541
10.184.10.78    4 65011.65000
                        47814555 10601750   520374    0    0     5w1d 4541
10.184.248.1    4 65001.65535
                          80831   79447   520374    0    0 02:41:51 566
10.184.248.2    4 65001.65535
                        3215614 3269391   520374    0    0     5w1d 566
10.184.248.3    4 65001.65535
                        3215776 3269344   520374    0    0     1w3d 566
10.184.248.4    4 65001.65535
                        3215676 3269383   520374    0    0 13:51:45 566
10.184.248.5    4 65001.65535
                        3200531 3269384   520374    0    0     5w1d 5
10.184.248.6    4 65001.65535
                        3197752 3266700   520374    0    0     5w1d 5

• show ip arp

sc2-01-r08lswa# show ip arp 10.184.248.2

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface       Flags
10.184.248.2    00:06:12  0200.7051.9d79  Vlan1611

• If you compare this IP and MAC, you can see that its the same of your T0 SR uplink of your edge02 node.

IP/Mask       : 10.184.248.2/24
MAC           : 02:00:70:51:9d:79

For further troubleshooting you can do packet capture from the edge nodes and ESXi server and analyze them using Wireshark.

Packet capture from Edge node

• Capture packets from the T0 SR uplink interface.

sc2-01-nsxt04-r08edge01(tier0_sr[5])> get interfaces | more
Wed Aug 17 2022 UTC 13:52:48.203
Logical Router
UUID                                   VRF    LR-ID  Name                              Type
fb1ad846-8757-4fdf-9cbb-5c22ba772b52   5      2      SR-sc2-01-nsxt04-tr               SERVICE_ROUTER_TIER0
Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)
    Interface     : c8b80ba1-93fc-5c82-a44f-4f4863b6413c
    Ifuid         : 286
    Mode          : cpu
    Port-type     : cpu
    Enable-mcast  : false

    Interface     : 4915d978-9c9a-58bc-84e2-cafe5442cba4
    Ifuid         : 287
    Mode          : blackhole
    Port-type     : blackhole

    Interface     : 899bcf30-83e2-46bb-9be2-8889ec52b354
    Ifuid         : 833
    Name          : tr-interconnect-edge01
    Fwd-mode      : IPV4_AND_IPV6
    Internal name : uplink-833
    Mode          : lif
    Port-type     : uplink
    IP/Mask       : 10.184.248.1/24
    MAC           : 02:00:70:d1:92:b1
    VLAN          : 1611
    Access-VLAN   : untagged
    LS port       : 15b971e9-7caa-43b7-86c1-96ff50453402
    Urpf-mode     : STRICT_MODE
    DAD-mode      : LOOSE
    RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)
    Admin         : up
    Op_state      : up
    Enable-mcast  : False
    MTU           : 9000
    arp_proxy     :

• Start a continuous ping from the TOR switches to the edge uplink IP (in this case ping 10.184.248.1 from TOR switches) before starting packet capture.

sc2-01-nsxt04-r08edge01> start capture interface 899bcf30-83e2-46bb-9be2-8889ec52b354 file uplink.pcap

Note:
Find the location of uplink.pcap file on TOR switches and SCP it locally to analyze using Wireshark.

 

Packet capture from ESXi

• In this example, we are capturing packets of sc2-01-nsxt04-r08edge01 VM from the switchports where its interfaces are connected. sc2-01-nsxt04-r08edge01 VM is running on ESXi node sc2-01-r08esx10.

[root@sc2-01-r08esx10:~] esxcli network vm list | grep edge
18790721  sc2-01-nsxt04-r08edge05                                                 3  , ,
18977245  sc2-01-nsxt04-r08edge01                                                 3  , ,

[root@sc2-01-r08esx10:/tmp] esxcli network vm port list -w 18977245
   Port ID: 67109446
   vSwitch: sc2-01-vc16-dvs
   Portgroup:
   DVPort ID: b60a80c0-ecd6-40bd-8d2b-fbd1f06bb172
   MAC Address: 02:00:70:33:a9:67
   IP Address: 0.0.0.0
   Team Uplink: vmnic1
   Uplink Port ID: 2214592517
   Active Filters:

   Port ID: 67109447
   vSwitch: sc2-01-vc16-dvs
   Portgroup:
   DVPort ID: 6e3d8057-fc23-4180-b0ba-bed90381f0bf
   MAC Address: 02:00:70:d1:92:b1
   IP Address: 0.0.0.0
   Team Uplink: vmnic1
   Uplink Port ID: 2214592517
   Active Filters:

   Port ID: 67109448
   vSwitch: sc2-01-vc16-dvs
   Portgroup:
   DVPort ID: c531df19-294d-4079-b39c-89a3b58e30ad
   MAC Address: 02:00:70:30:c7:01
   IP Address: 0.0.0.0
   Team Uplink: vmnic0
   Uplink Port ID: 2214592519
   Active Filters:

• Start a continuous ping from the TOR switches to the edge uplink IP (in this case ping 10.184.248.1 from TOR switches) before starting packet capture.

[root@sc2-01-r08esx10:/tmp] pktcap-uw --switchport 67109446 --dir 2 -o /tmp/67109446-02:00:70:33:a9:67.pcap --count 1000 & pktcap-uw --switchport 67109447 --dir 2 -o /tmp/67109447-02:00:70:d1:92:b1.pcap --count 1000 & pktcap-uw --switchport 67109448 --dir 2 -o /tmp/67109448-02:00:70:30:c7:01.pcap --count 1000

Note:
SCP the pcap files to laptop and use Wireshark to analyse them.
You can also do packet capture from physical uplinks (vmnic) of the ESXi node if required.

Hope it was useful. Cheers!

vSphere with Tanzu using NSX-T - Part30 - Troubleshooting inaccessible TKC with server pool members missing in the LB VS

Encountering issues with connectivity to your TKC apiserver/ control plane can be frustrating. One common problem we've seen is the kubeconfig failing to connect, often due to missing server pool members in the load balancer's virtual server (LB VS).

The Issue

The LB VS, which operates on port 6443, should have the control plane VMs listed as its member servers. When these members are missing, connectivity problems arise, disrupting your access to the TKC apiserver.

Troubleshooting steps

1. Access the TKC: Use the kubeconfig to access the TKC.
   

   ❯ KUBECONFIG=tkc.kubeconfig kubectl get node
   Unable to connect to the server: dial tcp 10.191.88.4:6443: i/o timeout
   ❯
   

2. Check the Load Balancer: In NSX-T, verify the status of the corresponding load balancer (LB). It may display a green status indicating success.
3. Inspect Virtual Servers: Check the virtual servers in the LB, particularly on port 6443. They might show as down.
4. Examine Server Pool Members: Look into the server pool members of the virtual server. You may find it empty.
5. SSH to Control Plane Nodes: Attempt to SSH into the TKC control plane nodes.
6. Run Diagnostic Commands: Execute diagnostic commands inside the control plane nodes to verify their status. The issue could be that the control plane VMs are in a hung state, and the container runtime is not running.
   

   vmware-system-user@tkc-infra-r68zc-jmq4j [ ~ ]$ sudo su
   root [ /home/vmware-system-user ]# crictl ps
   FATA[0002] failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded
   root [ /home/vmware-system-user ]#
   root [ /home/vmware-system-user ]# systemctl is-active containerd
   Failed to retrieve unit state: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
   root [ /home/vmware-system-user ]#
   root [ /home/vmware-system-user ]# systemctl status containerd
   WARNING: terminal is not fully functional
   -  (press RETURN)Failed to get properties: Failed to activate service 'org.freedesktop.systemd1'>
   lines 1-1/1 (END)lines 1-1/1 (END)
   

7. Check VM Console: From vCenter, check the console of the control plane VMs. You might see specific errors indicating issues.
   

   EXT4-fs (sda3): Delayed block allocation failed for inode 266704 at logical offset 10515 with max blocks 2 with error 5
   EXT4-fs (sda3): This should not happen!! Data will be lost
   EXT4-fs error (device sda3) in ext4_writepages:2905: IO failure
   EXT4-fs error (device sda3) in ext4_reserve_inode_write:5947: Journal has aborted
   EXT4-fs error (device sda3) xxxxxx-xxx-xxxx: unable to read itable block
   EXT4-fs error (device sda3) in ext4_journal_check_start:61: Detected aborted journal
   systemd[1]: Caught <BUS>, dumped core as pid 24777.
   systemd[1]: Freezing execution.
   

8. Restart Control Plane VMs: Restart the control plane VMs. Note that sometimes your admin credentials or administrator@vsphere.local credentials may not allow you to restart the TKC VMs. In such cases, decode the username and password from the relevant secret and use these credentials to connect to vCenter and restart the hung TKC VMs.
   

   ❯ kubectx wdc-01-vc17
   Switched to context "wdc-01-vc17".
   ❯
   ❯ kg secret -A | grep wcp
   kube-system                                 wcp-authproxy-client-secret                                               kubernetes.io/tls                                  3      291d
   kube-system                                 wcp-authproxy-root-ca-secret                                              kubernetes.io/tls                                  3      291d
   kube-system                                 wcp-cluster-credentials                                                   Opaque                                             2      291d
   vmware-system-nsop                          wcp-nsop-sa-vc-auth                                                       Opaque                                             2      291d
   vmware-system-nsx                           wcp-cluster-credentials                                                   Opaque                                             2      291d
   vmware-system-vmop                          wcp-vmop-sa-vc-auth                                                       Opaque                                             2      291d
   ❯
   ❯ kg secrets -n vmware-system-vmop wcp-vmop-sa-vc-auth
   NAME                  TYPE     DATA   AGE
   wcp-vmop-sa-vc-auth   Opaque   2      291d
   ❯ kg secrets -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml
   apiVersion: v1
   data:
     password: aWAmbHUwPCpKe1Uxxxxxxxxxxxx=
     username: d2NwLXZtb3AtdXNlci1kb21haW4tYzEwMDYtMxxxxxxxxxxxxxxxxxxxxxxxxQHZzcGhlcmUubG9jYWw=
   kind: Secret
   metadata:
     creationTimestamp: "2022-10-24T08:32:26Z"
     name: wcp-vmop-sa-vc-auth
     namespace: vmware-system-vmop
     resourceVersion: "336557268"
     uid: dcbdac1b-18bb-438c-ba11-76ed4d6bef63
   type: Opaque
   ❯
   
   ***Decrypt the username and password from the secret and use it to connect to the vCenter.
   ***Following is an example using PowerCLI:
   
   PS /Users/vineetha> get-vm gc-control-plane-f266h
   
   Name                 PowerState Num CPUs MemoryGB
   ----                 ---------- -------- --------
   gc-control-plane-f2… PoweredOn  2        4.000
   
   PS /Users/vineetha> get-vm gc-control-plane-f266h | Restart-VMGuest
   Restart-VMGuest: 08/04/2023 22:20:20	Restart-VMGuest		Operation "Restart VM guest" failed for VM "gc-control-plane-f266h" for the following reason: A general system error occurred: Invalid fault
   PS /Users/vineetha>
   PS /Users/vineetha> get-vm gc-control-plane-f266h | Restart-VM
   
   Confirm
   Are you sure you want to perform this action?
   Performing the operation "Restart-VM" on target "VM 'gc-control-plane-f266h'".
   [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y
   
   Name                 PowerState Num CPUs MemoryGB
   ----                 ---------- -------- --------
   gc-control-plane-f2… PoweredOn  2        4.000
   
   PS /Users/vineetha>
   

9. Verify System Pods and Connectivity: Once the control plane VMs are restarted, the system pods inside them will start, and the apiserver will become accessible using the kubeconfig. You should also see the previously missing server pool members reappear in the corresponding LB virtual server, and the virtual server on port 6443 will be up and show a success status.

Following these steps should help you resolve the connectivity issues with your TKC apiserver/control plane effectively.Ensuring that your load balancer's virtual server is correctly configured with the appropriate member servers is crucial for maintaining seamless access. This runbook aims to guide you through the process, helping you get your TKC apiserver back online swiftly.

Note: If required for critical production issues related to TKC accessibility I strongly recommend to raise a product support request.

Hope it was useful. Cheers!