Understanding NUMA: Its Impact on VM Performance in ESXi

VMware ESXi hosts use Non-Uniform Memory Access (NUMA) architecture to optimize CPU and memory locality. Each NUMA node consists of a subset of CPUs and memory. Accessing local memory within the same NUMA node is significantly faster than remote memory access. Misaligned NUMA configurations can lead to latency spikes, increased CPU Ready Time, and degraded VM performance.

Key symptoms

The common symptoms for Virtual Machines (VMs) on ESXi that have a misconfigured or misaligned Non-Uniform Memory Access (NUMA) configuration primarily manifest as performance degradation and latency. The main issue caused by NUMA misalignment is that the VM's vCPUs end up frequently having to access memory that belongs to a different physical NUMA node on the ESXi host (known as Remote Access), which is significantly slower than accessing local memory.

The resulting symptoms for the VM include:

• Overall Slowness and Unresponsiveness: Services and applications running inside the guest OS may respond slowly or intermittently. The entire VM can feel sluggish.

• High CPU Ready Time (%RDY): This is the most critical ESXi-level metric. CPU Ready Time represents the percentage of time a VM was ready to run but could not be scheduled on a physical CPU. High %RDY times (often above 5% or 10%) can indicate that the VM's vCPUs are struggling to get scheduled efficiently, which happens when they are spread across multiple NUMA nodes (NUMA spanning).

• Excessive Remote Memory Access: When a VM consumes more vCPUs or memory than is available on a single physical NUMA node, a portion of its memory traffic becomes "remote." You can check this using the esxtop utility on the ESXi host.

Common misconfigurations

Misalignment often occurs when the VM's vCPU and memory settings exceed the resources of a single physical NUMA node on the host. Common causes include:

• Over-Sized VM: Allocating more vCPUs than the physical cores available in a single physical NUMA node or allocating more memory than the physical memory on a single NUMA node.

• Hot-Add Features: Enabling CPU Hot-Add or Memory Hot-Add can disable vNUMA (Virtual NUMA) for the VM, preventing the VMkernel from presenting an optimized NUMA topology to the guest OS.

• Incorrect Cores per Socket Setting: While vSphere 6.5 and later are smarter about vNUMA, configuring the Cores per Socket value manually in a way that doesn't align with the host's physical NUMA topology can still lead to poor scheduling and memory placement, particularly when licensing dictates a low number of virtual sockets.

• Setting VM Limits: Setting a memory limit on a VM that is lower than its configured memory can force the VMkernel to allocate the remaining memory from a remote NUMA node.

Check NUMA assignments in ESXi

• SSH into the ESXi node.
• Issue the esxtop command and press m for memory view, then press f to enable the fields, G to enable NUMA information.

• You should be able to view the NUMA related information like NRMEM, NLMEM, and N%L.
• NRMEM (MB): NUMA Remote MEMory
• This is the current amount of a VM's memory (in MB) that is physically located on a remote NUMA node relative to where the VM's vCPUs are currently running.
• High NRMEM indicates NUMA locality issues, meaning the vCPUs must cross the high-speed interconnect (like Intel's QPI/UPI or AMD's Infinity Fabric) to access some of their data, which results in slower performance.
• NLMEM (MB): NUMA Local MEMory
• This is the current amount of a VM's memory (in MB) that is physically located on the local NUMA node, meaning it's on the same physical node as the vCPUs accessing it.
• The ESXi NUMA scheduler's goal is to maximize NLMEM to ensure fast memory access.
• N%L: NUMA % Locality
• This is the percentage of the VM's total memory that resides on the local NUMA node.
• A value close to 100% is ideal, indicating excellent memory locality. If this value drops below 80%, the VM may experience poor NUMA locality and potential performance issues due to slower remote memory access.
• Issue the esxtop command and press v to see the virtual machine screen.
• From the virtual machine screen note down the GID of the VM under consideration, and press q to exit the screen.
• Now issue the sched-stats -t numa-clients command. This will list down NUMA details of the VM. Check the groupID column to match the GID of the VM.
• For example, the GID of the VM I am looking at is 7886858. This is a 112 CPU VM which is running on an 8-socket physical host.

• You can see the VM is spread/ placed under NUMA nodes 0, 1, 2, and 3.
• The remoteMem is 0, for each of these NUMA nodes, which means they are accessing all the local memory of the NUMA node.
• To view physical NUMA details of the ESXI you can use sched-stats -t numa-pnode command. You can see this server has 8 NUMA nodes.

• To view the NUMA latency, you can use the sched-stats -t numa-latency command.

Verify NUMA node details at guest OS

Windows

• Easiest way is to go to Task Manager - Performance - CPU
• Right click on the CPU utilization graph and select Change graph to - NUMA nodes
• If there only one NUMA node, you may notice the option as greyed out.
• To get detailed info you can consider using the sysinternals utility coreinfo64.

Linux

• To view NUMA related details from the Linux guest OS layer, you can use the following commands:

lscpu | grep -i NUMA

dmesg | grep -i NUMA

Remediation

The most common remediation steps for fixing Non-Uniform Memory Access (NUMA) related performance issues in ESXi VMs revolve around right-sizing the VM to align its resources with the physical NUMA boundaries of the host.

The primary goal is to minimize Remote Memory Access (NRMEM) and maximize Local Memory Access (N%L). The vast majority of NUMA issues stem from a VM's resource allocation crossing a physical NUMA node boundary.

• Right-Size VMs: Keep vCPU count within physical cores of a single NUMA node.
• Evenly Divide Resources: For monster/ wide VMs, ensure the total vCPUs are configured such that they are evenly divisible by the number of physical NUMA nodes they span.
• Example: If a VM needs 16 vCPUs on a host with 12-core NUMA nodes, configure the vCPUs to be a multiple of a NUMA node count (e.g., 2 sockets $\times$ 8 cores per socket to create 2 vNUMA nodes, aligning with 2 pNUMA nodes).
• Cores per Socket Setting (Important for older vSphere/Licensing): While vSphere 6.5 and later automatically present an optimal vNUMA topology, you should still configure the Cores per Socket setting on the VM to create a vNUMA structure that aligns with the physical NUMA boundaries of the host. This helps the guest OS make better scheduling decisions.
• Disable VM CPU/ Memory Hot-Add: Plan capacity upfront.

NUMA awareness is critical for troubleshooting and optimizing VM performance on ESXi. Misconfigured NUMA placements can severely impact latency-sensitive workloads like databases and analytics. Regular checks at both the hypervisor and guest OS layers ensure memory locality, reduce latency, and improve efficiency.

References

• 64 Cores per NUMA Node Limit in Microsoft SQL Server: Recommendation for Efficiently Allocating Logical CPUs to SQL Server VMs on VMware vSphere - VMware Cloud Foundation (VCF) Blog 
• Architecting Microsoft SQL Server on VMware vSphere | VMware 
• Virtual Machine vCPU and vNUMA Rightsizing - Guidelines - VROOM! Performance Blog 
• Setting corespersocket can affect guest OS topologies 
• Performance Best Practices for VMware vSphere 7.0, Update 3 
• Home - Flings (broadcom.com)  (Virtual Machine Compute Optimizer)
• vSphere 7 Cores per Socket and Virtual NUMA - frankdenneman.nl 
• How many NUMA Nodes do I have? – SQLpassion 
• Coreinfo - Sysinternals | Microsoft Learn

Hope it was useful. Cheers!

Troubleshooting ESXi PSOD: A Quick Guide for SREs

When an ESXi host hits a Purple Screen of Death (PSOD), it’s more than just a crash - it’s a signal that something critical needs attention. Here’s how to handle it effectively.

What happens during a PSOD?

• The ESXi server displays a purple diagnostic screen.
• You’ll see alerts/ incidents for host connectivity, FC/ Ethernet link down, and related alarms.
• The console screen confirms the purple screen.

Immediate actions

• Capture screenshots of the PSOD from the console screen.
• Check server hardware health via the out-of-band management interface like iDRAC/ RMC/ BMC.
• Observe if the host is stuck or rebooting repeatedly.
• If ESXi reloads successfully, immediately place the node in Maintenance Mode via vCenter.
• If it keeps crashing, try to capture all PSOD instances.

Collecting logs

• Generate a support bundle from vCenter once the host is online.
• Collect server hardware logs.

Engage support

• Broadcom/ VMware: Share PSOD screenshots and ESXi support bundle for RCA.
• Hardware vendor: Attach server hardware logs, screenshots, and context for analysis.

Analyze crash dumps

• Look for these keywords in the core dump logs: BlueScreen, Backtrace, Exception
• In the ESXi support bundle you will find the crash dump logs under /var/core directory.
• Analyzing the core dump files should help you find the root cause of the PSOD event. It could be due to some hardware issue, bugs in ESXi hypervisor, faults in device firmware or drivers, etc.
• You may notice many vmkernel-zdump files, and to quickly filter out all the BlueScreen events, you can use the following PowerShell code snippet.

------------------------------------------------------------------

param(
    [string]$directoryPath,
    [string[]]$keywords
)

# Function to search for keywords in files
function Search-Files {
    param (
        [string]$path,
        [string[]]$keywords
    )

    # Get all files in the directory and subdirectories
    $files = Get-ChildItem -Path $path -Recurse -File

    # Loop through each file
    foreach ($file in $files) {
        # Read the content of the file
        $content = Get-Content -Path $file.FullName

        # Loop through each line in the file content
        foreach ($line in $content) {
            # Check if the line contains any of the keywords (case-insensitive)
            foreach ($keyword in $keywords) {
                if ($line -match "(?i)$keyword") {
                    # Print the file name and the matching line
                    Write-Output "File: $($file.FullName)"
                    Write-Output "Line: $line"
                    Add-Content -path out.txt -value $line
                    break
                }
            }
        }
    }
}

# Call the function with the provided parameters
Search-Files -path $directoryPath -keywords $keywords

------------------------------------------------------------------

• Save the above code snippet to a .ps1 file (example: find.ps1) and you can run it as follows:

> .\find.ps1 -directoryPath "C:\esx-esxi1.xre.com-2025-04-18--13.05-2106842\var\core" -keywords "bluescreen"
> .\find.ps1 -directoryPath "C:\esx-esxi1.xre.com-2025-04-18--13.05-2106842\var\core" -keywords "bluescreen", "#PF Exception"

• All the log lines that include the given keyword or keywords will be saved to out.txt file.
• A sample output of the above-mentioned code snippet against an ESXi core dump is given below.

• Once you identify the root cause of the PSOD event, you can start working towards the resolution which may involve replacing a faulty hardware component, updating firmware/ driver/ ESXi, etc.

References

• Interpreting an ESXi host purple diagnostic screen 
• Extracting the log file after an ESXi host fails with a purple screen error

Hope it was useful. Cheers!

Azure AI Foundry - Part4 - Deploy and use a generative AI model

Azure AI Foundry supports deploying large language models (LLMs). In this article, we will see how to deploy a model and use it.

Azure AI Foundry Portal

• Select your project - My assets - Models + endpoints - Deploy model
• Click Deploy base model
• Select the model you want to deploy (here I am selecting gpt-4.1) and click Confirm

• You can see the deployment details like capacity (token per minute), resource location etc. 
• Click on Create resource and deploy

• Now it will start creating the resource and this step may take a minute or so.
• Once it is done, it will take you to the following page where you can see the mode details on the model you just deployed.

• Click on Open in playground to test the model.
• Once the chat playground is open, you will see your deployment, and under that you will see a section where you can give the model instructions and context. An example is given in the following screenshot. Once the model instructions and context are provided make sure to click Apply changes button. 
• Now you can click on Generate prompt, provide the query and click on Send.
• You can also set values for limiting the maximum output token for the model response, temperature, frequency penalty etc. under the Parameters section.

• A sample response is provided in the following screenshot.

• To see the sample code, you can click on View code.

• You can also see code samples and authentication using API key as shown below.

• Metrics (total requests, token count, etc.) related to your LLM model deployment can be found on the following page.

 

Python

Sample code to interact with the model can be found in my GitHub repo.

Hope it was useful. Cheers!

Azure AI Foundry Blog Series

Azure AI Foundry is a comprehensive suite of tools and services designed to accelerate the development and deployment of AI solutions on the Azure platform. Throughout this blog series, we will cover various aspects of Azure AI Foundry.

Part1 - Create project
Part2 - Language translation using AI Services
Part3 - Abstractive text summarization
Part4 - Deploy and use a generative AI model

Azure AI Foundry - Part3 - Abstractive text summarization

In this article, I will show you how to use Azure Cognitive Services for text summarization. 

Azure AI Foundry portal

• AI Services - Language + Translator

• Summarize Information - Summarize text

• Select a connected AI service resource or create a new one.

• Playgrounds - Summarize Information - Summarize text

Python

Sample code to summarize a PDF can be found in my GitHub repo. Following is an example of a resume summary: 

Hope this was useful. Cheers!

Azure AI Foundry - Part2 - Language translation using AI Services

In this article, I will show you how to use an Azure AI Service available within the Azure AI Foundry project. We'll use the language translator as an example.

Azure AI Foundry portal

• Select AI Services. 
• Click on Language + Translator.

• Select Translation.

• Select Text Translation.

• Click on Try with your own.

• Here I am translating language from English to Malayalam. Take a look at the Connected Azure AI Services, you can see it is already connected to one. Incase if it is not connected to an Azure AI Services resource, you can click on Create a new AI Services resource, select a region, provide an AI Services name if you like to and click Create and connect. 

• You can also view the sample code by clicking on View code.

• Here is the sample code in Python and when you scroll down you can find the Resource key and Region details.

Python

import os, requests, uuid, json

resource_key = 'resource_key_here'
region = 'region_here'
endpoint = 'https://api.cognitive.microsofttranslator.com/'

# If you encounter any issues with the base_url or path, make sure
# that you are using the latest endpoint: https://docs.microsoft.com/azure/cognitive-services/translator/reference/v3-0-translate
path = '/translate?api-version=3.0'
params = '&to=ml'
constructed_url = endpoint + path + params

headers = {
    'Ocp-Apim-Subscription-Key': resource_key,
    'Ocp-Apim-Subscription-Region': region,
    'Content-type': 'application/json',
    'X-ClientTraceId': str(uuid.uuid4())
}

# You can pass more than one object in body.
body = [{
    'text' : 'where are you right now?'
}]
request = requests.post(constructed_url, headers=headers, json=body)
response = request.json()

print(json.dumps(response, sort_keys=True, indent=4, separators=(',', ': ')))

Sample output:

What you see is a unicode string and once it is converted you will see the corresponding Malayalam text. 

Curl

curl -X POST "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0&to=ml" \
> -H "Ocp-Apim-Subscription-Key: your_key_here" \
> -H "Ocp-Apim-Subscription-Region: your_region_here" \
> -H "Content-Type: application/json" \
> -d "[{'Text':'where are you now?'}]" -v

Sample output:

Hope this was useful. Cheers!

References

• Translator Translate Method - Azure AI services | Microsoft Learn
• What is Azure Text Translation? - Azure AI services | Microsoft Learn
• Q: How do I use Azure Cognitive Services like Translator? | Microsoft Community Hub

Azure AI Foundry - Part1 - Create project

Azure AI Foundry is a unified platform that helps you design, customize, build, test, deploy, and manage generative AI applications. In this article, I will show you how to create a project and a hub, which are the first steps to building your AI solution. Let's get started!

Create project using the portal

• Sign in to the Azure AI Foundry portal https://ai.azure.com.

• Click Create project.
• The project will have an auto generated name or you can provide one. 
• You can also notice that it creates a new Hub, and Storage account, Key Vault and AI Services under a new resource group.
• Click Create.
• The project is getting created now. This may take a minute or two.

• Once it's done, it will take you to this overview page.

• On the Azure AI Foundry portal, you can use the Management center to configure/ get more details about your project, connected resources, models, endpoints etc.

• Under the Hub or Project properties, if you select the Resource Group, it will open a new browser tab and navigate to the Azure portal where you can see all the Azure resources that have been created to support your hub and project.

Create project using Azure CLI

Note: Remove any existing installation of the ml and azure-cli-ml extensions and install new.

• az extension remove -n azure-cli-ml
• az extension remove -n ml
• az extension add -n ml
• az extension update -n ml
  
  
• az login
• az account set --subscription "subscription_id"
• az group create --name "resource_group_name" --location "location_name"
  
  
• az ml workspace create --kind hub --resource-group "resource_group_name" --name "hub_name"

• $hub_id = "you will get this id from the output of previous step"

• az ml workspace create --kind project --hub-id $hub_id --resource-group "resource_group_name" --name "project_name"

In the next part we will explore the AI Services that are available within your Azure AI Foundry project. Hope this was useful. Cheers!

References

• Create an Azure AI Foundry project in Azure AI Foundry portal - Azure AI Foundry | Microsoft Learn
• Install and set up the CLI (v2) - Azure Machine Learning | Microsoft Learn
• How to create a hub using the Azure Machine Learning SDK/CLI - Azure AI Foundry | Microsoft Learn