Working with GPUs - Part7 - Unable to determine the device handle for GPU

One of the more common GPU failures encountered in large-scale AI clusters is a situation where the Nvidia driver can no longer communicate with one or more GPUs. In these cases, nvidia-smi may partially work, showing healthy GPUs while reporting errors for the affected devices. This post walks through a real-world example involving Nvidia H100 GPUs on an 8 GPU Supermicro server where two GPUs became inaccessible and generated XID 79 - GPU has fallen off the bus errors.

Symptoms

The first indication of the problem was that nvidia-smi could not communicate with all GPUs in the system.

Running the following command: nvidia-smi -L

Notice that GPUs 0-5 are detected correctly, while two GPUs fail during enumeration because the driver can no longer obtain a valid device handle. This means the Nvidia Management Library (NVML) loaded and detected the GPU on the PCIe bus but failed when attempting to initialize communication with the GPU.

A common observation in this state is:

• Queries against healthy GPUs continue to work.
• Queries against the affected GPUs fail immediately.
• Workloads that require all GPUs may fail or become stuck.

Troubleshooting

Step 1: Check for XID Errors

The next step is to look for Nvidia XID events.

• dcgmi dmon -e 230 --count 1

• dmesg | egrep -i "xid|fallen"

In this case, you can see logs indicating GPU has fallen off the bus.

# dmesg | egrep -i "xid|fallen"
[6780696.284301] NVRM: Xid (PCI:0000:bf:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284304] NVRM: GPU 0000:bf:00.0: GPU has fallen off the bus.
[6780696.284647] NVRM: Xid (PCI:0000:e4:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284648] NVRM: GPU 0000:e4:00.0: GPU has fallen off the bus.

XID 79 indicates that the Nvidia driver lost communication with the GPU over PCIe. From the operating system's perspective, the device is no longer responding correctly to configuration or memory transactions.

Potential causes include:

• PCIe link failures
• GPU hardware issues
• Power-related problems
• Motherboard or PCIe switch issues
• Firmware or driver defects
• Unexpected device resets

Once this occurs, workloads using the affected GPU are typically unable to continue.

Step 2: Verify PCIe Device Health

Next, inspect the PCIe devices directly.

• lspci | grep -i nvidia
• lspci -vvv -s <pcie device id>

# lspci | grep -i nvidia
05:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
06:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
19:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
2d:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
3f:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
66:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
9b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
ae:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)

# lspci -vvv -s bf:00.0
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

# lspci -vvv -s e4:00.0
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

A healthy GPU normally returns detailed PCIe configuration information. When lspci reports: !!! Unknown header type 7f - it usually means the operating system can still "see" something at that PCIe address, but it cannot successfully read the device's PCI configuration space.

This is a strong indication that the device is no longer responding correctly and aligns with the earlier XID 79 messages.

Step 3: Run DCGM PCIe Diagnostics

Nvidia DCGM provides additional validation capabilities.

Run: dcgmi diag -r 3

This confirms that CUDA is unable to initialize the affected GPU, further validating that the device is no longer accessible from the driver stack. At this point, multiple layers of the stack indicate the same issue, and these findings strongly suggest that the GPUs are no longer responding properly over the PCIe fabric.

Step 4: Check syslog

Take a look at the syslog, and you should be able to see NVRM or XID events in it. Following is a sample log snippet:

kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxxx
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxx
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.

Step 5: Verify BMC

You can also take a look the server health event logs in the BMC for events like GPU or HBM missing.

Resolution

Following are the most common steps you can try:

• Power cycle the server. In most cases this will resolve the issue.
• If a power cycle is not resolving the issue, you may try reprovisioning the node by reinstalling the OS and other required software components.
• Last option is to request for RMA of the server/ affected part by working with the vendor.

Hope it was useful. Cheers!

Configure syslog forwarding in vCenter servers using Python

As a system administrator, it's essential to ensure that your vCenter servers are properly configured to collect and forward system logs to a central location for monitoring and analysis. In this blog, we'll explore how to configure syslog forwarding in vCenter servers using Python.

You can access the Python script from my GitHub repository: 
https://github.com/vineethac/VMware/tree/main/vCenter/syslog_forwarding

In this blog, we've demonstrated how to get, test, and set syslog forwarding configuration in vCenter servers using Python. By following these steps, you can ensure that your vCenter servers are properly configured to collect and forward system logs to a central location for monitoring and analysis. Remember to replace the placeholders in the config file with your actual vCenter server names, syslog server IP address or hostname, port, and protocol.

Hope it was useful. Cheers!