One of the more common GPU failures encountered in large-scale AI clusters is a situation where the Nvidia driver can no longer communicate with one or more GPUs. In these cases, nvidia-smi may partially work, showing healthy GPUs while reporting errors for the affected devices. This post walks through a real-world example involving Nvidia H100 GPUs on an 8 GPU Supermicro server where two GPUs became inaccessible and generated XID 79 - GPU has fallen off the bus errors.
Symptoms
The first indication of the problem was that nvidia-smi could not communicate with all GPUs in the system.
Running the following command: nvidia-smi -L
Notice that GPUs 0-5 are detected correctly, while two GPUs fail during enumeration because the driver can no longer obtain a valid device handle. This means the Nvidia Management Library (NVML) loaded and detected the GPU on the PCIe bus but failed when attempting to initialize communication with the GPU.
A common observation in this state is:
- Queries against healthy GPUs continue to work.
- Queries against the affected GPUs fail immediately.
- Workloads that require all GPUs may fail or become stuck.
Troubleshooting
Step 1: Check for XID Errors
The next step is to look for Nvidia XID events.
- dcgmi dmon -e 230 --count 1
- dmesg | egrep -i "xid|fallen"
In this case, you can see logs indicating GPU has fallen off the bus.
# dmesg | egrep -i "xid|fallen"
[6780696.284301] NVRM: Xid (PCI:0000:bf:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284304] NVRM: GPU 0000:bf:00.0: GPU has fallen off the bus.
[6780696.284647] NVRM: Xid (PCI:0000:e4:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284648] NVRM: GPU 0000:e4:00.0: GPU has fallen off the bus.XID 79 indicates that the Nvidia driver lost communication with the GPU over PCIe. From the operating system's perspective, the device is no longer responding correctly to configuration or memory transactions.
Potential causes include:
- PCIe link failures
- GPU hardware issues
- Power-related problems
- Motherboard or PCIe switch issues
- Firmware or driver defects
- Unexpected device resets
Once this occurs, workloads using the affected GPU are typically unable to continue.
Step 2: Verify PCIe Device Health
Next, inspect the PCIe devices directly.
- lspci | grep -i nvidia
- lspci -vvv -s <pcie device id>
# lspci | grep -i nvidia
05:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
06:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
19:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
2d:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
3f:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
66:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
9b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
ae:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
# lspci -vvv -s bf:00.0
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
!!! Unknown header type 7f
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
# lspci -vvv -s e4:00.0
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
!!! Unknown header type 7f
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
A healthy GPU normally returns detailed PCIe configuration information. When lspci reports: !!! Unknown header type 7f - it usually means the operating system can still "see" something at that PCIe address, but it cannot successfully read the device's PCI configuration space.
This is a strong indication that the device is no longer responding correctly and aligns with the earlier XID 79 messages.
Step 3: Run DCGM PCIe Diagnostics
Nvidia DCGM provides additional validation capabilities.
Run: dcgmi diag -r 3
This confirms that CUDA is unable to initialize the affected GPU, further validating that the device is no longer accessible from the driver stack. At this point, multiple layers of the stack indicate the same issue, and these findings strongly suggest that the GPUs are no longer responding properly over the PCIe fabric.
Step 4: Check syslog
Take a look at the syslog, and you should be able to see NVRM or XID events in it. Unfortunately, I wasn't able to capture those relevant log snippets from syslog.
Step 5: Verify BMC
You can also take a look the server health event logs in the BMC for events like GPU or HBM missing.
Resolution
Following are the most common steps you can try:
- Power cycle the server. In most cases this will resolve the issue.
- If a power cycle is not resolving the issue, you may try reprovisioning the node by reinstalling the OS and other required software components.
- Last option is to request for RMA of the server/ affected part by working with the vendor.
