Working with GPUs - Part7 - Unable to determine the device handle for GPU

One of the more common GPU failures encountered in large-scale AI clusters is a situation where the Nvidia driver can no longer communicate with one or more GPUs. In these cases, nvidia-smi may partially work, showing healthy GPUs while reporting errors for the affected devices. This post walks through a real-world example involving Nvidia H100 GPUs on an 8 GPU Supermicro server where two GPUs became inaccessible and generated XID 79 - GPU has fallen off the bus errors.

Symptoms

The first indication of the problem was that nvidia-smi could not communicate with all GPUs in the system.

Running the following command: nvidia-smi -L

Notice that GPUs 0-5 are detected correctly, while two GPUs fail during enumeration because the driver can no longer obtain a valid device handle. This means the Nvidia Management Library (NVML) loaded and detected the GPU on the PCIe bus but failed when attempting to initialize communication with the GPU.

A common observation in this state is:

• Queries against healthy GPUs continue to work.
• Queries against the affected GPUs fail immediately.
• Workloads that require all GPUs may fail or become stuck.

Troubleshooting

Step 1: Check for XID Errors

The next step is to look for Nvidia XID events.

• dcgmi dmon -e 230 --count 1

• dmesg | egrep -i "xid|fallen"

In this case, you can see logs indicating GPU has fallen off the bus.

# dmesg | egrep -i "xid|fallen"
[6780696.284301] NVRM: Xid (PCI:0000:bf:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284304] NVRM: GPU 0000:bf:00.0: GPU has fallen off the bus.
[6780696.284647] NVRM: Xid (PCI:0000:e4:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284648] NVRM: GPU 0000:e4:00.0: GPU has fallen off the bus.

XID 79 indicates that the Nvidia driver lost communication with the GPU over PCIe. From the operating system's perspective, the device is no longer responding correctly to configuration or memory transactions.

Potential causes include:

• PCIe link failures
• GPU hardware issues
• Power-related problems
• Motherboard or PCIe switch issues
• Firmware or driver defects
• Unexpected device resets

Once this occurs, workloads using the affected GPU are typically unable to continue.

Step 2: Verify PCIe Device Health

Next, inspect the PCIe devices directly.

• lspci | grep -i nvidia
• lspci -vvv -s <pcie device id>

# lspci | grep -i nvidia
05:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
06:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
19:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
2d:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
3f:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
66:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
9b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
ae:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)

# lspci -vvv -s bf:00.0
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

# lspci -vvv -s e4:00.0
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

A healthy GPU normally returns detailed PCIe configuration information. When lspci reports: !!! Unknown header type 7f - it usually means the operating system can still "see" something at that PCIe address, but it cannot successfully read the device's PCI configuration space.

This is a strong indication that the device is no longer responding correctly and aligns with the earlier XID 79 messages.

Step 3: Run DCGM PCIe Diagnostics

Nvidia DCGM provides additional validation capabilities.

Run: dcgmi diag -r 3

This confirms that CUDA is unable to initialize the affected GPU, further validating that the device is no longer accessible from the driver stack. At this point, multiple layers of the stack indicate the same issue, and these findings strongly suggest that the GPUs are no longer responding properly over the PCIe fabric.

Step 4: Check syslog

Take a look at the syslog, and you should be able to see NVRM or XID events in it. Following is a sample log snippet:

kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxxx
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxx
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.

Step 5: Verify BMC

You can also take a look the server health event logs in the BMC for events like GPU or HBM missing.

Resolution

Following are the most common steps you can try:

• Power cycle the server. In most cases this will resolve the issue.
• If a power cycle is not resolving the issue, you may try reprovisioning the node by reinstalling the OS and other required software components.
• Last option is to request for RMA of the server/ affected part by working with the vendor.

Hope it was useful. Cheers!

Working with GPUs - Part5 - XID errors

If you are running large-scale AI training or LLM inference, you already know that managing a GPU cluster is less about "if" things break, and more about "when" and "why". In this post, we’ll demystify Nvidia XID errors, interpret them in the context of H100 NVLink systems, and outline a practical approach to triage and remediation.

XID (short for eXception ID) errors are diagnostic messages emitted by the Nvidia kernel driver (NVRM) when a GPU encounters an abnormal condition or fault. While some point to minor software glitches, others signal catastrophic hardware failures. With the H100 equipped with High Bandwidth Memory (HBM3) and NVLink interconnects - understanding these errors are critical to minimizing downtime.

How do we identify if any GPUs has XID errors

DCGMI diag

One way to identify XID errors is from DCGMI level 3 tests. If a critical or fatal hardware XID fires while the Level 3 tests are actively running (or if a sticky hardware error state was already present), the test will fail and output specific error strings. Here is an example:

# dcgmi diag -r 3
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.1.8                                          |
| Driver Version Detected   | 550.90.07                                      |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - All                                     |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
| Diagnostic                | Pass - GPUs: 1, 2, 3, 4, 5, 6, 7               |
|                           | Fail - GPU: 0                                  |
| Warning                   | GPU 0 Found 56954234 faulty memory elements o  |
|                           | n GPU 0 Run a field diagnostic on the GPU.     |
| Info                      | GPU 0 Allocated space for 137 output matricie  |
|                           | s from 75937126809 bytes available., GPU 0 Ru  |
|                           | nning with precisions: FP64 1, FP32 1, FP16 1  |
|                           | , GPU 0 GPU 0 calculated at approximately 230  |
|                           | 2.72 gigaflops during this test                |
+-----  Stress  ------------+------------------------------------------------+
| Targeted Stress           | Pass - All                                     |
| Targeted Power            | Pass - GPUs: 1, 2, 3, 4, 5, 6, 7               |
|                           | Fail - GPU: 0                                  |
| Warning                   | GPU 0 Detected 43 xid_errors for GPU 0         | < xid_error
| Info                      | GPU 0 GPU 0 power average: 161 W               |
| Info                      | GPU 1 GPU 1 power average: 170 W               |
| Info                      | GPU 2 GPU 2 power average: 164 W               |
| Info                      | GPU 3 GPU 3 power average: 158 W               |
| Info                      | GPU 4 GPU 4 power average: 154 W               |
| Info                      | GPU 5 GPU 5 power average: 169 W               |
| Info                      | GPU 6 GPU 6 power average: 158 W               |
| Info                      | GPU 7 GPU 7 power average: 159 W               |
| Memory Bandwidth          | Pass - All                                     |
| EUD Test                  | Skip - All                                     |
+---------------------------+------------------------------------------------+

Note that while DCGMI is exceptionally good at flagging structural hardware faults, dcgmi diag will generally not flag application-level or user-space errors, even if they generate XIDs.

DCGMI dmon

Here is another way to look for XID errors using dcgmi dmon.

# dcgmi dmon -e 230 --count 1
#Entity   XIDER
ID
GPU 7     0
GPU 6     0
GPU 5     0
GPU 4     0
GPU 3     0
GPU 2     0
GPU 1     0
GPU 0     43

• -e 230 is the filed id that shows the XID errors. The value shown under XIDER column is the specific XID error.
• Note: If there are non‑zero values, that would mean one or more GPUs had logged Xid errors, and we need to cross‑reference the specific Xid codes in the kernel log and documentation to understand the nature of the fault.
• Ref: Field Identifiers — NVIDIA DCGM Documentation latest documentation

OS logs

You will also find these XID errors in OS kernel logs, and syslog. Following are some examples:

# dmesg | grep -i xid
[  747.179157] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 1): Out Of Range Address
[  747.180533] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a7b0=0x100000e 0x56a7b4=0x20 0x56a7a8=0x1f81fb60 0x56a7ac=0x1174
[  747.209815] NVRM: Xid (PCI:0000:19:00): 43, pid=8548, name=nvvs, Ch 00000009
[ 1250.627528] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 3, SM 0): Out Of Range Address
[ 1250.629013] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x568730=0x100000e 0x568734=0x20 0x568728=0x1f81fb60 0x56872c=0x1174
[ 1250.657381] NVRM: Xid (PCI:0000:19:00): 43, pid=10603, name=nvvs, Ch 00000009
[45911.449627] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 0): Out Of Range Address
[45911.451042] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a730=0x103000e 0x56a734=0x20 0x56a728=0x1f81fb60 0x56a72c=0x1174
[45911.479823] NVRM: Xid (PCI:0000:19:00): 43, pid=79302, name=nvvs, Ch 00000009


# journalctl -k | grep -i xid
May 04 20:30:00 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 1): Out Of Range Address
May 04 20:30:00 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a7b0=0x100000e 0x56a7b4=0x20 0x56a7a8=0x1f81fb60 0x56a7ac=0x1174
May 04 20:30:00 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 43, pid=8548, name=nvvs, Ch 00000009
May 04 20:38:23 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 3, SM 0): Out Of Range Address
May 04 20:38:23 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x568730=0x100000e 0x568734=0x20 0x568728=0x1f81fb60 0x56872c=0x1174
May 04 20:38:23 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 43, pid=10603, name=nvvs, Ch 00000009
May 05 09:02:43 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 0): Out Of Range Address
May 05 09:02:43 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a730=0x103000e 0x56a734=0x20 0x56a728=0x1f81fb60 0x56a72c=0x1174
May 05 09:02:43 xx110-r113-node-02 kernel: NVRM: Xid (PCI:0000:19:00): 43, pid=79302, name=nvvs, Ch 00000009


# grep -i xid /var/log/syslog
May  4 20:30:00 xx110-r113-node-02 kernel: [  747.179157] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 1): Out Of Range Address
May  4 20:30:00 xx110-r113-node-02 kernel: [  747.180533] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a7b0=0x100000e 0x56a7b4=0x20 0x56a7a8=0x1f81fb60 0x56a7ac=0x1174
May  4 20:30:00 xx110-r113-node-02 kernel: [  747.209815] NVRM: Xid (PCI:0000:19:00): 43, pid=8548, name=nvvs, Ch 00000009
May  4 20:38:23 xx110-r113-node-02 kernel: [ 1250.627528] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 3, SM 0): Out Of Range Address
May  4 20:38:23 xx110-r113-node-02 kernel: [ 1250.629013] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x568730=0x100000e 0x568734=0x20 0x568728=0x1f81fb60 0x56872c=0x1174
May  4 20:38:23 xx110-r113-node-02 kernel: [ 1250.657381] NVRM: Xid (PCI:0000:19:00): 43, pid=10603, name=nvvs, Ch 00000009
May  4 20:40:44 xx110-r113-node-02 drpcli[4139]: Starting xid error detection test...
May  4 20:40:44 xx110-r113-node-02 drpcli[4139]: [MANDATORY] test_gpu_xid_errors: PASS, GPU XID error check passed. No errors found.
May  5 09:02:43 xx110-r113-node-02 kernel: [45911.449627] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics SM Warp Exception on (GPC 6, TPC 5, SM 0): Out Of Range Address
May  5 09:02:43 xx110-r113-node-02 kernel: [45911.451042] NVRM: Xid (PCI:0000:19:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x56a730=0x103000e 0x56a734=0x20 0x56a728=0x1f81fb60 0x56a72c=0x1174
May  5 09:02:43 xx110-r113-node-02 kernel: [45911.479823] NVRM: Xid (PCI:0000:19:00): 43, pid=79302, name=nvvs, Ch 00000009

Common XID errors in H100 NVL GPUs

Application/ CUDA errors: XID 11/25/32/37/69/80 are often caused by application bugs. These are typically recoverable after the application restart.

Memory/ ECC errors: XID 48/64/94/95/140 are caused by GPU memory/ ECC/ remapping related errors or events. Immediate action is to reset the GPU, and if the problem persists, contact your hardware vendor.

NVLink fabric fault: XID 74 indicates a problem with a connection from the GPU to another GPU or NVSwitch over NVLink. A GPU reset or node reboot is needed to clear this error. This event may indicate a hardware failure with the link itself or may indicate a problem with the device at the remote end of the link. For example, if a GPU fails, another GPU connected to it over NVLink may report an XID 74 simply because the link went down as a result. The nvidia-smi nvlink command can provide additional details on NVLink errors, and connection information on the links. If this error is seen repeatedly and GPU reset or node reboot fails to clear the condition, contact your hardware vendor for support.

Here is the full list of XIDs, including their applicability across platforms (H100, B100, GB200, etc.): Analyzing Xid Errors with the Xid Catalog — XID Errors

References

• Analyzing Xid Errors with the Xid Catalog — XID Errors

Hope this was useful. Cheers!