vineethac.blogspot.com: H100 SXM5

One of the more common GPU failures encountered in large-scale AI clusters is a situation where the Nvidia driver can no longer communicate with one or more GPUs. In these cases, nvidia-smi may partially work, showing healthy GPUs while reporting errors for the affected devices. This post walks through a real-world example involving Nvidia H100 GPUs on an 8 GPU Supermicro server where two GPUs became inaccessible and generated XID 79 - GPU has fallen off the bus errors.

Symptoms

The first indication of the problem was that nvidia-smi could not communicate with all GPUs in the system.

Running the following command: nvidia-smi -L

Notice that GPUs 0-5 are detected correctly, while two GPUs fail during enumeration because the driver can no longer obtain a valid device handle. This means the Nvidia Management Library (NVML) loaded and detected the GPU on the PCIe bus but failed when attempting to initialize communication with the GPU.

A common observation in this state is:

Queries against healthy GPUs continue to work.
Queries against the affected GPUs fail immediately.
Workloads that require all GPUs may fail or become stuck.

Troubleshooting

Step 1: Check for XID Errors

The next step is to look for Nvidia XID events.

dcgmi dmon -e 230 --count 1

dmesg | egrep -i "xid|fallen"

In this case, you can see logs indicating GPU has fallen off the bus.

# dmesg | egrep -i "xid|fallen"
[6780696.284301] NVRM: Xid (PCI:0000:bf:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284304] NVRM: GPU 0000:bf:00.0: GPU has fallen off the bus.
[6780696.284647] NVRM: Xid (PCI:0000:e4:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[6780696.284648] NVRM: GPU 0000:e4:00.0: GPU has fallen off the bus.

XID 79 indicates that the Nvidia driver lost communication with the GPU over PCIe. From the operating system's perspective, the device is no longer responding correctly to configuration or memory transactions.

Potential causes include:

PCIe link failures
GPU hardware issues
Power-related problems
Motherboard or PCIe switch issues
Firmware or driver defects
Unexpected device resets

Once this occurs, workloads using the affected GPU are typically unable to continue.

Step 2: Verify PCIe Device Health

Next, inspect the PCIe devices directly.

lspci | grep -i nvidia
lspci -vvv -s <pcie device id>

# lspci | grep -i nvidia
05:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
06:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
19:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
2d:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
3f:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
66:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
9b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
ae:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)

# lspci -vvv -s bf:00.0
bf:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

# lspci -vvv -s e4:00.0
e4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

A healthy GPU normally returns detailed PCIe configuration information. When lspci reports: !!! Unknown header type 7f - it usually means the operating system can still "see" something at that PCIe address, but it cannot successfully read the device's PCI configuration space.

This is a strong indication that the device is no longer responding correctly and aligns with the earlier XID 79 messages.

Step 3: Run DCGM PCIe Diagnostics

Nvidia DCGM provides additional validation capabilities.

Run: dcgmi diag -r 3

This confirms that CUDA is unable to initialize the affected GPU, further validating that the device is no longer accessible from the driver stack. At this point, multiple layers of the stack indicate the same issue, and these findings strongly suggest that the GPUs are no longer responding properly over the PCIe fabric.

Step 4: Check syslog

Take a look at the syslog, and you should be able to see NVRM or XID events in it. Following is a sample log snippet:

kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003187] pcieport 0000:18:00.0: pciehp: Slot(1-1): Card not present
kernel	[19209.003184] pcieport 0000:18:00.0: pciehp: Slot(1-1): Link Down
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.003246] NVRM: Attempting to remove device 0000:19:00.0 with non-zero usage count!
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxxx
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174506] NVRM: GPU at PCI:0000:19:00: GPU-16xx3295-1242-bdc9-b0bf-0d540dxxxx
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174516] NVRM: GPU 0000:19:00.0: GPU has fallen off the bus.
kernel	[19209.174524] NVRM: A GPU crash dump has been created. If possible, please run
kernel	[19209.174524] NVRM: nvidia-bug-report.sh as root to collect this data before
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174513] NVRM: Xid (PCI:0000:19:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.
kernel	[19209.174524] NVRM: the NVIDIA kernel module is unloaded.
kernel	[19209.174511] NVRM: GPU Board Serial Number: xx551xx014xx7
kernel	[19209.174517] NVRM: GPU 0000:19:00.0: GPU serial number is xx551xx014xx7.

Step 5: Verify BMC

You can also take a look the server health event logs in the BMC for events like GPU or HBM missing.

Resolution

Following are the most common steps you can try:

Power cycle the server. In most cases this will resolve the issue.
If a power cycle is not resolving the issue, you may try reprovisioning the node by reinstalling the OS and other required software components.
Last option is to request for RMA of the server/ affected part by working with the vendor.

Hope it was useful. Cheers!

Now that we have a foundational understanding of the core utilities used to monitor and manage GPUs, let's dive into the hardware architecture of the NVIDIA H100 SXM. To truly understand GPU computing, it is essential to visualize how data flows through the silicon. The following overview maps out the internal components of the H100, providing a clear frame of reference so you can easily correlate key architectural terms such as Streaming Multiprocessors (SMs), Tensor Cores, L2 Cache, High Bandwidth Memory (HBM3), etc.

Overview

H100 is released around Sep 2022
Based on the Hopper architecture
It comes in two form factors

PCIe (300W)
SXM (700W)

Has 80 GB HBM3 memory (3.35 TB/s)
132 SMs
528 Tensor cores (4 per SM)
80B transistors on a custom 4N process node

Architecture

Image ref: NVIDIA Hopper Architecture In-Depth | NVIDIA Technical Blog

HBM3 - High Bandwidth Memory

This is the off-chip 80 GB device memory.
Divided in 5 stacks and connected via 10 independent 512-bit memory controllers.
Data flow: SM - L1 - L2 - memory partition/ crossbar - memory controller - HBM stack
H100 SXM5 has 5 HBM3 stacks.
HBM3 stack is DRAM.

L2 cache

50 MB of L2 cache, divided into two 25 MB partitions.
L2 cache is SRAM.

Unified shared memory + L1 cache

256 KB size 33 TB/s bandwidth per SM divided into 32 banks, each 32 bits (4 bytes) wide.
These are SRAM.

Registers

Every thread gets a private set of on-chip registers.
They have very high bandwidth, and very low latency.
256 KB per SM.

Gigathread engine

This is the hardware that takes a kernel launch and hands out thread blocks to SMs.
It tracks which thread blocks are not yet started, running, and finished.
When an SM has capacity to run another thread block, the Gigathread engine assigns the next thread block to that SM.
This ensures intelligent work distribution for optimal GPU utilization.

SM - Streaming Multiprocessor

SMs are the fundamental execution unit of the GPU which executes thread blocks of a CUDA kernel.
H100 SXM5 GPU has 132 SMs.
Following are the components of SM:

FP32 CUDA cores, Int/FP64 units
4th gen Tensor cores
Shared memory/ L1 cache
L1 instruction cache
Warp scheduler
Dispatch units
Registers
L0 instruction cache

Each SM is divided into 4 identical sub-divisions called Quadrants or SMSPs (SM Sub Partitions).

TMA - Tensor Memory Accelerator

Each SM has a TMA unit.
Offloads the tensor copy operations from the SMs.

Tensor core

They are really fast units for performing MMA operations (Matrix Multiply Accumulate).
4 tensor cores per SM.

GPC - Graphics Processing Cluster

It is a group of 18 SMs.
There are 8 GPCs in a H100.
Each GPC is connected to its own chunk of L2 cache.
GPCs also enable the use of distributed shared memory between the SMs.

TPC - Texture Processing Unit

Single TPC holds 2 SMs.
Job of TPC is to have shared SM block, so that communication between the two SMs is really fast.

NVLink

4th gen NVLink.
18 NVLink 4.0 lanes which gives 900 GB/s of GPU-GPU bandwidth.

References