Working with GPUs - Part2 - Memory fault indicators

AI workloads rely heavily on GPU memory reliability and, memory faults can silently degrade performance long before a GPU outright fails. Understanding which signals truly indicate GPU memory issues, and how to act on them is essential for stable operations at scale. This post focuses on authoritative memory fault indicators on Nvidia H100 GPUs, how they differ, and how to use them together to make correct operational decisions.

HBM3 (High Bandwidth Memory 3) memory on H100 delivers massive bandwidth, but it operates under extreme thermal, electrical, and utilization stress. When memory reliability starts degrading:

• Model training can fail intermittently
• NCCL performance may collapse unexpectedly
• Silent data corruption risks increase
• Faulty GPUs can impact entire multi‑GPU jobs

Early detection lets you drain, reset, isolate, or RMA a GPU before a customer‑visible incident occurs.

Primary indicators

ECC errors

ECC (Error Correcting Code) detects and reports bit‑level memory errors in HBM3.

ECC error types:

• Correctable Errors (CE)
• Single‑bit errors fixed automatically
• Indicate beginning of memory stress or aging
• Uncorrectable Errors (UE)
• Multi‑bit errors not recoverable
• High risk of data corruption
• Immediate GPU isolation required

nvidia-smi -q -d ECC

nvidia-smi --query-gpu=index,name,ecc.errors.corrected.volatile.device_memory,ecc.errors.corrected.volatile.dram,ecc.errors.corrected.volatile.sram,ecc.errors.corrected.volatile.total --format=csv

nvidia-smi --query-gpu=index,name,ecc.errors.uncorrected.volatile.device_memory,ecc.errors.uncorrected.volatile.dram,ecc.errors.uncorrected.volatile.sram,ecc.errors.uncorrected.volatile.total --format=csv

Notes

• Rising CE counts is an early warning. Monitor closely.
• Any UE count > 0, then drain workload, isolate GPU, and proceed to fix it.

Remapped Rows

Row Remapping is a hardware healing mechanism in H100 HBM3.

When the GPU identifies failing memory rows:

• Faulty rows are permanently retired
• Spare rows are mapped in their place

nvidia-smi -q -d ROW_REMAPPER

Notes

• Remapped Rows Pending = Yes
• GPU detected bad memory rows
• Reset required to complete remap
• Remapped Rows > 0
• Hardware has already consumed spare memory
• Strong early RMA signal

Row remapping is the earliest and strongest indicator of degrading HBM3 memory; often appearing before serious ECC failures.

Hope it was useful. Cheers!