vineethac.blogspot.com: HBM3

Now that we have a foundational understanding of the core utilities used to monitor and manage GPUs, let's dive into the hardware architecture of the NVIDIA H100 SXM. To truly understand GPU computing, it is essential to visualize how data flows through the silicon. The following overview maps out the internal components of the H100, providing a clear frame of reference so you can easily correlate key architectural terms such as Streaming Multiprocessors (SMs), Tensor Cores, L2 Cache, High Bandwidth Memory (HBM3), etc.

Overview

H100 is released around Sep 2022
Based on the Hopper architecture
It comes in two form factors

PCIe (300W)
SXM (700W)

Has 80 GB HBM3 memory (3.35 TB/s)
132 SMs
528 Tensor cores (4 per SM)
80B transistors on a custom 4N process node

Architecture

Image ref: NVIDIA Hopper Architecture In-Depth | NVIDIA Technical Blog

HBM3 - High Bandwidth Memory

This is the off-chip 80 GB device memory.
Divided in 5 stacks and connected via 10 independent 512-bit memory controllers.
Data flow: SM - L1 - L2 - memory partition/ crossbar - memory controller - HBM stack
H100 SXM5 has 5 HBM3 stacks.
HBM3 stack is DRAM.

L2 cache

50 MB of L2 cache, divided into two 25 MB partitions.
L2 cache is SRAM.

Unified shared memory + L1 cache

256 KB size 33 TB/s bandwidth per SM divided into 32 banks, each 32 bits (4 bytes) wide.
These are SRAM.

Registers

Every thread gets a private set of on-chip registers.
They have very high bandwidth, and very low latency.
256 KB per SM.

Gigathread engine

This is the hardware that takes a kernel launch and hands out thread blocks to SMs.
It tracks which thread blocks are not yet started, running, and finished.
When an SM has capacity to run another thread block, the Gigathread engine assigns the next thread block to that SM.
This ensures intelligent work distribution for optimal GPU utilization.

SM - Streaming Multiprocessor

SMs are the fundamental execution unit of the GPU which executes thread blocks of a CUDA kernel.
H100 SXM5 GPU has 132 SMs.
Following are the components of SM:

FP32 CUDA cores, Int/FP64 units
4th gen Tensor cores
Shared memory/ L1 cache
L1 instruction cache
Warp scheduler
Dispatch units
Registers
L0 instruction cache

Each SM is divided into 4 identical sub-divisions called Quadrants or SMSPs (SM Sub Partitions).

TMA - Tensor Memory Accelerator

Each SM has a TMA unit.
Offloads the tensor copy operations from the SMs.

Tensor core

They are really fast units for performing MMA operations (Matrix Multiply Accumulate).
4 tensor cores per SM.

GPC - Graphics Processing Cluster

It is a group of 18 SMs.
There are 8 GPCs in a H100.
Each GPC is connected to its own chunk of L2 cache.
GPCs also enable the use of distributed shared memory between the SMs.

TPC - Texture Processing Unit

Single TPC holds 2 SMs.
Job of TPC is to have shared SM block, so that communication between the two SMs is really fast.

NVLink

4th gen NVLink.
18 NVLink 4.0 lanes which gives 900 GB/s of GPU-GPU bandwidth.

References

Hope it was useful. Cheers!

AI workloads rely heavily on GPU memory reliability and, memory faults can silently degrade performance long before a GPU outright fails. Understanding which signals truly indicate GPU memory issues, and how to act on them is essential for stable operations at scale. This post focuses on authoritative memory fault indicators on Nvidia H100 GPUs, how they differ, and how to use them together to make correct operational decisions.

HBM3 (High Bandwidth Memory 3) memory on H100 delivers massive bandwidth, but it operates under extreme thermal, electrical, and utilization stress. When memory reliability starts degrading:

Model training can fail intermittently
NCCL performance may collapse unexpectedly
Silent data corruption risks increase
Faulty GPUs can impact entire multi‑GPU jobs

Early detection lets you drain, reset, isolate, or RMA a GPU before a customer‑visible incident occurs.

Primary indicators

ECC errors

ECC (Error Correcting Code) detects and reports bit‑level memory errors in DRAM and SRAM. A H100 SXM GPU has multiple memory types like:

HBM3 (80 GB off-chip) - DRAM
Shared L2 cache (50 MB on the GPU die) - SRAM
L1 cache and shared memory within a SM - SRAM
Registers within a SM - SRAM

ECC error types:

Correctable Errors (CE)

Single‑bit errors fixed automatically
Indicate beginning of memory stress or aging

Uncorrectable Errors (UE)

Multi‑bit errors not recoverable
High risk of data corruption
Immediate GPU isolation required

nvidia-smi -q -d ECC

As per Nvidia documentation, regarding SRAM uncorrectable errors, the RMA criteria is met for events outlined below. Any of the following events will trigger the SRAM Threshold Exceeded flag:

More than 4 UCE Unique Count events within an address bank for parity protected SRAMs.
More than 2 UCE Unique Count events within an address bank for SECDED ECC protected SRAMs.

nvidia-smi --query-gpu=index,name,ecc.errors.corrected.volatile.device_memory,ecc.errors.corrected.volatile.dram,ecc.errors.corrected.volatile.sram,ecc.errors.corrected.volatile.total --format=csv

nvidia-smi --query-gpu=index,name,ecc.errors.uncorrected.volatile.device_memory,ecc.errors.uncorrected.volatile.dram,ecc.errors.uncorrected.volatile.sram,ecc.errors.uncorrected.volatile.total --format=csv

Notes

Rising CE counts is an early warning. Monitor closely.
Any UE count > 0, then drain workload, reboot/ isolate GPU, and if possible, ideally you should use these GPUs which already has history of UCE events for running low-priority jobs.
RMA criteria is met when SRAM Threshold Exceeded flag is set to Yes.

Remapped Rows

Row Remapping is a hardware healing mechanism in H100 HBM3.

When the GPU identifies failing memory rows:

Faulty rows are permanently retired
Spare rows are mapped in their place

nvidia-smi -q -d ROW_REMAPPER