Underflow occurs when a numerical value or intermediate result is too small for its representation to preserve accurately. In reduced-precision computation, it appears when limited exponent range or precision causes small magnitudes to lose detail. FP16 is vulnerable because it has fewer exponent and mantissa bits than FP32; stale scaling factors can produce the same failure during FP8 quantization. section 5.5.3 section 5.6.5
Underflow appears in several implementation bottlenecks:
- Reduced-precision formats: FP16 can lose detail for very small values, while BF16 preserves FP32's exponent range and is therefore more resistant to range-related failures. section 5.5.3
- Matrix multiplication: summing thousands of small FP8 products in a limited-precision accumulator can make intermediate sums inaccurate and introduce substantial final error. section 5.6.3
- Quantization: scaling a small current tensor with a historical factor calibrated for larger values can squash its representable detail. section 5.6.5
Online scaling prevents stale-range failures
For each current activation or weight tensor, find its maximum absolute value, derive a fresh scaling factor, and quantize immediately. This aligns the representable range with the current data rather than historical maxima, avoiding underflow and overflow caused by stale calibration. section 5.6.5
Figure 5.30 The core strategy for increasing accumulation precision: intermediate results are periodically moved from the low-precision environment of the Tensor Cores to the high-precision environment of the CUDA Cores.

Visualizes how low-precision Tensor Core accumulation is periodically promoted to an FP32 CUDA Core register to reduce accumulation underflow. section 5.6.3

