liveBook · Manning

Underflow

As explained in Build a DeepSeek Model (From Scratch)

Underflow is the condition in which a numerical value is too small in magnitude for a format to represent its detail accurately.

Underflow occurs when a numerical value or intermediate result is too small for its representation to preserve accurately. In reduced-precision computation, it appears when limited exponent range or precision causes small magnitudes to lose detail. FP16 is vulnerable because it has fewer exponent and mantissa bits than FP32; stale scaling factors can produce the same failure during FP8 quantization. section 5.5.3 section 5.6.5

Underflow appears in several implementation bottlenecks:

  • Reduced-precision formats: FP16 can lose detail for very small values, while BF16 preserves FP32's exponent range and is therefore more resistant to range-related failures. section 5.5.3
  • Matrix multiplication: summing thousands of small FP8 products in a limited-precision accumulator can make intermediate sums inaccurate and introduce substantial final error. section 5.6.3
  • Quantization: scaling a small current tensor with a historical factor calibrated for larger values can squash its representable detail. section 5.6.5

Online scaling prevents stale-range failures

For each current activation or weight tensor, find its maximum absolute value, derive a fresh scaling factor, and quantize immediately. This aligns the representable range with the current data rather than historical maxima, avoiding underflow and overflow caused by stale calibration. section 5.6.5

Figure 5.30 The core strategy for increasing accumulation precision: intermediate results are periodically moved from the low-precision environment of the Tensor Cores to the high-precision environment of the CUDA Cores.

Visualizes how low-precision Tensor Core accumulation is periodically promoted to an FP32 CUDA Core register to reduce accumulation underflow. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Connects underflow to exponent range, mantissa precision, accumulation registers, scaling factors, and mitigation strategies.
Examples
Uses FP16 detail loss, thousands of FP8 products, stale scaling factors, and Tensor Core promotion as concrete cases.
What it assumes
Assumes familiarity with floating-point formats, quantization, matrix multiplication, and GPU computational units.
What you can do after
Diagnose underflow by examining format range, accumulator precision, and whether quantization scales match current tensor magnitudes.
How it connects
Links underflow to floating-point formats, quantization, overflow, FP8, FP16, BF16, matrix multiplication, Tensor Cores, and CUDA Cores.

Also explained in

Compare how 2 books explain underflow →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.