liveBook · Manning

Dynamic range

As explained in Build a DeepSeek Model (From Scratch)

Dynamic range is the span between the smallest and largest magnitudes that a numerical representation or computation can represent effectively.

Dynamic range is the span of magnitudes a numerical format can represent effectively, from very small values to very large ones. In floating-point representations, exponent bits control this range, while mantissa bits control precision and the spacing between representable values. section 5.5.3

Reducing bit width usually reduces dynamic range, precision, or both. FP32 provides broad range and high precision; FP16 sacrifices both, while BF16 retains FP32-like range with fewer mantissa bits. Limited range can produce overflow, underflow, or loss of detail. section 5.5.3

FP8 makes the range–precision trade-off explicit:

  • E5M2 uses more exponent bits, giving wider dynamic range but lower precision. section 5.6.4
  • E4M3 uses more mantissa bits, giving higher precision but a smaller range. section 5.6.4
  • Fine-grained quantization manages range locally by scaling smaller activation groups and weight blocks independently. section 5.6.2

An outlier can erase smaller values

Quantizing [2.0, 3.0, 500.0] with one scale based on 500 produces approximately [1, 1, 127]. After de-quantization, the smaller values become approximately [3.94, 3.94], losing their distinction. Groupwise scaling prevents the outlier from squashing unrelated values. section 5.6.2

Figure 5.18 A comparison of FP32 and FP16.

Compare how exponent and mantissa allocations change between FP32 and FP16, affecting range, precision, and memory. section 5.5.3

Figure 5.27 Fine-Grained Quantization for an activation vector. The vector is partitioned into smaller groups. Each group is scaled independently based on the maximum value within that group, preserving precision for values in groups that do not contain large outliers.

Inspect how partitioning an activation vector lets each group preserve precision despite different local magnitudes. section 5.6.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains exponent allocation, format comparisons, overflow, underflow, and blockwise range management.
Examples
Compares FP32, FP16, BF16, E4M3, and E5M2, then quantizes an outlier-containing vector.
What it assumes
Builds from sign, exponent, and mantissa roles before comparing floating-point formats.
What you can do after
Evaluate numerical formats and choose local scaling strategies that reduce overflow and precision loss.
How it connects
Connects exponent allocation to mantissa precision, quantization, fine-grained quantization, overflow, and underflow.

Also explained in

Compare how 2 books explain dynamic range →

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.