liveBook · Manning

Mantissa

As explained in Build a DeepSeek Model (From Scratch)

A mantissa, or significand, is the component of a floating-point representation that determines the significant digits and precision of a value.

The mantissa, also called the significand, stores the significant digits of a floating-point number. It determines precision: more mantissa bits produce more significant digits and smaller gaps between representable values. The exponent instead controls dynamic range, while the sign bit records positivity or negativity. section 5.5.3

Compare the precision and range trade-offs across common numerical formats:

  • FP32 allocates 23 mantissa bits, alongside 8 exponent bits, providing high precision and broad dynamic range. section 5.5.3
  • FP16 allocates 10 mantissa bits and 5 exponent bits, reducing memory while narrowing both precision and range; overflow and underflow become more likely. section 5.5.3
  • BF16 keeps FP32's 8 exponent bits but reduces the mantissa to 7 bits, preserving dynamic range at lower precision. section 5.5.3
  • FP8 E5M2 favors range with 5 exponent bits and 2 mantissa bits, whereas E4M3 favors precision with 4 exponent bits and 3 mantissa bits. section 5.6.4

Mantissa bits become especially important in quantized training. A conventional FP8 strategy uses E4M3 for the forward pass and E5M2 for gradients, where large outliers can overflow the narrower range. Fine-grained quantization scales small, independent blocks, limiting outlier effects and allowing E4M3's higher precision to be used uniformly for forward and backward operations. section 5.6.4

Figure 5.18 A comparison of FP32 and FP16.

Use the FP32-versus-FP16 comparison to visualize how reducing mantissa and exponent bits changes precision and dynamic range. section 5.5.3

Figure 5.19 A comparison of FP32 and BFloat16.

Use the FP32-versus-BF16 comparison to see how BF16 preserves exponent capacity while sacrificing mantissa precision. section 5.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines mantissa precision, compares concrete bit allocations, and connects local scaling to uniform FP8 format selection.
Examples
Compares FP32, FP16, BF16, E5M2, and E4M3 through bit allocations and format diagrams.
What you can do after
Evaluate numerical formats by precision, range, memory cost, and suitability for forward or backward quantized training.
How it connects
Connects mantissa to floating-point representation, exponent, precision, dynamic range, quantization, FP8, FP16, BF16, and mixed-precision training.

Also explained in

Compare how 7 books explain mantissa →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.