liveBook · Manning

Exponent

As explained in Build a DeepSeek Model (From Scratch)

An exponent is the component of a floating-point representation that determines the scale and dynamic range of represented values.

An exponent allocates bits to magnitude in a floating-point number. It controls the binary decimal point, determining dynamic range: more exponent bits allow values to span a wider interval, from very small to very large. Mantissa bits instead control precision, so each format balances range against representational detail. section 5.5.3

The allocation becomes concrete across common numerical formats:

  • FP32 uses 8 exponent bits and 23 mantissa bits, providing broad dynamic range and high precision. section 5.5.3
  • FP16 uses 5 exponent bits and 10 mantissa bits, reducing memory while narrowing range and precision and increasing overflow or underflow risk. section 5.5.3
  • BF16 retains FP32's 8 exponent bits but reduces the mantissa to 7 bits, preserving dynamic range at lower precision. section 5.5.3
  • FP8 E5M2 favors range with 5 exponent bits, while E4M3 favors precision with 4 exponent bits and 3 mantissa bits. section 5.6.4

Local scaling changes the FP8 choice

A conventional mixed-precision design uses E4M3 for forward computation and E5M2 for gradients, whose outliers may exceed E4M3's range. Fine-grained quantization scales small, independent blocks, limiting local outlier effects and enabling uniform E4M3 for forward and backward operations. The design effectively shares exponent capacity across grouped elements while retaining more mantissa bits. section 5.6.4

Figure 5.18 A comparison of FP32 and FP16.

Compare how FP32 and FP16 distribute bits between exponent and mantissa. section 5.5.3

Figure 5.19 A comparison of FP32 and BFloat16.

Compare BF16's FP32-sized exponent with its reduced mantissa. section 5.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines exponent bits, compares FP32, FP16, BF16, E5M2, and E4M3, then applies the trade-off to fine-grained quantization.
Examples
Compares FP32, FP16, BF16, E5M2, and E4M3 through their exponent-mantissa allocations and training roles.
What it assumes
Assumes familiarity with floating-point numbers and introduces sign, exponent, mantissa, dynamic range, and precision together.
What you can do after
Evaluate numerical formats by predicting their dynamic range, precision, and susceptibility to overflow or underflow.
How it connects
Connects floating-point representation, mantissa, dynamic range, precision, overflow, underflow, and fine-grained quantization.

Also explained in

Compare how 6 books explain exponent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.