liveBook · Manning

Dynamic range

As explained in Quantization and Fast Inference

Dynamic range is the span of magnitudes that a numerical representation can encode.

Dynamic range is the span of magnitudes a numerical representation can encode. In fixed-point formats, allocating more bits to the integer part expands that span but reduces precision; allocating more to the fractional part does the reverse. Calibration measures activation ranges to help place the binary point. section 2.1

When one range cannot fit

A Q0.7 format clamps values such as 5.2 to 0.99, while Q7.0 rounds tiny gradients such as 0.001 to zero. Neither suits both large and small values; floating-point moves its scale with an exponent. section 2.1

Figure 2.3 The range-versus-precision tradeoff in Q-format. For a fixed 8-bit container, every bit allocated to the integer part doubles the representable range but halves the precision (step size). Moving the binary point left (more fractional bits) gives finer resolution at the cost of a narrower range; moving it right does the reverse. There is no free lunch: the total information capacity is fixed by the bit-width.

The diagram shows how reallocating bits in an 8-bit Q-format trades range for resolution. section 2.1

How Quantization and Fast Inference covers it

How deep it goes
Explains the fixed-point range–precision tradeoff with Q-format failures, then motivates floating-point without detailing IEEE-754 encoding.
Examples
Contrasts Q0.7 saturation at 5.2 with Q7.0 rounding a 0.001 gradient to zero.
What you can do after
Apply calibration to measured activation ranges to choose a fixed-point binary point.
How it connects
Connects fixed-point formats, floating-point, quantization, and calibration.

Also explained in

Compare how 2 books explain dynamic range →

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.