A floating-point value represents a number using a significand, or mantissa, and an exponent: value = mantissa × 2^exponent. Each value carries its own scale, so representable values are finely spaced at small magnitudes and more widely spaced at large magnitudes. This flexibility comes with memory and arithmetic costs that matter in neural network inference. section 1.3.1 section 2.1
A toy floating-point format
Suppose the mantissa has four fractional bits and is rounded to the nearest multiple of 1/16. The value 0.10 is approximately 1.60 × 2⁻⁴. Rounding the mantissa to 1.625 gives 1.625 × 2⁻⁴ = 0.1015625. The exponent scales the rounded mantissa, so the spacing between representable values changes with magnitude. section 2.1
Adding floating-point values involves aligning exponents; results also require normalization and rounding to fit the format. Integer arithmetic instead works on fixed-width values without those floating-point steps. In inference, the extra precision and hardware complexity may not justify their cost in memory traffic, energy, or throughput. section 1.3.1 section 1.3.2
Quantization replaces floating-point values with a bounded integer grid and a shared scale. INT8 has 256 evenly spaced values; choosing where to place that grid trades range against resolution. A grid too narrow can clamp large values, while one too coarse can round small values to zero. Calibration measures activation ranges to help make that choice. section 1.3 section 1.3.2 section 2.1
Figure 1.3 Floating point concentrates precision near zero (top), leaving large values sparsely represented. Integers use uniform spacing across your chosen range (bottom). Quantization is the act of deciding where to place that fixed grid.

Compare floating-point spacing with the uniformly spaced integer grid used in quantization. section 1.3.2
For deployment, compare the savings from lower precision against the accuracy a model can tolerate. Reducing 16-bit floating-point storage to 8-bit or 4-bit integers can reduce memory footprint and data movement roughly in proportion to bit width, but the right choice depends on the model, hardware, and acceptable quality drift. section 1.4 section 12.4

