liveBook · Manning

Resolution

As explained in Quantization and Fast Inference

Resolution is the spacing between adjacent representable numerical values.

Resolution is the spacing between adjacent representable numerical values. A finer grid represents nearby values more closely; a coarser grid sacrifices precision. Quantization therefore balances resolution against the range the grid must cover. section 1.3.2 section 8.2.1

In Q3.4 fixed-point, the resolution is 1/16, or 0.0625: stored integers are multiplied by this scale to recover real values. With a fixed bit-width, allocating more bits to the integer range makes the step size coarser. section 2.1

Resolution can also vary across a format. In [0, 0.5], E4M3 represents 49 values, compared with 32 for symmetric INT8; this denser near-zero spacing can reduce error when values cluster there. section 8.2.1

Figure 2.3 The range-versus-precision tradeoff in Q-format. For a fixed 8-bit container, every bit allocated to the integer part doubles the representable range but halves the precision (step size). Moving the binary point left (more fractional bits) gives finer resolution at the cost of a narrower range; moving it right does the reverse. There is no free lunch: the total information capacity is fixed by the bit-width.

The diagram shows how moving the binary point in an 8-bit Q-format trades finer resolution for narrower range, or broader range for coarser steps. section 2.1

How Quantization and Fast Inference covers it

How deep it goes
Defines representable-value spacing and compares fixed-point steps with near-zero density in low-precision formats.
Examples
Compares Q3.4 steps of 0.0625 with E4M3's 49 and symmetric INT8's 32 representable values in [0, 0.5].
What you can do after
Evaluate number formats by comparing their step sizes and representable-value density across the ranges that matter.
How it connects
Connects resolution to quantization, floating-point, fixed-point, INT8, and FP8 formats.

Also explained in

Compare how 6 books explain resolution →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.