Fine-grained quantization reduces precision locally instead of scaling an entire tensor with one factor. It partitions activations into small groups and weights into blocks, assigning each region its own scaling factor. This limits the damage caused by outliers: a large value affects its own region without squashing precision elsewhere. section 5.6.2 section 5.7
A practical quantization workflow is:
- Partition an activation vector or weight matrix into small groups or blocks. section 5.6.2
- Find the maximum absolute value independently within each region and derive its scaling factor. section 5.6.2
- Scale and quantize each region separately, then use its corresponding factor during de-quantization. section 5.6.2
- Use local dynamic-range control to preserve precision and support higher-precision E4M3 across forward and backward operations. section 5.6.4
Why grouping matters
With tensor-wise scaling, [2.0, 3.0, 500.0] uses 500.0 as its maximum. Mapping to the integer range −128 to 127 produces [1, 1, 127], erasing the distinction between 2.0 and 3.0. Smaller groups prevent the 500.0 outlier from determining the scale for unrelated values. section 5.6.2
Figure 5.27 Fine-Grained Quantization for an activation vector. The vector is partitioned into smaller groups. Each group is scaled independently based on the maximum value within that group, preserving precision for values in groups that do not contain large outliers.

Visualize an activation vector divided into independently scaled groups, then compare precision in groups with and without large outliers. section 5.6.2
Figure 5.28 Fine-Grained Quantization for a weight matrix. The matrix is divided into smaller blocks (e.g., W₁₁, W₁₂, etc.), and each block is quantized independently with its own unique scaling factor.

Visualize a weight matrix divided into blocks, each carrying its own scaling factor. section 5.6.2
