Block quantization assigns separate scaling parameters to fixed-size groups of values instead of sharing one scale across an entire tensor. Local scales limit the influence of outliers and recover precision, especially at 4 bits, but require extra metadata and compatible kernels. section 3.5 section 8.2.2
For a weight matrix, groups commonly follow the input dimension: each scale covers a contiguous portion of a dot product. One symmetric INT4 implementation computes each group’s scale as max(abs(values)) / 7, subject to a small minimum; it divides by the scale, rounds and clamps to [-8, 7], then multiplies by the scale to reconstruct the values. Two-dimensional blocks can instead share one scale across a tile, such as 32 × 32. section 3.5
Listing 3.27 Block quantization
def block_quantize(tensor, block_size=(32, 32), bits=4):
rows, cols = tensor.shape
block_rows, block_cols = block_size
q_max = (1 << (bits - 1)) - 1
tensor_blocked = tensor.reshape(
rows // block_rows, block_rows, cols // block_cols, block_cols
).permute(0, 2, 1, 3) #A
abs_max = tensor_blocked.abs().amax(dim=(2, 3)) #B
scales = torch.clamp(abs_max / q_max, min=1e-8)
quantized = torch.round(tensor_blocked / scales[:, :, None, None])
return quantized.clamp(-q_max - 1, q_max), scales
See how a two-dimensional INT4 implementation computes one scale per tile. section 3.5
Smaller groups generally reduce reconstruction error, but require more scales and potentially more scale changes during computation. With row-major weights, grouping along the input dimension also makes weights within a group consecutive in memory, supporting coalesced loads and vectorized dequantization. section 3.5
INT4 storage and group size
For a 4096 × 4096 matrix, packed INT4 weights occupy 8 MB. With group size 128 and one FP16 scale per group, scales add 256 KB: the 8.25 MB total is 3.88× smaller than the 32 MB FP16 baseline. A format that also stores an FP16 zero-point per group has twice the metadata: 0.25 extra bits per weight at group size 128. section 3.5 section 7.3
Block scaling also applies to floating-point formats. For FP8 activations, a block of 32 with one FP32 scale adds 12.5% to the value storage. In an OPT-6.7B activation experiment, block-32 FP8 reduced reconstruction MSE by 1.4–25.9× across layers compared with per-tensor FP8. section 8.2.2
FP4 quality and storage
On OPT-6.7B, per-tensor FP4 raised WikiText-2 perplexity from the BF16 baseline of 16.26 to 26.09. FP4 with FP32 block scales recovered perplexity to 16.49 at block-16 and 16.54 at block-32, with estimated model sizes of 4.66 GB and 3.89 GB respectively. These measurements used dequantization to BF16 for matrix multiplication. section 8.3.4
For INT4 weights, group-128 is a practical starting point; measure error and metadata cost before shrinking it. For FP4, NVFP4 uses an FP8 scale per 16 values plus a tensor-wide FP32 scale, reducing scale overhead. Native FP4 throughput depends on hardware: Blackwell supports NVFP4 tensor cores, while H100 and earlier GPUs must dequantize FP4 before matrix multiplication. section 3.5 section 8.3.4

