liveBook · Manning

Block quantization

As explained in Quantization and Fast Inference

A technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.

Block quantization assigns separate scaling parameters to fixed-size groups of values instead of sharing one scale across an entire tensor. Local scales limit the influence of outliers and recover precision, especially at 4 bits, but require extra metadata and compatible kernels. section 3.5 section 8.2.2

For a weight matrix, groups commonly follow the input dimension: each scale covers a contiguous portion of a dot product. One symmetric INT4 implementation computes each group’s scale as max(abs(values)) / 7, subject to a small minimum; it divides by the scale, rounds and clamps to [-8, 7], then multiplies by the scale to reconstruct the values. Two-dimensional blocks can instead share one scale across a tile, such as 32 × 32. section 3.5

Listing 3.27 Block quantization
def block_quantize(tensor, block_size=(32, 32), bits=4):
    rows, cols = tensor.shape
    block_rows, block_cols = block_size
    q_max = (1 << (bits - 1)) - 1
    tensor_blocked = tensor.reshape(
        rows // block_rows, block_rows, cols // block_cols, block_cols
    ).permute(0, 2, 1, 3)                                         #A
    abs_max = tensor_blocked.abs().amax(dim=(2, 3))               #B
    scales = torch.clamp(abs_max / q_max, min=1e-8)
    quantized = torch.round(tensor_blocked / scales[:, :, None, None])
    return quantized.clamp(-q_max - 1, q_max), scales
 

See how a two-dimensional INT4 implementation computes one scale per tile. section 3.5

Smaller groups generally reduce reconstruction error, but require more scales and potentially more scale changes during computation. With row-major weights, grouping along the input dimension also makes weights within a group consecutive in memory, supporting coalesced loads and vectorized dequantization. section 3.5

INT4 storage and group size

For a 4096 × 4096 matrix, packed INT4 weights occupy 8 MB. With group size 128 and one FP16 scale per group, scales add 256 KB: the 8.25 MB total is 3.88× smaller than the 32 MB FP16 baseline. A format that also stores an FP16 zero-point per group has twice the metadata: 0.25 extra bits per weight at group size 128. section 3.5 section 7.3

Block scaling also applies to floating-point formats. For FP8 activations, a block of 32 with one FP32 scale adds 12.5% to the value storage. In an OPT-6.7B activation experiment, block-32 FP8 reduced reconstruction MSE by 1.4–25.9× across layers compared with per-tensor FP8. section 8.2.2

FP4 quality and storage

On OPT-6.7B, per-tensor FP4 raised WikiText-2 perplexity from the BF16 baseline of 16.26 to 26.09. FP4 with FP32 block scales recovered perplexity to 16.49 at block-16 and 16.54 at block-32, with estimated model sizes of 4.66 GB and 3.89 GB respectively. These measurements used dequantization to BF16 for matrix multiplication. section 8.3.4

For INT4 weights, group-128 is a practical starting point; measure error and metadata cost before shrinking it. For FP4, NVFP4 uses an FP8 scale per 16 values plus a tensor-wide FP32 scale, reducing scale overhead. Native FP4 throughput depends on hardware: Blackwell supports NVFP4 tensor cores, while H100 and earlier GPUs must dequantize FP4 before matrix multiplication. section 3.5 section 8.3.4

How Quantization and Fast Inference covers it

How deep it goes
Works through scale computation, reconstruction, metadata, memory layout and hardware constraints, including INT4 implementations and FP4 deployment comparisons.
Examples
Compares INT4 storage for a 4096 × 4096 matrix and FP4 perplexity and model size on OPT-6.7B.
What you can do after
Builds a symmetric block quantizer from scratch and evaluates group sizes against reconstruction error, storage and hardware support.
How it connects
Connects block quantization to per-channel quantization, post-training quantization, GPTQ and outlier sensitivity.

Also explained in

Compare how 2 books explain block quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.