Block quantization
Block quantization partitions numerical values into groups and assigns quantization parameters, such as scale factors, separately to each group.
Quantization and Fast Inference
A technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.
- How it is used
- Uses local scales to preserve precision when quantizing weights or activations at low bit widths.
- How deep it goes
- Works through scale computation, reconstruction, metadata, memory layout and hardware constraints, including INT4 implementations and FP4 deployment comparisons.
- Examples
- Compares INT4 storage for a 4096 × 4096 matrix and FP4 perplexity and model size on OPT-6.7B.
- What you can do after
- Builds a symmetric block quantizer from scratch and evaluates group sizes against reconstruction error, storage and hardware support.
Read how Quantization and Fast Inference explains block quantization →
CUDA for LLMs
Block-wise quantization assigns a separate scale factor to each spatial block of a tensor.
- How it is used
- Uses local scales to quantize spatial regions with differing value ranges.
- How deep it goes
- Walks through CUDA quantization and dequantization kernels but does not provide a block-specific scale-computation kernel.
- Examples
- Illustrates spatial granularity with CNN feature maps and uses block quantization in Flash Attention 3’s FP8 path.
- What you can do after
- Evaluate block granularity against accuracy, scale overhead, and measured performance.
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- channel dimensionA channel dimension is a dimension of image or tensor data whose positions distinguish separate color components or feature maps.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
Related concepts
- quantizationkind of · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- post-training quantizationuses · Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
- quantization scaleuses · A quantization scale is a multiplicative factor relating quantized values or steps to the numerical values they represent.
- int4 quantizationused by · INT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
- NVFP4used by · NVFP4 is NVIDIA’s 4-bit floating-point format, which represents values in blocks using shared scaling factors.
Questions about block quantization
What is block quantization?
Block quantization partitions numerical values into groups and assigns quantization parameters, such as scale factors, separately to each group.
Which Manning books explain block quantization?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning block quantization?
Manning books that teach block quantization usually introduce accuracy, activation, channel dimension and matrix-matrix multiplication first.
What is block quantization a kind of?
Block quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

