liveBook · Manning

Block quantization

Block quantization partitions numerical values into groups and assigns quantization parameters, such as scale factors, separately to each group.

How 2 Manning books explain it

Quantization and Fast Inference

A technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.

How it is used
Uses local scales to preserve precision when quantizing weights or activations at low bit widths.
How deep it goes
Works through scale computation, reconstruction, metadata, memory layout and hardware constraints, including INT4 implementations and FP4 deployment comparisons.
Examples
Compares INT4 storage for a 4096 × 4096 matrix and FP4 perplexity and model size on OPT-6.7B.
What you can do after
Builds a symmetric block quantizer from scratch and evaluates group sizes against reconstruction error, storage and hardware support.

Read how Quantization and Fast Inference explains block quantization →

CUDA for LLMs

Block-wise quantization assigns a separate scale factor to each spatial block of a tensor.

How it is used
Uses local scales to quantize spatial regions with differing value ranges.
How deep it goes
Walks through CUDA quantization and dequantization kernels but does not provide a block-specific scale-computation kernel.
Examples
Illustrates spatial granularity with CNN feature maps and uses block quantization in Flash Attention 3’s FP8 path.
What you can do after
Evaluate block granularity against accuracy, scale overhead, and measured performance.

Read how CUDA for LLMs explains block quantization →

What to know first

Related concepts

Questions about block quantization

What is block quantization?

Block quantization partitions numerical values into groups and assigns quantization parameters, such as scale factors, separately to each group.

Which Manning books explain block quantization?

Quantization and Fast Inference and CUDA for LLMs.

What should I know before learning block quantization?

Manning books that teach block quantization usually introduce accuracy, activation, channel dimension and matrix-matrix multiplication first.

What is block quantization a kind of?

Block quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.