liveBook · Manning

Quantization

As explained in Rearchitecting LLMs

Quantization represents model parameters or intermediate states with lower-precision numerical formats to reduce storage and computation requirements.

Quantization reduces memory use by representing model weights or intermediate states at lower precision. The trade-off is lost precision and, depending on the hardware and implementation, lower throughput. Use it when memory is the bottleneck; if a model already fits, fine-tuning without quantization can be faster. section 7.2 section 8.1.1

In parameter-efficient fine-tuning, frozen base weights still occupy GPU memory even though only small adapters are trained. Reducing weights from 16 to 8 bits halves their storage; using 4 bits reduces it to a quarter. The freed memory can hold gradients and optimizer states. section 7.2

Four bits permit only 16 distinct values. Uniform quantization spaces these levels evenly, but many pretrained LLM weights cluster near zero. NF4 concentrates more levels in that dense region, preserving the weight distribution more faithfully than uniform 4-bit quantization. section 7.2

Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.

Compare the reconstructed weight distributions for 8-bit uniform, 4-bit uniform, and NF4 quantization. section 7.2

The same principle applies to the KV cache, whose memory grows with context length. An inference engine compresses key and value tensors before storing them, then dequantizes them when attention uses them for a new query. Smaller cache entries can support longer prompts, but conversion costs and precision loss matter. section 8.1 section 8.1.1

Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.

Compare dynamically growing cache memory with vLLM's preallocated KV block pool. Weight quantization expands the pool; cache quantization fits more tokens into it. section 8.2

To evaluate a quantization strategy, keep the model, prompt, and generation limit fixed:

  • Measure static VRAM, dynamic cache growth, and throughput against an unquantized baseline. Inspect generated text as well as memory and speed. section 8.1.3
  • For vLLM, compare the baseline with NF4 weights and FP8 KV cache using the experiment script's -nf4 and -fp8 flags. Record KV token capacity and throughput; the engine preallocates a fixed VRAM budget. section 8.2 section 8.2.1
  • Repeat on the deployment GPU and inference engine. Kernel behavior and hardware support can change both memory overhead and speed. section 8.1 section 8.1.3

Weight versus cache quantization on a T4

For Llama-3.2-3B, 4-bit weights reduced static VRAM from 6.1 GB to 2.2 GB relative to the FP16 baseline. A separate 4-bit KV cache configuration reduced dynamic VRAM growth by about 3×. Both quantized configurations had slightly lower throughput; in the tested prompt, the weight-quantized model repeated while the baseline and cache-quantized model remained coherent. section 8.1.3

How Rearchitecting LLMs covers it

How deep it goes
Explains NF4 level placement and the KV cache write-read cycle, then compares memory, throughput, and generated output.
Examples
Compares Llama-3.2-3B weight and KV cache configurations on T4 and L4 GPUs, and runs NF4 and FP8 vLLM variants.
What you can do after
Apply it via vLLM experiment flags and evaluate memory, token capacity, throughput, and output quality against a baseline.
How it connects
Connects quantization to parameter-efficient fine-tuning, attention, KV caching, and model pruning.

Also explained in

Compare how 22 books explain quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.