Quantization reduces memory use by representing model weights or intermediate states at lower precision. The trade-off is lost precision and, depending on the hardware and implementation, lower throughput. Use it when memory is the bottleneck; if a model already fits, fine-tuning without quantization can be faster. section 7.2 section 8.1.1
In parameter-efficient fine-tuning, frozen base weights still occupy GPU memory even though only small adapters are trained. Reducing weights from 16 to 8 bits halves their storage; using 4 bits reduces it to a quarter. The freed memory can hold gradients and optimizer states. section 7.2
Four bits permit only 16 distinct values. Uniform quantization spaces these levels evenly, but many pretrained LLM weights cluster near zero. NF4 concentrates more levels in that dense region, preserving the weight distribution more faithfully than uniform 4-bit quantization. section 7.2
Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.
Compare the reconstructed weight distributions for 8-bit uniform, 4-bit uniform, and NF4 quantization. section 7.2
The same principle applies to the KV cache, whose memory grows with context length. An inference engine compresses key and value tensors before storing them, then dequantizes them when attention uses them for a new query. Smaller cache entries can support longer prompts, but conversion costs and precision loss matter. section 8.1 section 8.1.1
Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.
Compare dynamically growing cache memory with vLLM's preallocated KV block pool. Weight quantization expands the pool; cache quantization fits more tokens into it. section 8.2
To evaluate a quantization strategy, keep the model, prompt, and generation limit fixed:
- Measure static VRAM, dynamic cache growth, and throughput against an unquantized baseline. Inspect generated text as well as memory and speed. section 8.1.3
- For vLLM, compare the baseline with NF4 weights and FP8 KV cache using the experiment script's -nf4 and -fp8 flags. Record KV token capacity and throughput; the engine preallocates a fixed VRAM budget. section 8.2 section 8.2.1
- Repeat on the deployment GPU and inference engine. Kernel behavior and hardware support can change both memory overhead and speed. section 8.1 section 8.1.3
Weight versus cache quantization on a T4
For Llama-3.2-3B, 4-bit weights reduced static VRAM from 6.1 GB to 2.2 GB relative to the FP16 baseline. A separate 4-bit KV cache configuration reduced dynamic VRAM growth by about 3×. Both quantized configurations had slightly lower throughput; in the tested prompt, the weight-quantized model repeated while the baseline and cache-quantized model remained coherent. section 8.1.3





















