LLM inference uses GPU memory for model parameters and for the KV cache, which grows with context length. Memory-bound workloads can also spend time waiting for data to move between memory and the GPU. section 4.1.2 section 8.1
Pruning and quantization target different parts of the memory footprint:
- Removing complete Transformer blocks reduces model size and eliminates those blocks’ contributions to KV-cache growth and memory traffic. section 4.1.2
- Quantizing the KV cache reduces dynamic memory use, allowing larger prompts to be processed without running out of GPU memory. section 8.1
- Attention pruning can reduce dynamic VRAM, but more pruning can sharply degrade benchmark scores and generation quality. section 8.4.2
Table 8.2 Dynamic VRAM delta across pruning configurations
|
Configuration
|
Dynamic VRAM
|
Vs Baseline
|
|
Baseline
|
109.57 MB
|
_
|
|
3 layers removed
|
96.06 MB
|
-12.3%
|
|
6 layers removed
|
78.72 MB
|
-28.2%
|
|
15 layers removed
|
58.63 MB
|
-46.5%
|
Compare dynamic VRAM against the baseline across three attention-pruning configurations. section 8.4.2












