Attention pruning removes attention modules from a neural model to reduce memory requirements. The engineering trade-off is model quality: removing more modules can save more memory, but some tasks and generation behaviors may degrade before others. section 8.4.2
A practical selection strategy ranks attention modules by importance and removes low-scoring candidates. In Llama-3.2-3B and Llama-3.1-8B, low scores on the Cosmopedia dataset concentrate between 60% and 80% of network depth, while the first and last layers remain more active. This pattern is an observation for these models and data, not a universal removal rule. section 8.3.2
Table 8.1 Attention modules with the lowest importance score in Llama-3.2-3B and Llama-3.1-8B
|
Rank
|
Llama-3.2-3B
|
Score
|
Llama-3.1-8B
|
Score
|
|
1
|
Layer 21
|
0.009053
|
Layer 24
|
0.004922
|
|
2
|
Layer 22
|
0.010405
|
Layer 23
|
0.007695
|
|
3
|
Layer 23
|
0.012417
|
Layer 22
|
0.008828
|
|
4
|
Layer 20
|
0.012783
|
Layer 25
|
0.009258
|
|
5
|
Layer 18
|
0.014297
|
Layer 26
|
0.010508
|
Compare the five lowest-scoring attention modules in the two Llama models. section 8.3.2
To assess a pruning configuration:
- Score and rank the attention modules, then select low-scoring layers for removal. As a control, compare against removing the same number of high-scoring layers. section 8.3.2 section 8.6
- Measure static VRAM, dynamic VRAM delta, and throughput against the unpruned baseline. Check the reliability of runtime conditions before interpreting speed differences. section 8.4.2 section 8.6
- Run the same benchmarks and text-generation tests on both models. Examine task-specific degradation rather than relying on a single aggregate result. section 8.4.2 section 8.6
Removing attention modules from Llama-3.2-3B
Removing three modules removes roughly 75 million parameters, equivalent to 150 MB in FP16. Against the baseline, four of five benchmarks changed little, but Lambada fell 9.2%; with six modules removed, Lambada fell 19.1%. Removing 15 caused scores to collapse across all five tasks. Dynamic VRAM fell from 109.57 MB at baseline to 96.06 MB, 78.72 MB, and 58.63 MB for the three, six, and 15-module configurations, respectively. Throughput measurements on a shared Colab T4 are directional, not definitive. section 8.4.2
Attention pruning can also be tested alongside 4-bit KV cache quantization. Compare memory, throughput, and generated text against the full model with the same quantization setup. For a six-module-pruned model exhibiting repetition loops, knowledge distillation offers a recovery experiment; its results still need evaluation. section 8.6
