Attention Drop removes attention layers from a transformer while retaining the MLP layers. It is a training-free pruning technique intended to reduce resource requirements, but the useful pruning level depends on the task, model, and acceptable quality loss. section 8.5.1 section 8.5.2
Figure 8.11 The four pruning strategies explored in He et al. (2024). Each column shows which modules are removed from a Transformer block (crossed out) and which are preserved. The bar at the bottom represents the SDR (Speedup Degradation Ratio). A shorter bar indicates a better trade-off between speed gain and quality loss. Attention Drop achieves the lowest SDR by a wide margin, making it the most efficient strategy despite leaving MLP layers intact.
Compare attention-only pruning with block, MLP, and joint layer removal. section 8.5.1
Evaluate a pruned model against an unpruned baseline for both quality and speed. The Speedup Degradation Ratio (SDR) divides the percentage performance drop by the percentage speedup gained; a lower value indicates less quality loss for the speed gained. Measure actual throughput as well: on a 3B model, removing 11% of attention layers produced no measurable speedup, while removing 54% produced a modest gain alongside immediate generation collapse. section 8.4.2 section 8.5.1
Table 8.2 Dynamic VRAM delta across pruning configurations
|
Configuration
|
Dynamic VRAM
|
Vs Baseline
|
|
Baseline
|
109.57 MB
|
_
|
|
3 layers removed
|
96.06 MB
|
-12.3%
|
|
6 layers removed
|
78.72 MB
|
-28.2%
|
|
15 layers removed
|
58.63 MB
|
-46.5%
|
Compare dynamic VRAM measurements with the baseline across three pruning configurations. section 8.4.2
Quality depends on the evaluation
On multiple-choice benchmarks, removing eight attention layers from Llama-2-13B changed average performance from 68.2 to 68.1. In a separate 3B generation test, removing three layers preserved coherence on short prompts but produced factual inconsistencies in sustained long-form output. Static option-selection benchmarks can miss that degradation. section 8.5.1 section 8.5.2
For generation, test sustained outputs and consider a recovery step. One option is knowledge distillation with OptiPFair: a short run can balance ground-truth and teacher-output losses with alpha=0.6 and beta=0.40. This recovery is separate from the training-free pruning step. section 8.4.2 section 8.5.2
Attention Drop can be combined with quantization: tests using 4-bit AWQ reported less than 1% performance difference from the pruned, unquantized model across the tested benchmarks. Do not assume the same pruning choice suits every task; in dense retrieval, attention layers were found to be critical while MLPs showed greater redundancy. section 8.5.2
