liveBook · Manning

Attention Drop

As explained in Rearchitecting LLMs

Attention Drop is a training-free technique that removes attention layers from a transformer model to reduce computational or memory requirements.

Attention Drop removes attention layers from a transformer while retaining the MLP layers. It is a training-free pruning technique intended to reduce resource requirements, but the useful pruning level depends on the task, model, and acceptable quality loss. section 8.5.1 section 8.5.2

Figure 8.11 The four pruning strategies explored in He et al. (2024). Each column shows which modules are removed from a Transformer block (crossed out) and which are preserved. The bar at the bottom represents the SDR (Speedup Degradation Ratio). A shorter bar indicates a better trade-off between speed gain and quality loss. Attention Drop achieves the lowest SDR by a wide margin, making it the most efficient strategy despite leaving MLP layers intact.

Compare attention-only pruning with block, MLP, and joint layer removal. section 8.5.1

Evaluate a pruned model against an unpruned baseline for both quality and speed. The Speedup Degradation Ratio (SDR) divides the percentage performance drop by the percentage speedup gained; a lower value indicates less quality loss for the speed gained. Measure actual throughput as well: on a 3B model, removing 11% of attention layers produced no measurable speedup, while removing 54% produced a modest gain alongside immediate generation collapse. section 8.4.2 section 8.5.1

Table 8.2 Dynamic VRAM delta across pruning configurations
Configuration
Dynamic VRAM
Vs Baseline
Baseline
109.57 MB
_
3 layers removed
96.06 MB
-12.3%
6 layers removed
78.72 MB
-28.2%
15 layers removed
58.63 MB
-46.5%

Compare dynamic VRAM measurements with the baseline across three pruning configurations. section 8.4.2

Quality depends on the evaluation

On multiple-choice benchmarks, removing eight attention layers from Llama-2-13B changed average performance from 68.2 to 68.1. In a separate 3B generation test, removing three layers preserved coherence on short prompts but produced factual inconsistencies in sustained long-form output. Static option-selection benchmarks can miss that degradation. section 8.5.1 section 8.5.2

For generation, test sustained outputs and consider a recovery step. One option is knowledge distillation with OptiPFair: a short run can balance ground-truth and teacher-output losses with alpha=0.6 and beta=0.40. This recovery is separate from the training-free pruning step. section 8.4.2 section 8.5.2

Attention Drop can be combined with quantization: tests using 4-bit AWQ reported less than 1% performance difference from the pruned, unquantized model across the tested benchmarks. Do not assume the same pruning choice suits every task; in dense retrieval, attention layers were found to be critical while MLPs showed greater redundancy. section 8.5.2

How Rearchitecting LLMs covers it

How deep it goes
Compares pruning strategies and measured trade-offs, and describes optional distillation recovery and task-specific limitations.
Examples
Compares Llama-2-13B multiple-choice scores and tests long-form generation after pruning a 3B model.
What you can do after
Evaluate/understand pruning trade-offs using baseline quality, throughput, memory, and generation checks.
How it connects
Connects attention pruning with MLP pruning, knowledge distillation, quantization, and dense retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.