Attention removal reduces a transformer model by physically removing attention sub-layers from selected blocks. Unlike fine-tuning, structural modification changes the model itself rather than only updating its weights. The trade-off is a smaller model and potentially smaller KV cache at the risk of degraded task performance and serving incompatibilities. section 1.2.5 section 12.10
In a Llama decoder block, the attention phase runs input_layernorm followed by self_attn; the MLP phase runs post_attention_layernorm followed by the MLP. Removing attention leaves the MLP phase intact. The attention normalization is removed with self_attn because it no longer feeds a component. section 8.4
A practical removal workflow:
- Calibrate and score attention layers, then select candidates with the lowest importance scores. Choose the removal count conservatively and evaluate it against the base model. section 1.2.5 section 8.4
- Delete self_attn and input_layernorm from each selected block. Replace its forward path so it runs the retained normalization and MLP with the residual connection; otherwise the original forward method will call deleted attributes. section 8.4
- Compare benchmarks and generation against the base model. As a selection check, remove the highest-scoring layers in a separate experiment and compare the degradation with low-score pruning. section 8.6
- If quality degrades, use knowledge recovery before domain-specific fine-tuning. Generation may require recovery even when selection among predefined options remains viable without it. section 1.2.5 section 8.5.2
Listing 8.8 Selecting Attention Modules to remove
def select_layers_to_drop(importance_scores, num_layers_to_drop):
sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
return [idx for idx, _ in sorted_layers[:num_layers_to_drop]]
layers_to_drop = select_layers_to_drop(attention_importance, 3)
print(f"Layers selected for removal: {sorted(layers_to_drop)}")
The selection function sorts attention importance scores in ascending order and returns the requested number of layer indices. section 8.4
Conservative removal in a 3B model
Selecting three low-scoring attention layers removes attention from blocks 21, 22, and 23 while retaining their MLPs. Removing three additional layers produced visible short-generation degradation and immediate long-form collapse in the reported test; the result is not a universal pruning limit. section 8.4
For deployment, a block without attention is no longer standard and holds no KV state. The serving engine may need custom model registration, and its cache manager may assume uniform per-layer allocation. Verify loading and actual cache savings on the target engine. Attention removal can be combined with quantization, but the best pruning target depends on the task: attention is critical in the cited dense-retrieval setting. section 12.10 section 12.5 section 8.5.2
