liveBook · Manning

Structural modification

As explained in Rearchitecting LLMs

Attention removal is a model modification that removes attention sub-layers from selected transformer blocks.

Attention removal reduces a transformer model by physically removing attention sub-layers from selected blocks. Unlike fine-tuning, structural modification changes the model itself rather than only updating its weights. The trade-off is a smaller model and potentially smaller KV cache at the risk of degraded task performance and serving incompatibilities. section 1.2.5 section 12.10

In a Llama decoder block, the attention phase runs input_layernorm followed by self_attn; the MLP phase runs post_attention_layernorm followed by the MLP. Removing attention leaves the MLP phase intact. The attention normalization is removed with self_attn because it no longer feeds a component. section 8.4

A practical removal workflow:

  • Calibrate and score attention layers, then select candidates with the lowest importance scores. Choose the removal count conservatively and evaluate it against the base model. section 1.2.5 section 8.4
  • Delete self_attn and input_layernorm from each selected block. Replace its forward path so it runs the retained normalization and MLP with the residual connection; otherwise the original forward method will call deleted attributes. section 8.4
  • Compare benchmarks and generation against the base model. As a selection check, remove the highest-scoring layers in a separate experiment and compare the degradation with low-score pruning. section 8.6
  • If quality degrades, use knowledge recovery before domain-specific fine-tuning. Generation may require recovery even when selection among predefined options remains viable without it. section 1.2.5 section 8.5.2
Listing 8.8 Selecting Attention Modules to remove
def select_layers_to_drop(importance_scores, num_layers_to_drop):
 
    sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])
    return [idx for idx, _ in sorted_layers[:num_layers_to_drop]]
 
layers_to_drop = select_layers_to_drop(attention_importance, 3)
print(f"Layers selected for removal: {sorted(layers_to_drop)}")

The selection function sorts attention importance scores in ascending order and returns the requested number of layer indices. section 8.4

Conservative removal in a 3B model

Selecting three low-scoring attention layers removes attention from blocks 21, 22, and 23 while retaining their MLPs. Removing three additional layers produced visible short-generation degradation and immediate long-form collapse in the reported test; the result is not a universal pruning limit. section 8.4

For deployment, a block without attention is no longer standard and holds no KV state. The serving engine may need custom model registration, and its cache manager may assume uniform per-layer allocation. Verify loading and actual cache savings on the target engine. Attention removal can be combined with quantization, but the best pruning target depends on the task: attention is critical in the cited dense-retrieval setting. section 12.10 section 12.5 section 8.5.2

How Rearchitecting LLMs covers it

How deep it goes
Walks through score-based selection, module deletion, forward-path replacement, validation, and serving risks.
Examples
Selects blocks 21–23 in a 3B model and proposes inverse-pruning and combined KV-cache quantization exercises.
What you can do after
Apply it via PyTorch module modification, then compare task quality and verify serving compatibility.
How it connects
Connects attention pruning with knowledge distillation, LoRA fine-tuning, KV cache quantization, and dense retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.