Normalization modules mark key points in a decoder layer: input_layernorm precedes attention, while post_attention_layernorm follows attention and its residual connection, before the MLP. section 8.3.1 section 8.4
(0-15): 16 x LlamaDecoderLayer(
(self_attn): LlamaAttention(
...
)
(mlp): LlamaMLP(
...
)
(input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
(post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
)
Inspect the Llama and Gemma decoder-layer listings: their boundary modules are implemented as RMSNorm. section 8.3.1
PyTorch forward pre-hooks on these modules capture hidden states before attention and after its residual addition. Comparing the captured tensors supports cosine-similarity scoring of attention-layer importance. When removing attention, remove its input normalization with it; retain post_attention_layernorm and the MLP. section 8.3.1 section 8.4











