A residual connection adds a module’s input to its transformed output, allowing information to bypass the transformation. When measuring attention-module importance, compare the input with the output after the attention result has been added back; measuring only the attention output can misidentify removable modules. section 8.5 section 8.3.1
Scoring attention modules
Hooks capture the hidden state before attention at input_layernorm and the post-attention tensor, which already includes the residual, at post_attention_layernorm. Cosine similarity between these states provides a residual-inclusive importance measure; omitting the residual can make the estimate misleading. section 8.3.1 section 8.5
(0-15): 16 x LlamaDecoderLayer(
(self_attn): LlamaAttention(
...
)
(mlp): LlamaMLP(
...
)
(input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
(post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
)
Inspect the Llama decoder-layer listing to locate the normalization components used as attention hook points. section 8.3.1

















