liveBook · Manning

Residual connection

As explained in Rearchitecting LLMs

A residual connection adds a module's input representation to its transformed output so information can bypass the transformation.

A residual connection adds a module’s input to its transformed output, allowing information to bypass the transformation. When measuring attention-module importance, compare the input with the output after the attention result has been added back; measuring only the attention output can misidentify removable modules. section 8.5 section 8.3.1

Scoring attention modules

Hooks capture the hidden state before attention at input_layernorm and the post-attention tensor, which already includes the residual, at post_attention_layernorm. Cosine similarity between these states provides a residual-inclusive importance measure; omitting the residual can make the estimate misleading. section 8.3.1 section 8.5

(0-15): 16 x LlamaDecoderLayer(
    (self_attn): LlamaAttention(
       ...
    )
    (mlp): LlamaMLP(
        ...
    )
    (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
    (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
)

Inspect the Llama decoder-layer listing to locate the normalization components used as attention hook points. section 8.3.1

How Rearchitecting LLMs covers it

How deep it goes
Explains hook locations and why excluding the residual can distort attention importance scores.
Examples
Uses Llama and Gemma decoder-layer hook points and contrasts importance measurement with versus without the residual.
What you can do after
Evaluate attention-module importance by comparing input and output cosine similarity with the residual included.
How it connects
Connects residual connections to cosine similarity, attention modules, and Transformer blocks.

Also explained in

Compare how 18 books explain residual connection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.