liveBook · Manning

Layer normalization

As explained in Rearchitecting LLMs

Layer normalization standardizes neural-network activations within each representation to stabilize computation.

Normalization modules mark key points in a decoder layer: input_layernorm precedes attention, while post_attention_layernorm follows attention and its residual connection, before the MLP. section 8.3.1 section 8.4

(0-15): 16 x LlamaDecoderLayer(
    (self_attn): LlamaAttention(
       ...
    )
    (mlp): LlamaMLP(
        ...
    )
    (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
    (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)
)

Inspect the Llama and Gemma decoder-layer listings: their boundary modules are implemented as RMSNorm. section 8.3.1

PyTorch forward pre-hooks on these modules capture hidden states before attention and after its residual addition. Comparing the captured tensors supports cosine-similarity scoring of attention-layer importance. When removing attention, remove its input normalization with it; retain post_attention_layernorm and the MLP. section 8.3.1 section 8.4

How Rearchitecting LLMs covers it

How deep it goes
Shows PyTorch hook placement and module removal, but gives no normalization equation or parameter-level derivation.
Examples
Compares normalization modules in Llama and Gemma, then removes attention's input normalization while retaining post-attention normalization.
What you can do after
Apply PyTorch hooks to capture tensors around attention and score layers for pruning.
How it connects
Ties normalization to self-attention, residual connections, MLPs, and cosine-similarity importance scoring.

Also explained in

Compare how 12 books explain layer normalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.