Layer normalization normalizes activations across features within each example, helping stabilize neural-network optimization. In Transformer blocks, it works alongside residual connections, self-attention, and feed-forward networks. section 3.1.1
Use layer normalization to understand the main structural elements of a Transformer block:
- Place it in the block architecture with residual shortcuts, self-attention, and the position-wise feed-forward network. section 3.1.1
- Distinguish it from token embeddings and positional information, which prepare representations before the first Transformer block. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

The Transformer architecture diagram shows layer normalization marked as “Add & LN” around encoder and decoder attention and MLP blocks. section 3.1.1











