Layer normalization standardizes activations within each example across its feature dimensions. In Transformer stacks, it stabilizes the repeated sequence of attention and feed-forward transformations, helping residual connections carry representations through many layers. section 5.1
Inside a Transformer layer
An encoder combines multihead self-attention with a feed-forward network: attention mixes information across tokens, while the feed-forward network transforms each position. Residual addition and layer normalization follow these sublayers. Decoder layers apply the same treatment after masked self-attention, encoder-decoder attention, and the feed-forward network. section 5.1
Figure 5.1 A simplified schematic of the Transformer architecture. The model consists of an encoder (left) and a decoder (right), each comprising repeated stacks of sublayers, including multihead attention mechanisms, feed-forward neural networks, and postprocessing steps such as residual connections and layer normalization. Positional encodings (depicted as circled crosses) are added to the input embeddings to explicitly represent token order. The diagram emphasizes the cross-attention operation (dashed box labeled “Cross-attention”), in which the decoder’s queries (Q) interact with the encoder’s keys (K) and values (V). During cross-attention, each decoding step computes attention scores to determine the relevance of encoded input representations, producing a context vector that guides token generation. This selective alignment enables the decoder to integrate contextual information from the encoder, improving the accuracy and coherence of the output sequence.
Use the Transformer schematic to locate layer normalization among repeated encoder and decoder sublayers, alongside attention, feed-forward networks, residual connections, and positional encodings. section 5.1
