Layer normalization standardizes each example’s activations across its features, producing zero mean and unit standard deviation. Unlike batch-dependent normalization, it treats each input independently, helping stabilize neural-network training. section 2.1.2 section 2.3.2
In a transformer, place normalization inside encoder and decoder sublayers alongside residual connections. Self-attention and feed-forward transformations operate on normalized representations, while residual additions preserve information flow. section 2.1.2 section 3.3.4
A transformer block applies the mechanism in a repeatable sequence:
- Normalize the input before the self-attention transformation. section 3.3.4
- Add the attention output back to the original representation through a residual connection. section 3.3.4
- Normalize the updated representation before the feed-forward network, then add the feed-forward output residually. section 3.3.4
Encoder-block implementation
A PyTorch-style block defines two normalization layers: one before multi-head attention and one before the MLP. Its forward pass computes attention from normalized input, adds that result to x, normalizes the updated x, computes the MLP output, and adds it again.
