Layer normalization stabilizes transformer hidden states by computing statistics separately for each sample and normalizing across each token’s feature dimensions. Learned gain and bias then rescale and shift the result, so later activations are not required to retain exactly mean 0 and variance 1. section 12.3.2
In an encoder-only protein transformer, layer normalization:
- Follows a residual addition that combines the original residue representation with attention output. section 12.3.2
- Helps stabilize hidden-state dynamics while supporting deep stacks of encoder layers. section 12.3.2
- Repeats after each major encoder component alongside the residual connection. section 12.3.2
Residue-context update
After attention pulls residue 1 toward contextual information from a distant cysteine, the residual path preserves residue 1’s original identity before layer normalization prepares the combined representation for the next component. section 12.3.2
Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

Use the architecture diagram to locate layer normalization within the repeated encoder stack that processes protein sequences in parallel. section 12.3.2











