liveBook · Manning

Layer normalization

As explained in Build AI Drug Discovery Pipelines

Layer normalization normalizes features within each individual sample and applies learned scale and shift parameters.

Layer normalization stabilizes transformer hidden states by computing statistics separately for each sample and normalizing across each token’s feature dimensions. Learned gain and bias then rescale and shift the result, so later activations are not required to retain exactly mean 0 and variance 1. section 12.3.2

In an encoder-only protein transformer, layer normalization:

  • Follows a residual addition that combines the original residue representation with attention output. section 12.3.2
  • Helps stabilize hidden-state dynamics while supporting deep stacks of encoder layers. section 12.3.2
  • Repeats after each major encoder component alongside the residual connection. section 12.3.2

Residue-context update

After attention pulls residue 1 toward contextual information from a distant cysteine, the residual path preserves residue 1’s original identity before layer normalization prepares the combined representation for the next component. section 12.3.2

Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

Use the architecture diagram to locate layer normalization within the repeated encoder stack that processes protein sequences in parallel. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains per-sample feature normalization, learned scale and shift, and repeated Add & Norm placement without implementation code.
Examples
Uses residue representations, attention context, and a distant cysteine as the concrete architectural example.
What it assumes
Assumes familiarity with token representations and attention, while explaining residual addition and normalization behavior directly.
What you can do after
Evaluate/understand where normalization fits in a protein-transformer encoder and why learned affine parameters remain necessary.
How it connects
Connects layer normalization to residual connections, self-attention, encoder-only transformers, and vanishing gradients.

Also explained in

Compare how 12 books explain layer normalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.