Layer normalization

As explained in Build a Large Language Model (From Scratch)

A normalization technique that stabilizes neural network training by normalizing activations to have more consistent statistics across layers.

Layer normalization stabilizes neural-network training by normalizing each activation vector to a consistent mean and variance. It aims for mean 0 and unit variance, helping optimization converge toward effective weights. In GPT and transformer architectures, LayerNorm appears around multi-head attention and before the final output layer. section 4.2 section 4.8

For a tensor shaped like a batch of activation vectors, compute statistics across the last dimension with keepdim=True. Subtract the per-vector mean, divide by the square root of variance plus a small epsilon, then restore learned scale and shift parameters. The implementation uses unbiased=False for the variance and initializes scale to ones and shift to zeros. section 4.2 section 4.2

Normalizing a six-value activation vector

A Linear(5, 6) layer followed by ReLU produces two six-value activation vectors with nonzero means and small variances. Normalizing across each row produces approximately zero means and unit variances. The LayerNorm class then applies learned scale and shift, yielding variances close to one because epsilon is included in the denominator. section 4.2 section 4.2

Figure 4.5 An illustration of layer normalization where the six outputs of the layer, also called activations, are normalized such that they have a 0 mean and a variance of 1.

Use the illustration to follow how six layer outputs are transformed to mean 0 and variance 1. section 4.2

In the GPT pipeline, token and positional embeddings flow through repeated transformer blocks. Each block contains attention, a feed-forward network, dropout, and LayerNorm; the completed model applies a final LayerNorm before mapping hidden states to vocabulary logits. The same module also remains part of the architecture when GPT is adapted for classification. section 4.6 section 6.5

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains per-vector statistics, epsilon, trainable scale and shift, tensor dimensions, and a PyTorch implementation.

Examples

Demonstrates normalization on two six-value ReLU activation vectors and verifies their means and variances with PyTorch.

What it assumes

Assumes basic neural-network tensors and activations while treating deep gradient mathematics as unnecessary.

What you can do after

Build it from scratch as a reusable PyTorch LayerNorm module and insert it into a GPTModel.

How it connects

Connects layer normalization to transformer blocks, multi-head attention, feed-forward networks, dropout, embeddings, and GPT classification heads.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA normalization technique that standardizes activations within each example rather than across a batch.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA normalization technique that normalizes activations across features within each layer or sample.
• 

  liveBookmanning.com
  Sutskever's ListA normalization technique that standardizes activations within each individual example across its feature dimensions.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)Layer normalization normalizes the activations within each example to stabilize neural-network computation and training.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionLayerNormalization is a normalization layer that standardizes activations within each individual sequence or sample independently of other batch elements.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition

Compare how 7 books explain layer normalization →

Related concepts

• reduction operationuses · A reduction operation combines multiple input values or partial results according to a combining rule to produce fewer aggregate results, often a single result.
• batch normalizationcontrasts with · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
• transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
• softmaxcontrasts with · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
• FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.