Layer normalization
Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
Inside Deep Learning: Math, Algorithms, Models
A normalization technique that standardizes activations within each example rather than across a batch.
- How it is used
- Deploys normalization between neural layers to support faster, more reliable training, especially with small batches, recurrent models, and GANs.
- How deep it goes
- Explains per-example statistics, learned affine parameters, PyTorch shapes, and CNN bookkeeping, but defers deeper reasons for LN’s flexibility.
- Examples
- Uses fully connected networks, convolutional networks with pooling, GAN generator and discriminator blocks, and stacked GRUCell text models.
- What you can do after
- Apply it via PyTorch’s nn.LayerNorm by matching normalized dimensions to each tensor shape and inserting it before activations.
Read how Inside Deep Learning: Math, Algorithms, Models explains layer normalization →
Build a Large Language Model (From Scratch)
A normalization technique that stabilizes neural network training by normalizing activations to have more consistent statistics across layers.
- How it is used
- Stabilizes activations throughout GPT transformer blocks and before the final output layer.
- How deep it goes
- Explains per-vector statistics, epsilon, trainable scale and shift, tensor dimensions, and a PyTorch implementation.
- Examples
- Demonstrates normalization on two six-value ReLU activation vectors and verifies their means and variances with PyTorch.
- What you can do after
- Build it from scratch as a reusable PyTorch LayerNorm module and insert it into a GPTModel.
Read how Build a Large Language Model (From Scratch) explains layer normalization →
Learn Generative AI with PyTorch
A normalization technique that normalizes activations across features within each layer or sample.
- How it is used
- Stabilizes Transformer training within attention and feed-forward sublayers and normalizes representations before an output head.
- How deep it goes
- Defines zero-mean, unit-standard-deviation normalization and implements feature-axis statistics, epsilon, and learned scale and shift without a mathematical derivation.
- Examples
- Uses a from-scratch PyTorch class, a Transformer encoder layer, a GPT-2XL decoder block, and a music Transformer.
- What you can do after
- Build it from scratch in PyTorch or apply it via `nn.LayerNorm` within Transformer decoder blocks.
Read how Learn Generative AI with PyTorch explains layer normalization →
Sutskever's List
A normalization technique that standardizes activations within each individual example across its feature dimensions.
- How it is used
- Stabilizes repeated attention and feed-forward transformations throughout Transformer encoder and decoder stacks.
- How deep it goes
- Defines per-example feature normalization and placement after sublayers, but does not derive its calculation.
- Examples
- Shows normalization in repeated encoder and decoder layers within a simplified Transformer schematic.
- What you can do after
- Evaluate how normalization supports stable information flow through repeated Transformer sublayers.
Build a Text-to-Image Generator (from Scratch)
Layer normalization normalizes the activations within each example to stabilize neural-network computation and training.
- How it is used
- Stabilizes attention and feed-forward sublayers throughout encoder and decoder transformer stacks.
- How deep it goes
- Defines per-example standardization and traces placement in transformer blocks, but does not derive the normalization calculation.
- Examples
- Uses encoder and decoder transformer layers plus a four-block vision-transformer encoder to show normalization placement.
- What you can do after
- Implement it via PyTorch-style LayerNorm calls inside attention and MLP residual blocks.
Read how Build a Text-to-Image Generator (from Scratch) explains layer normalization →
Deep Learning with R, Third Edition
LayerNormalization is a normalization layer that standardizes activations within each individual sequence or sample independently of other batch elements.
- How it is used
- Stabilizes Transformer sequence representations when batch-based normalization is unsuitable.
- How deep it goes
- Explains per-sequence mean and variance normalization, epsilon, and learned parameters without deriving implementation internals.
- Examples
- Demonstrates normalization pseudocode and its placement in an R Transformer encoder block.
- What you can do after
- Apply it via LayerNormalization when building stackable Transformer encoder blocks for sequence data.
Read how Deep Learning with R, Third Edition explains layer normalization →
Deep Learning with Python, Third Edition
- How it is used
- Uses layer normalization in Transformer encoders to normalize sequence data independently across examples.
- How deep it goes
- Defines sequence-wise normalization, contrasts it with batch normalization, and shows pseudocode and encoder-layer placement.
- Examples
- Demonstrates normalization with NumPy-like pseudocode and shows Keras LayerNormalization layers in an encoder block.
- What you can do after
- Apply it via Keras LayerNormalization layers in a Transformer encoder block.
Read how Deep Learning with Python, Third Edition explains layer normalization →
Related concepts
- reduction operationuses · A reduction operation combines multiple input values or partial results according to a combining rule to produce fewer results, often a single aggregate.
- batch normalizationcontrasts with · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
- transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
- FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.
- softmaxcontrasts with · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.