Layer normalization
Layer normalization standardizes neural-network activations across specified feature dimensions independently for each example, often using learned scale and bias parameters.
Inside Deep Learning: Math, Algorithms, Models
A normalization technique that standardizes activations within each example rather than across a batch.
- How it is used
- Stabilizes activations between network layers to support optimization, including in fully connected networks, CNNs, GANs, and RNNs.
- How deep it goes
- Works through per-example statistics, learned scale and shift, and shape-specific PyTorch construction, then compares training results.
- Examples
- Uses explicit feature-wise loops, fully connected and convolutional PyTorch models, a GAN recipe, and normalized GRUCell outputs.
- What you can do after
- Apply it via PyTorch nn.LayerNorm by matching normalized dimensions to activation shapes and accounting for pooling in CNNs.
Read how Inside Deep Learning: Math, Algorithms, Models explains layer normalization →
CUDA for LLMs
Layer normalization normalizes a neural network's activations across the features of each input, typically with learned scaling and shifting.
- How it is used
- Places normalization before transformer attention and feed-forward operations and before the final output projection.
- How deep it goes
- Derives per-row mean, variance, and affine output, then traces naive, parallel, and fused CUDA kernel strategies.
- Examples
- Works through a four-feature row and profiles a 4096 × 768 workload.
- What you can do after
- Diagnose performance with Nsight Compute by comparing occupancy, memory throughput, and runtime across LayerNorm kernels.
Build a Large Language Model (From Scratch)
A normalization technique that stabilizes neural network training by normalizing activations to have more consistent statistics across layers.
- How it is used
- Stabilizes activations throughout GPT transformer blocks and before the final output layer.
- How deep it goes
- Explains per-vector statistics, epsilon, trainable scale and shift, tensor dimensions, and a PyTorch implementation.
- Examples
- Demonstrates normalization on two six-value ReLU activation vectors and verifies their means and variances with PyTorch.
- What you can do after
- Build it from scratch as a reusable PyTorch LayerNorm module and insert it into a GPTModel.
Read how Build a Large Language Model (From Scratch) explains layer normalization →
Build AI Drug Discovery Pipelines
Layer normalization normalizes features within each individual sample and applies learned scale and shift parameters.
- How it is used
- Stabilizes hidden states inside repeated encoder blocks for protein language modeling.
- How deep it goes
- Explains per-sample feature normalization, learned scale and shift, and repeated Add & Norm placement without implementation code.
- Examples
- Uses residue representations, attention context, and a distant cysteine as the concrete architectural example.
- What you can do after
- Evaluate/understand where normalization fits in a protein-transformer encoder and why learned affine parameters remain necessary.
Read how Build AI Drug Discovery Pipelines explains layer normalization →
Learn Generative AI with PyTorch
A normalization technique that normalizes activations across features within each layer or sample.
- How it is used
- Uses layer normalization to stabilize Transformer sublayers and normalize decoder outputs before vocabulary heads.
- How deep it goes
- Works through last-dimension statistics, epsilon, learned scale and shift, and placement before sublayers.
- Examples
- Illustrates the technique in a translation encoder, GPT-2XL decoder blocks, and a music Transformer.
- What you can do after
- Builds layer normalization from scratch and applies it via PyTorch nn.LayerNorm in Transformer blocks.
Read how Learn Generative AI with PyTorch explains layer normalization →
Sutskever's List
A normalization technique that standardizes activations within each individual example across its feature dimensions.
- How it is used
- Stabilizes repeated attention and feed-forward transformations throughout Transformer encoder and decoder stacks.
- How deep it goes
- Defines per-example feature normalization and placement after sublayers, but does not derive its calculation.
- Examples
- Shows normalization in repeated encoder and decoder layers within a simplified Transformer schematic.
- What you can do after
- Evaluate how normalization supports stable information flow through repeated Transformer sublayers.
Build a Text-to-Image Generator (from Scratch)
Layer normalization normalizes the activations within each example to stabilize neural-network computation and training.
- How it is used
- Stabilizes attention and feed-forward sublayers throughout encoder and decoder transformer stacks.
- How deep it goes
- Defines per-example standardization and traces placement in transformer blocks, but does not derive the normalization calculation.
- Examples
- Uses encoder and decoder transformer layers plus a four-block vision-transformer encoder to show normalization placement.
- What you can do after
- Implement it via PyTorch-style LayerNorm calls inside attention and MLP residual blocks.
Read how Build a Text-to-Image Generator (from Scratch) explains layer normalization →
Building Specialized AI Systems
Layer normalization normalizes activations across features within each example to stabilize neural-network optimization.
- How it is used
- Stabilizes optimization as a normalization component within Transformer blocks.
- How deep it goes
- Identifies normalization as a block component and shows its placement, but does not derive its computation.
- Examples
- Uses a Transformer architecture diagram marking layer normalization alongside encoder and decoder attention and MLP blocks.
- What you can do after
- Evaluate its structural role when reading Transformer architectures and fine-tuning diagrams.
Read how Building Specialized AI Systems explains layer normalization →
Rearchitecting LLMs
Layer normalization standardizes neural-network activations within each representation to stabilize computation.
- How it is used
- Uses normalization-module boundaries to capture attention inputs and post-attention residual outputs for layer-importance scoring.
- How deep it goes
- Shows PyTorch hook placement and module removal, but gives no normalization equation or parameter-level derivation.
- Examples
- Compares normalization modules in Llama and Gemma, then removes attention's input normalization while retaining post-attention normalization.
- What you can do after
- Apply PyTorch hooks to capture tensors around attention and score layers for pruning.
Applied Reinforcement Learning
A normalization technique that rescales and recenters activations within each layer to stabilize neural-network computation.
- How it is used
- Stabilizes activation scales inside Transformer sub-layers by recentering and rescaling values at each layer.
- How deep it goes
- Explains its stabilizing effect and placement as LayerNorm(x + SubLayer(x)), without detailing its computation.
- Examples
- Uses the Transformer encoder pattern LayerNorm(x + SubLayer(x)) to illustrate its placement.
- What you can do after
- Understand how LayerNorm stabilizes activations within a Transformer encoder.
Read how Applied Reinforcement Learning explains layer normalization →
Deep Learning with R, Third Edition
LayerNormalization is a normalization layer that standardizes activations within each individual sequence or sample independently of other batch elements.
- How it is used
- Stabilizes Transformer sequence representations when batch-based normalization is unsuitable.
- How deep it goes
- Explains per-sequence mean and variance normalization, epsilon, and learned parameters without deriving implementation internals.
- Examples
- Demonstrates normalization pseudocode and its placement in an R Transformer encoder block.
- What you can do after
- Apply it via LayerNormalization when building stackable Transformer encoder blocks for sequence data.
Read how Deep Learning with R, Third Edition explains layer normalization →
Deep Learning with Python, Third Edition
- How it is used
- Normalizes representations in Transformer encoder blocks for sequence data.
- How deep it goes
- Shows sequence-wise mean and variance pseudocode and uses LayerNormalization in an encoder block.
- Examples
- Shows NumPy-like sequence-statistics pseudocode and Keras LayerNormalization calls in an encoder.
- What you can do after
- Apply it via Keras LayerNormalization in a Transformer encoder block.
Read how Deep Learning with Python, Third Edition explains layer normalization →
Also covered in
- Deep Learning with PyTorch, Second Edition
- Hugging Face in Action
- Natural Language Processing in Action, Second Edition
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Grokking AI Algorithms, Second Edition
- Machines that ThinkIn early access (MEAP)
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- batch normalizationBatch normalization is a neural-network technique that normalizes activations using statistics computed across a batch during training and stored running statistics during inference.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- residual connectionA residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.
Related concepts
- batch normalizationcontrasts with · Batch normalization is a neural-network technique that normalizes activations using statistics computed across a batch during training and stored running statistics during inference.
- transformer blockpart of · A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
- affine transformationhas part · An affine transformation is a mapping of the form x ↦ Ax + b, where A is a linear map and b is a fixed offset.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- reduction operationuses · A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
- rmsnormalternative to · RMSNorm is a neural-network normalization method that rescales activations by their root-mean-square magnitude without subtracting their mean.
Questions about layer normalization
What is layer normalization?
Layer normalization standardizes neural-network activations across specified feature dimensions independently for each example, often using learned scale and bias parameters.
Which Manning books explain layer normalization?
Inside Deep Learning: Math, Algorithms, Models, CUDA for LLMs, Build a Large Language Model (From Scratch), Build AI Drug Discovery Pipelines, Learn Generative AI with PyTorch, Sutskever's List, Build a Text-to-Image Generator (from Scratch), Building Specialized AI Systems and 4 more.
What should I know before learning layer normalization?
Manning books that teach layer normalization usually introduce attention, batch normalization, recurrent neural network and residual connection first.
What is the difference between layer normalization and batch normalization?
Layer normalization standardizes neural-network activations across specified feature dimensions independently for each example, often using learned scale and bias parameters. Batch normalization is a neural-network technique that normalizes activations using statistics computed across a batch during training and stored running statistics during inference.
What is layer normalization part of?
Layer normalization is part of transformer block. A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
What are the parts of layer normalization?
Layer normalization includes affine transformation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.











