Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
Layer normalization normalizes sequence data independently for each sequence, rather than collecting statistics across a batch. That makes it a practical alternative to batch normalization in a Transformer encoder. section 15.3.2
A Transformer encoder
An encoder block applies LayerNormalization after self-attention and again after its feedforward network. The layer preserves the input and output shape, allowing encoder blocks to be stacked. section 15.3.2
How Deep Learning with Python, Third Edition covers it
How deep it goes
Defines sequence-wise normalization, contrasts it with batch normalization, and shows pseudocode and encoder-layer placement.
Examples
Demonstrates normalization with NumPy-like pseudocode and shows Keras LayerNormalization layers in an encoder block.
What you can do after
Apply it via Keras LayerNormalization layers in a Transformer encoder block.
How it connects
Connects layer normalization to batch normalization, Transformer encoders, self-attention, and residual connections.
Build a Large Language Model (From Scratch)A normalization technique that stabilizes neural network training by normalizing activations to have more consistent statistics across layers.
Deep Learning with R, Third EditionLayerNormalization is a normalization layer that standardizes activations within each individual sequence or sample independently of other batch elements.
reduction operationuses · A reduction operation combines multiple input values or partial results according to a combining rule to produce fewer results, often a single aggregate.
batch normalizationcontrasts with · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.
softmaxcontrasts with · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.
