liveBook · Manning

Layer normalization

As explained in Applied Reinforcement Learning

A normalization technique that rescales and recenters activations within each layer to stabilize neural-network computation.

Layer normalization recenters and rescales activations at each layer to stabilize their scale. In a Transformer encoder, each sub-layer’s output is written as LayerNorm(x + SubLayer(x)), combining normalization with a residual connection. section 9.6

Figure 9.8 Transformer architecture.

The architecture figure shows where layer normalization sits in the Transformer encoder. section 9.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains its stabilizing effect and placement as LayerNorm(x + SubLayer(x)), without detailing its computation.
Examples
Uses the Transformer encoder pattern LayerNorm(x + SubLayer(x)) to illustrate its placement.
What you can do after
Understand how LayerNorm stabilizes activations within a Transformer encoder.
How it connects
Connects LayerNorm with Transformer encoders, residual connections, multi-head self-attention, and feedforward networks.

Also explained in

Compare how 12 books explain layer normalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.