Batch normalization

As explained in Sutskever's List

Batch normalization is a normalization technique that standardizes intermediate activations using statistics computed from training minibatches.

Batch normalization stabilizes layer inputs during training by normalizing activations with the mean and variance computed from each minibatch. A minibatch is the subset of training data processed in one forward and backward pass. More consistent activation distributions can reduce internal covariate shift, support higher learning rates, reduce sensitivity to weight initialization, and speed convergence. section 4.4.1

In pre-activation ResNets, batch normalization and ReLU are placed before the convolutional layers, leaving the skip connection unchanged. This design had slightly higher training error but lower test error than the original post-activation design; the difference was attributed to a regularizing effect from batch normalization at each convolution’s input. section 3.4.1

Sequence-wise normalization in DS2

Deep Speech 2 adapted batch normalization along the temporal dimension. This sequence-wise form helped stabilize recurrent-network training and improve generalization; alongside SortaGrad, it supported training deeper networks with greater stability. section 4.4.1 section 4.4.2

Figure 3.5 Left: the original residual unit shows two “weight” layers (that is, convolutional layers), with batch normalization (BN) and ReLU activations between them, followed by an additional step that merges with the identity connection. Right: the revised residual unit reorganizes the operations by placing BN and ReLU before the weight layers, simplifying the pathway. The gray arrows highlight the direct paths (that is, skip connections) that support the smooth forward and backward flow of information through the residual connections. Used with permission granted by the lead author (Kaiming He).

Compare the original residual unit with the pre-activation layout, including where batch normalization sits relative to the convolutional layers and skip connection. section 3.4.1

How Sutskever's List covers it

How deep it goes

Defines minibatch mean-and-variance normalization and describes its temporal adaptation, without giving equations or implementation details.

Examples

Contrasts post-activation and pre-activation residual units and describes DS2's temporal adaptation.

What you can do after

Understand how minibatch statistics stabilize layer inputs and why DS2 adapts normalization to recurrent sequences.

How it connects

Links batch normalization to ResNet pre-activation, recurrent neural networks, and internal covariate shift.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA normalization technique that standardizes activations using statistics computed across a batch.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA neural-network layer or operation that normalizes activations using batch statistics and behaves differently in training versus evaluation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionBatch normalization is a neural-network normalization technique that standardizes layer activations using mini-batch statistics during training and stored moving statistics during inference.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA normalization technique used in neural networks that stabilizes activations during training, commonly implemented as a normalization layer.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA normalization technique that standardizes activations within minibatches to stabilize and speed neural-network training.
• 

  liveBookmanning.com
  Deep Learning with JAXA neural-network normalization layer that normalizes activations using batch statistics during training and stored running statistics during inference.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksBatch normalization is a neural-network technique that normalizes intermediate activations using statistics computed over training batches.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsBatch normalization is a neural-network technique that normalizes intermediate activations using batch statistics during training.

Compare how 9 books explain batch normalization →

Related concepts

• layer normalizationcontrasts with · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• local response normalizationalternative to · Local response normalization scales an activation relative to neighboring activations at the same spatial location across adjacent feature maps.
• standard deviationuses · Standard deviation is the nonnegative square root of variance and measures the typical dispersion of values around their mean in the original units.
• exponential moving averageuses · An exponential moving average is a recursively updated weighted average that gives greater influence to recent observations while exponentially decreasing the influence of older ones.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.