Batch normalization

As explained in Inside Deep Learning: Math, Algorithms, Models

A normalization technique that standardizes activations using statistics computed across a batch.

Batch normalization (BN) standardizes activations with statistics computed from the current batch, then uses learned parameters γ and β to scale and shift them. It often accelerates convergence and improves accuracy without increasing representational capacity. section 6.2.1 section 6.2.3 section 6.2.7

Choose the normalization dimensions and PyTorch module from the tensor shape:

• For (B, D) fully connected data, compute each feature’s mean and standard deviation across the B batch items; use torch.nn.BatchNorm1d(D). section 6.2.2
• For (B, C, D), normalize each channel across B × D values; use torch.nn.BatchNorm1d(C). section 6.2.2
• For (B, C, W, H) or higher-dimensional channel data, normalize over channels with BatchNorm2d or BatchNorm3d. section 6.2.2
• During inference, use running estimates accumulated from earlier batches so predictions do not depend on other examples; batch size one is unsuitable for training BN. section 6.2.2

Adding BN to fully connected and convolutional networks

Insert nn.BatchNorm1d(n) after each nn.Linear layer in a fully connected network, or nn.BatchNorm2d(out_filters) after each nn.Conv2d layer in a CNN, before the activation. The Fashion-MNIST example uses batches of 128 and compares BN models with earlier ReLU baselines. section 6.2.3

Figure 6.1 Three versions of a network with three hidden layers and one output layer: (a) the normal approach we have learned thus far; (b) and (c) two different ways to add a normalization layer

Use the network diagrams to compare an ordinary layer stack with architectures that insert normalization layers. section 6.2.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains feature and channel statistics, tensor-shape mappings, running inference estimates, and capacity limits, but not a definitive causal theory.

Examples

Demonstrates BatchNorm1d and BatchNorm2d in Fashion-MNIST fully connected and convolutional models, with accuracy comparisons and implementation listings.

What it assumes

Assumes familiarity with batches, tensor dimensions, neural-network layers, activations, and basic PyTorch model construction.

What you can do after

Apply it via PyTorch by selecting BatchNorm1d, BatchNorm2d, or BatchNorm3d from tensor shape and inserting it into network blocks.

How it connects

Connects batch normalization to layer normalization, linear layers, convolutional layers, activation functions, and generative adversarial networks.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA neural-network layer or operation that normalizes activations using batch statistics and behaves differently in training versus evaluation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionBatch normalization is a neural-network normalization technique that standardizes layer activations using mini-batch statistics during training and stored moving statistics during inference.
• 

  liveBookmanning.com
  Sutskever's ListBatch normalization is a normalization technique that standardizes intermediate activations using statistics computed from training minibatches.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA normalization technique used in neural networks that stabilizes activations during training, commonly implemented as a normalization layer.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA normalization technique that standardizes activations within minibatches to stabilize and speed neural-network training.
• 

  liveBookmanning.com
  Deep Learning with JAXA neural-network normalization layer that normalizes activations using batch statistics during training and stored running statistics during inference.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksBatch normalization is a neural-network technique that normalizes intermediate activations using statistics computed over training batches.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsBatch normalization is a neural-network technique that normalizes intermediate activations using batch statistics during training.

Compare how 9 books explain batch normalization →

Related concepts

• layer normalizationcontrasts with · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• local response normalizationalternative to · Local response normalization scales an activation relative to neighboring activations at the same spatial location across adjacent feature maps.
• standard deviationuses · Standard deviation is the nonnegative square root of variance and measures the typical dispersion of values around their mean in the original units.
• exponential moving averageuses · An exponential moving average is a recursively updated weighted average that gives greater influence to recent observations while exponentially decreasing the influence of older ones.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.