Batch normalization

As explained in Evolutionary Deep Learning: Genetic algorithms and neural networks

Batch normalization is a neural-network technique that normalizes intermediate activations using statistics computed over training batches.

Batch normalization is a neural-network layer that normalizes intermediate activations, helping convolutional models generalize and reducing memorization. section 7.1.1 section 8.1.4

For each convolutional feature map, it calculates the mean and variance, subtracts the mean, and divides by the square root of the variance. Values therefore remain centered near zero, reducing vanishing and exploding gradient problems. section 8.1.4

Adding the layer to a CNN

A Keras CNN places BatchNormalization between convolution and max pooling. A comparable autoencoder adds the layer throughout both encoder and decoder, alongside dropout; excessive use can worsen results and require manual architecture tuning. section 7.1.1 section 8.1.5

Listing 7.6 EDL_7_1_Keras_CNN.ipynb: Adding pooling

model = models.Sequential()
model.add(layers.Conv2D(64, (3, 3), activation='relu', 
 padding="same", input_shape=(28, 28, 1)))              #1
model.add(layers.BatchNormalization())                    #2
model.add(layers.MaxPooling2D((2, 2), padding="same"))    #3
model.add(layers.Flatten())
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(10))
 
model.summary()
...
=================================================================
Total params: 1,607,946
Trainable params: 1,607,818                               #4
Non-trainable params: 128
_________________________________________________________________

Inspect a runnable Keras CNN showing BatchNormalization placement, pooling, and parameter counts. section 7.1.1

Figure 8.7 The BatchNormalization process

Use the figure to inspect per-feature-map means, variances, and normalized values. section 8.1.4

How Evolutionary Deep Learning: Genetic algorithms and neural networks covers it

How deep it goes

Explains feature-map statistics and the normalization equation, then demonstrates layer placement without deriving training internals.

Examples

Demonstrates Keras CNN and autoencoder architectures, feature-map normalization, training curves, parameter summaries, and a deliberately overused configuration.

What it assumes

Assumes basic neural-network layers and training while teaching normalization, gradient problems, and practical layer insertion.

What you can do after

Apply it via Keras BatchNormalization layers in CNNs and autoencoders, then inspect validation behavior and tune excessive use.

How it connects

Connects batch normalization to convolutional neural networks, autoencoders, dropout, pooling, generalization, and vanishing or exploding gradients.

liveBookmanning.com

Read Evolutionary Deep Learning: Genetic algorithms and neural networks on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA normalization technique that standardizes activations using statistics computed across a batch.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA neural-network layer or operation that normalizes activations using batch statistics and behaves differently in training versus evaluation.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionBatch normalization is a neural-network normalization technique that standardizes layer activations using mini-batch statistics during training and stored moving statistics during inference.
• 

  liveBookmanning.com
  Sutskever's ListBatch normalization is a normalization technique that standardizes intermediate activations using statistics computed from training minibatches.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA normalization technique used in neural networks that stabilizes activations during training, commonly implemented as a normalization layer.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA normalization technique that standardizes activations within minibatches to stabilize and speed neural-network training.
• 

  liveBookmanning.com
  Deep Learning with JAXA neural-network normalization layer that normalizes activations using batch statistics during training and stored running statistics during inference.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsBatch normalization is a neural-network technique that normalizes intermediate activations using batch statistics during training.

Compare how 9 books explain batch normalization →

Related concepts

• layer normalizationcontrasts with · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• local response normalizationalternative to · Local response normalization scales an activation relative to neighboring activations at the same spatial location across adjacent feature maps.
• standard deviationuses · Standard deviation is the nonnegative square root of variance and measures the typical dispersion of values around their mean in the original units.
• exponential moving averageuses · An exponential moving average is a recursively updated weighted average that gives greater influence to recent observations while exponentially decreasing the influence of older ones.

This explainer was generated by Manning's knowledge graph from the text of Evolutionary Deep Learning: Genetic algorithms and neural networks. The section links point to the parts of the book it draws on.