How deeply does Deep Learning with JAX cover batch normalization?

Explains normalization, learned transformations, running statistics, and separate training and inference behavior.

Batch normalization

A neural-network normalization layer that normalizes activations using batch statistics during training and stored running statistics during inference.

Want faster training and better model performance? Batch normalization normalizes layer activations, then learns how to scale and shift them. section 11.2.1

Batch normalization changes behavior between training and inference:

During training, calculate each activation’s batch mean and variance, normalize by standard deviation, then apply learned scale and shift parameters. section 11.2.1
Maintain exponential moving averages of mean and variance separately from learned parameters. section 11.2.1
During inference, use the stored averages and fixed learned scale and shift values, even for a single input. section 11.2.1

In Flax, treat the running statistics as model state rather than ordinary parameters; training updates that state through a mutable model-state path.

How Deep Learning with JAX covers it

liveBook manning.com

Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA normalization technique that standardizes activations using statistics computed across a batch.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA neural-network layer or operation that normalizes activations using batch statistics and behaves differently in training versus evaluation.
liveBook manning.com

Deep Learning with R, Third EditionBatch normalization is a neural-network normalization technique that standardizes layer activations using mini-batch statistics during training and stored moving statistics during inference.
liveBook manning.com

Sutskever's ListBatch normalization is a normalization technique that standardizes intermediate activations using statistics computed from training minibatches.
liveBook manning.com

Deep Learning with Python, Third EditionA normalization technique used in neural networks that stabilizes activations during training, commonly implemented as a normalization layer.
liveBook manning.com

Learn Generative AI with PyTorchA normalization technique that standardizes activations within minibatches to stabilize and speed neural-network training.
liveBook manning.com

Evolutionary Deep Learning: Genetic algorithms and neural networksBatch normalization is a neural-network technique that normalizes intermediate activations using statistics computed over training batches.
liveBook manning.com

Distributed Machine Learning PatternsBatch normalization is a neural-network technique that normalizes intermediate activations using batch statistics during training.

Compare how 9 books explain batch normalization →

Related concepts

layer normalizationcontrasts with · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
local response normalizationalternative to · Local response normalization scales an activation relative to neighboring activations at the same spatial location across adjacent feature maps.
standard deviationuses · Standard deviation is the nonnegative square root of variance and measures the typical dispersion of values around their mean in the original units.
exponential moving averageuses · An exponential moving average is a recursively updated weighted average that gives greater influence to recent observations while exponentially decreasing the influence of older ones.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.