Batch normalization is a layer that normalizes intermediate activations, not only the input data. It centers values around zero and scales them toward unit variance, helping when successive transformations change activation statistics during training. section 9.3
Training versus inference
During training, BatchNormalization uses the mean and variance of the current batch. During inference, it uses exponential moving averages collected from batches seen during training, avoiding dependence on a large representative batch at prediction time. section 9.3
Use these practical placement rules when building a convolutional network:
- Place a batch-normalization layer after a convolution; disabling the convolution bias is shown as a compatible pattern. section 9.3
- Choose the activation ordering deliberately: normalization can follow a convolution with an existing activation, or precede a separate ReLU activation. section 9.3
- Treat it as an architecture pattern rather than a guaranteed explanation of improved performance; its precise mechanism remains uncertain. section 9.3
In practice, the main observed benefit appears to be smoother gradient propagation, much like residual connections, which can make deeper networks easier to train. It is commonly combined with residual connections in deep ConvNets. section 9.3 section 20.1.6
Xception-like image classifier
A mini Xception-like model places BatchNormalization layers before activations inside repeated convolutional blocks, alongside separable convolutions, max pooling, and residual connections. The complete architecture reaches 90.8% test accuracy on dogs-versus-cats, compared with 83.9% for the preceding model; the improvement belongs to the combined architecture choices. section 9.5
x = layers.Conv2D(32, 3, use_bias=False)(x) #1 x = layers.BatchNormalization()(x) x = layers.Activation("relu")(x) #2
Compare a convolution, batch normalization, and separately applied ReLU in a common pre-activation arrangement. section 9.3
