Batch normalization keeps inputs to activation functions in a useful range. If those inputs land far into a function’s saturated region, gradients can become small and training can slow. Normalizing intermediate values can support higher learning rates, reduce dependence on initialization, and provide a regularizing effect. section 8.5.2
For each feature, batch normalization uses the mean and standard deviation measured across a minibatch to shift and scale values. Because a sample’s result depends on the other samples drawn into its batch, this variation also acts as a form of regularization. section 8.5.2
Figure 8.13 BatchNorm for three samples with four input features involves calculating the mean and standard deviation for each feature across the samples. The mean and standard deviation are then used to normalize the input features such that each feature has zero mean and unit variance.
Three samples with four features show the calculation feature by feature: each feature is normalized using statistics collected across the samples. section 8.5.2
Put it before an activation
In a PyTorch convolutional network, `nn.BatchNorm2d(num_features=n_chans1)` normalizes the output of a convolution before `torch.tanh` and max pooling. A second normalization layer uses the second convolution’s output-channel count. section 8.5.2
# In[50]:
class NetBatchNorm(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv1_batchnorm = nn.BatchNorm2d(num_features=n_chans1)
self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3,
padding=1)
self.conv2_batchnorm = nn.BatchNorm2d(num_features=n_chans1 // 2)
self.fc1 = nn.Linear(8 * 8 * n_chans1 // 2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = self.conv1_batchnorm(self.conv1(x))
out = F.max_pool2d(torch.tanh(out), 2)
out = self.conv2_batchnorm(self.conv2(out))
out = F.max_pool2d(torch.tanh(out), 2)
out = out.view(-1, 8 * 8 * self.n_chans1 // 2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
The model listing shows both normalization layers in their forward-pass positions. section 8.5.2
Normalization can also sit at a network’s input. A 3D CT classifier uses `nn.BatchNorm3d(1)` in its tail before its convolutional backbone, so the rest of the network no longer sees the original Hounsfield-unit scale. Input cleaning still matters: extreme values can skew the statistics used for normalization. PyTorch models also have training and evaluation modes; call `model.eval()` for inference rather than leaving a trained network in training mode. section 13.4.2 section 12.3.1 section 8.5.2
