Group normalization stabilizes neural-network activations by dividing feature maps into predefined groups, then recentering and rescaling values within each group before activation. section 12.17
The process works like small teams receiving separate brightness and contrast adjustments:
- Divide the output feature maps into groups. section 12.17
- Calculate one mean and standard deviation for each group. section 12.17
- Normalize values using the statistics from their own group. section 12.17
Eight feature maps, one group
A convolution produces an 8×8×8 tensor: eight feature maps, each with an 8×8 grid. Using one group for all eight maps, examine the first pixel in every map, calculate their combined mean and standard deviation, and normalize those eight values together before sending them to the activation function. section 12.17
Figure 12.30 The first pixel of all feature maps produced by the first convolutional layer
View the first pixel across the feature maps before normalization, then compare the resulting normalized values in the companion calculation. section 12.17
