Normalization changes input values to a more useful scale for learning. If one input produces much larger gradients than another, a single learning rate may update the model’s parameters unevenly. Rescaling the inputs can bring those gradients closer together. section 5.4.4
A small rescaling that works
In a temperature-conversion model, `t_un = 0.1 * t_u` brings the input closer to the range −1 to 1. With `t_un` passed to the training loop, the parameter updates no longer blow up at a learning rate of `1e-2`. This simple rescaling is sufficient for the example, even though it is not a complete statistical normalization. section 5.4.4
A common alternative standardizes each feature separately: subtract its mean, then divide by its standard deviation. For a tensor whose rows are observations and columns are features, compute column statistics with `dim=0`, then use `data_normalized = (data - data_mean) / torch.sqrt(data_var)`. This gives features with different original scales a comparable starting point. section 4.3.5
For images, apply the same idea per channel: `v_n[c] = (v[c] - mean[c]) / stdev[c]`. Compute the channel statistics across the dataset first; `transforms.Normalize` uses the supplied values rather than calculating them. Keeping channels on comparable scales helps them be updated through gradient descent with the same learning rate. It also keeps inputs nearer ranges where the described activation functions have nonzero gradients. section 7.1.4
# In[19]:
transformed_cifar10 = datasets.CIFAR10(
data_path, train=True, download=False,
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean=mean, std=std)
]))
The CIFAR-10 dataset applies `ToTensor` and then `Normalize` through `transforms.Compose`. The supplied channel means and standard deviations are calculated beforehand. section 7.1.4
Normalization is a choice to test, not one fixed recipe. A training exercise replaces `nn.BatchNorm` with custom normalization and compares fixed offsets, scales, and nonlinear options such as square roots. The word also appears as a stage in a text-tokenization pipeline, without a specific text-normalization method being detailed. section 14.8 section 9.8
