A neural network can have trouble learning from inputs whose features have very different ranges. Normalization puts values on more comparable scales before training. For each feature, a common approach is to subtract its training-set mean and divide by its training-set standard deviation. The result is centered near zero with a standard deviation of one. section 4.3.2
Listing 4.25 Normalizing the data
mean = train_data.mean(axis=0) std = train_data.std(axis=0) x_train = (train_data - mean) / std x_test = (test_data - mean) / std
The NumPy example computes one mean and standard deviation per feature, then applies those same training-derived values to both training and test inputs. Do not calculate normalization statistics from the test set. section 4.3.2
Choose the scaling to match the data:
- For grayscale images encoded from 0 to 255, cast values to float32 and divide by 255 to obtain values from 0 to 1. section 6.2.1
- For weather timeseries, normalize each measured variable independently. Pressure near 1,000 and another measurement near 3 should not be treated as though they share a scale; compute statistics from the training timesteps only. section 13.2.1
- For regression targets much larger than initial predictions, scale the targets too. If prices are divided by 100,000 during training, multiply predictions by 100,000 to report dollar values. section 4.3.2
Input normalization happens before data enters the model. Batch normalization instead normalizes intermediate activations with a Keras BatchNormalization layer. It uses current-batch statistics during training and moving averages of training statistics during inference. It can help learning, but its precise reason for helping is not settled. section 9.3
