A neural network learns by changing its weights to reduce a loss. Trying a higher and lower value for every weight would require too many expensive forward passes. Gradient descent uses the gradient of the loss to update the weights together instead. section 2.4
One coefficient, two directions
Suppose a coefficient is 0.3 and the batch loss is 0.5. Raising it to 0.35 raises the loss to 0.6; lowering it to 0.25 drops the loss to 0.4. The lower value looks promising, but testing both directions for every coefficient is impractical. A gradient supplies the direction of change without that trial-and-error search for each weight. section 2.4
This works because the model’s operations are differentiable: small changes in its weights produce predictable changes in loss. The gradient describes how loss varies across possible weight changes. For a mini-batch, training computes one gradient-descent update and applies it to the weights. Backpropagation is part of the surrounding explanation of how neural networks learn. section 2.4 section 4.0 section 2.0
Listing 5.7 Training an MNIST model with an incorrectly high learning rate
(train_images, train_labels), _ = mnist.load_data() train_images = train_images.reshape((60000, 28 * 28)) train_images = train_images.astype("float32") / 255 model = keras.Sequential( [ layers.Dense(512, activation="relu"), layers.Dense(10, activation="softmax"), ] ) model.compile( optimizer=keras.optimizers.RMSprop(learning_rate=1.0), loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) model.fit( train_images, train_labels, epochs=10, batch_size=128, validation_split=0.2 )
This MNIST run sets RMSprop’s learning rate to 1.0. Training and validation accuracy stall around 20%–40%; compare it with the lower-rate run. section 5.3.1
Listing 5.8 The same model with a more appropriate learning rate
model = keras.Sequential( [ layers.Dense(512, activation="relu"), layers.Dense(10, activation="softmax"), ] ) model.compile( optimizer=keras.optimizers.RMSprop(learning_rate=1e-2), loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) model.fit( train_images, train_labels, epochs=10, batch_size=128, validation_split=0.2 )
The same model uses a learning rate of 1e-2 and is able to train. The comparison makes learning rate a practical first check when training stalls. section 5.3.1
If the loss is stuck, check the gradient-descent configuration:
- Try a lower or higher learning rate. Too high can overshoot a good fit; too low can make progress appear stalled. section 5.3.1
- Try a larger batch size for gradients that are less noisy. Optimizer choice and weight initialization also matter, but changing learning rate and batch size while holding the rest constant is usually sufficient to get training started. section 5.3.1
