A learning rate is the step size used when changing model weights during optimization. After computing the loss gradient, training moves each parameter in the opposite direction by an amount scaled by the learning rate. section 2.4.3 section 2.6
Choose the value with the training behavior in mind:
- Too small: updates are tiny, so descent can require many iterations and appear to stall. section 2.4.3 section 5.3.1
- Too large: updates can overshoot a useful fit and send optimization to poor or seemingly random locations. section 2.4.3 section 5.3.1
- Tune it together with batch size when training stalls; larger batches can make gradients more informative and less noisy. section 5.3.1
Training MNIST with RMSprop
A dense MNIST model using RMSprop with a learning rate of 1 reaches only about 20%–40% accuracy. Keeping the model and batch size unchanged while reducing the learning rate to 1e-2 allows training to proceed. section 5.3.1
Listing 2.9 A single step of training
one_training_step <- function(model, images_batch, labels_batch) {
predictions <- model$call(images_batch) #1
loss <-
op_sparse_categorical_crossentropy(labels_batch, predictions) #2
average_loss <- op_mean(loss)
gradients <- get_gradients_of_loss_wrt_weights(loss, model$weights)
update_weights(gradients, model$weights) #3
loss
}
Follow one training step from predictions and loss through automatic differentiation to weight updates. section 2.5.2
