Gradient descent trains a neural network by repeatedly adjusting its weights to reduce a loss: a number measuring how far its predictions are from the targets. Instead of testing changes to thousands of weights one at a time, it uses the loss gradient to find an update direction for all of them together. section 2.4
Train a handwritten-digit classifier
In R with Keras, load MNIST, build a two-layer classifier, choose an optimizer and loss, then fit the model: library(keras3) .[.[train_images, train_labels], .] <- dataset_mnist() train_images <- array_reshape(train_images / 255, c(60000, 28 * 28)) model <- keras_model_sequential() |> layer_dense(512, activation = "relu") |> layer_dense(10, activation = "softmax") model |> compile(optimizer = "adam", loss = "sparse_categorical_crossentropy", metrics = "accuracy") fit(model, train_images, train_labels, epochs = 5, batch_size = 128) Here, Keras handles the weight updates during fitting; `batch_size = 128` sets how many training examples are used at a time. section 2.1
One mini-batch update works as follows:
- Draw a batch of inputs and targets. Run the model to make predictions—the forward pass—and calculate their loss. section 2.4.3
- Calculate the gradient of that loss with respect to the weights—the backward pass. Differentiable model operations make this possible; automatic differentiation can perform the calculation. section 2.4 section 2.5.2
- Move each weight a little against its gradient: `weight <- weight - learning_rate * gradient`. Repeat with further batches. The learning rate controls the size of each move. section 2.4.3
one_training_step <- function(model, images_batch, labels_batch) {
with(tf$GradientTape() %as% tape, {
predictions <- model$call(images_batch)
per_sample_losses <-
op_sparse_categorical_crossentropy(labels_batch, predictions)
average_loss <- op_mean(per_sample_losses)
})
gradients <- tape$gradient(average_loss, model$weights)
update_weights(gradients, model$weights)
average_loss
}
This training-step listing records a forward pass and average loss with TensorFlow’s `GradientTape`, obtains gradients for the model weights, and passes them to a weight-update function. section 2.5.2
Mini-batch stochastic gradient descent draws batches at random. Using one example per update is true stochastic gradient descent; using the entire dataset is batch gradient descent. Mini-batches balance update cost against the quality of the gradient estimate. section 2.4.3
When training stalls
An MNIST classifier trained with an excessively large learning rate of `1` reaches only about 20%–40% training and validation accuracy before stalling; with `1e-2`, it can train. If loss is stuck, try changing the learning rate: large steps can overshoot a useful fit, while tiny steps can make progress look stalled. Increasing batch size can make gradients less noisy. section 5.3.1
