Training a model means adjusting its weights to reduce a loss. Stochastic gradient descent (SGD) does this using randomly drawn training examples rather than calculating every update from the full dataset. In practice, mini-batch SGD is the useful compromise: one example per update is true SGD, while using the entire dataset makes each update more expensive. section 2.4.3 section 2.6
For each mini-batch:
- Draw inputs and targets, then run the model to get predictions—a forward pass. section 2.4.3
- Measure the loss and compute its gradient with respect to the weights—a backward pass. section 2.4.3
- Move the weights a little against the gradient: W -= learning_rate * gradient. section 2.4.3
The learning rate controls the size of each move. Too small can mean many iterations; too large can send updates to unhelpful locations. The optimizer specifies the exact update rule—SGD with momentum, for example, uses more than the current gradient. section 2.4.3 section 2.6
Handwritten-digit training
A two-layer Keras classifier trained with fit(..., epochs=5, batch_size=128) processes mini-batches of 128 images. For each batch, backpropagation computes a loss gradient and the optimizer updates the weights. Here, the configured optimizer is Adam: mini-batch training does not require choosing an optimizer named SGD. section 2.5
def training_step(inputs, targets): predictions = model(inputs) loss = mean_squared_error(targets, predictions) loss.backward() optimizer.step() model.zero_grad() return loss
This PyTorch training step makes the order concrete: predict, calculate loss, call backward(), update parameters with optimizer.step(), then clear gradients with model.zero_grad(). Clearing matters because gradients otherwise accumulate across steps. section 3.4.2
