Optimization selects model parameters that reduce a loss function. Picture the weights skiing down an error surface toward a low point, using gradients computed from training examples or mini-batches. section 5.3
Stochastic gradient descent (SGD) updates weights from one sample or a mini-batch rather than the entire dataset. Its learning rate controls step size: larger values may converge faster but can overshoot the minimum. Momentum accelerates movement in a promising direction and slows movement away from the target. section 5.2.5
A practical PyTorch optimization loop follows this sequence:
- Tune hyperparameters such as learning rate and momentum for the problem rather than treating their initial values as universally effective. section 5.2.5
Optimizing a name classifier
A PyTorch logistic-regression network predicts whether names are associated with female labels. Binary cross-entropy supplies the loss, weighted by name frequencies; SGD receives the model parameters, a 0.02 learning rate, and 0.001 momentum. Across 200 epochs, the loop progressively lowers loss while test accuracy rises in the reported run.
For convolutional neural networks, optimization learns kernel weights instead of requiring hand-crafted text patterns. RMSprop improves gradient estimates by maintaining an exponentially decaying moving average of recent gradients, which can speed NLP CNN training. section 7.2.1 section 7.5.4
Figure 5.6 Convex error curve
Use the convex error-curve figure to connect parameter choices with loss minima and the direction of optimization. section 5.3
