Optimization is the iterative process of minimizing a loss function by adjusting neural-network weights. Different optimization algorithms determine how those updates proceed, including SGD, momentum, Nesterov, AdaGrad, and Adam. section 9.0
With a fixed architecture, training learns the connector weights that produce useful class boundaries. The loss measures departure from desired outputs; optimization repeatedly moves the weights toward lower loss. section 9.0
A PyTorch SGD training step follows this sequence:
- Compute model scores for each batch and evaluate the loss. section 9.2.3
- Clear previous gradients, backpropagate the loss, and update weights in the negative-gradient direction. section 9.2.3
- Repeat the loop across batches and epochs; initialization and learning rate affect convergence. section 9.2.3
Statsville classifier
A small PyTorch classifier uses CrossEntropyLoss and SGD with a learning rate of 0.02. Its training loop reduces loss across epochs while predictions become more accurate for the three height-and-weight classes. section 9.2.3
Figure 9.6 Height and weight of various Statsville residents. Class 0 (man) is represented by the right-most cluster, class 1 (woman) by the middle cluster, and class 2 (child) by the left-most cluster.
Use the class-cluster visualization to connect parameter updates and decreasing loss with changing classification regions. section 9.2.3
