Optimization systematically adjusts model parameters to reduce a loss function, decrease prediction error, or improve accuracy. In deep learning, those parameters are weights and biases. section 8.2.6
Gradient-based approaches differ in how much training data they use for each update:
- Batch gradient descent updates in the direction of the negative gradient using the full training dataset, seeking a function's global minimum but potentially running slowly on large or high-dimensional data. section 8.2.6
- Stochastic gradient descent updates coefficients after each training instance, reducing the time required for an update. section 8.2.6
- Mini-batch gradient descent uses small data subsets to balance speed and accuracy, while adding batch size as a hyperparameter commonly set to powers of two such as 32, 64, or 128. section 8.2.6
Backpropagation supplies the mechanism for optimizing a network. Continuous optimizer feedback improves weights and biases until the network is trained, after which it can predict on unseen data. section 8.4.2
Figure 8.8 The concept of gradient descent. It is the mechanism to minimize the loss function.
Visualize gradient descent as repeated movement downward toward a global minimum while minimizing the loss function. section 8.2.6
