A weight matrix holds learned coefficients that parameterize connections between neural-network units. It is part of the model parameters, denoted θ: changing its entries changes the network’s behavior. The matrix describes what can be learned; a training method determines how its entries change. section 1.4 section 10.4
There are two ways to reason about θ here. In Bayesian inference, parameters are random variables: a prior p(θ) and likelihood p(x|θ) determine a posterior p(θ|x) proportional to p(x|θ)p(θ). In neural-network optimization, θ is updated to reduce a loss. These are different treatments of parameters, not interchangeable update rules. section 1.4 section 10.4
For stochastic gradient descent, let g be the loss gradient with respect to θ, computed using a mini-batch. An update takes a step in the negative gradient direction: θₖ₊₁ = θₖ − εₖg. The learning rate εₖ controls the step size. If it is too large, updates can miss an optimum; if too small, training can take unnecessarily long. section 10.4
Training example
A convolutional neural network trained on CIFAR-100 with Keras/TensorFlow provides a concrete setting for comparing optimizers. Its training curves report cross-entropy loss and validation accuracy, rather than displaying each weight-matrix entry. section 10.4
Figure 10.13 Cross-entropy loss and validation accuracy of LeNet trained on cifar100 using different optimizers
Compare the loss and validation-accuracy curves to see how optimizer choice affects training the model’s parameters. section 10.4
