Optimization updates a neural network’s weights so its predictions produce a lower loss. The loss supplies feedback: the optimizer uses it to move weights in a direction that improves the current examples. section 1.6 section 2.5
A practical training loop connects four pieces:
- The model maps input data to predictions through layers containing weight tensors. section 2.5
- The loss function compares predictions with targets and produces the quantity to minimize. section 1.6 section 3.6.4
- Backpropagation computes the loss gradient with respect to the weights; gradient descent then moves the weights opposite the derivative. section 2.4.1 section 2.5
- The optimizer selects the specific stochastic-gradient-descent rules, such as those used by Adam, RMSprop, or SGD with momentum. section 2.5 section 3.6.4
Configuring optimization with Keras
A Keras model can specify Adam during compilation, then train for five epochs with batches of 128 samples: model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]); model.fit(train_images, train_labels, epochs=5, batch_size=128). The loss is optimized, while accuracy is monitored rather than directly optimized. section 2.5 section 3.6.4
Figure 2.26 Relationship between the network, layers, loss function, and optimizer
Use the relationship diagram to connect layers and weights, the loss function, and the optimizer within training. section 2.5
