train_model_simple is a compact PyTorch training loop for pretraining a GPTModel. It iterates through epochs and batches, computes loss, backpropagates gradients, updates weights, tracks tokens, and periodically evaluates progress. section 5.2
The loop follows this flow:
- Set training mode, reset gradients, calculate batch loss, call loss.backward(), and update parameters with the optimizer. section 5.2
- Track tokens seen and the global step, then call evaluate_model at the configured evaluation frequency. section 5.2
- Measure training and validation loss in evaluation mode with gradients and dropout disabled. section 5.2 section 5.4
- Generate text from a start context to provide a qualitative check alongside numeric losses. section 5.2
A pretraining run
Create a GPTModel, move it to the selected device, initialize AdamW, and call train_model_simple with ten epochs, evaluation every five steps, five evaluation batches, and the context “Every effort moves you.” Training and validation losses fall initially, while later validation stagnation reveals overfitting; generated samples become coherent but may memorize training passages. section 5.2
Figure 5.11 A typical training loop for training deep neural networks in PyTorch consists of numerous steps, iterating over the batches in the training set for several epochs. In each loop, we calculate the loss for each training set batch to determine loss gradients, which we use to update the model weights so that the training set loss is minimized.
The training-loop flowchart maps epoch iteration, batch loss, gradient computation, weight updates, evaluation, and sample generation into one workflow. section 5.2
