Optimization adjusts model parameters in the direction that reduces training loss. Gradient descent estimates that direction from computed gradients; the learning rate controls step size, with LLM fine-tuning commonly using rates between 1e-5 and 1e-3. section 3.1.2
Choose the optimizer according to dataset scale, model size, memory limits, and overfitting risk:
- Use stochastic gradient descent (SGD) to estimate gradients from small random batches, accepting noisy steps that can explore the loss landscape. section 3.1.2
- Use Adam when parameters need individually adapted learning rates; use AdamW when improved weight decay is needed to reduce overfitting. section 3.1.2
- Use Adafactor when memory efficiency matters for very large models. section 3.1.2
One training step
For each input-target batch, the model performs a forward pass, computes cross-entropy loss, clears old gradients, backpropagates the loss, and calls optimizer.step() to update parameters. Logging the loss periodically helps track training progress. section 3.1.2





































