The learning rate is the optimization step size: it controls how far model weights move on each update while training. In gradient descent, the update moves opposite the loss gradient; the learning rate determines the size of that downhill move. It is therefore a central control for both convergence speed and training stability. section 3.1.2
Choosing a rate is a trade-off. A rate that is too large can overshoot lower-loss settings, make loss fluctuate, and prevent convergence. A rate that is too small produces stable but painfully slow progress. For LLMs, cited values range from 1e-5 to 1e-3; pre-training commonly uses 1e-4 to 3e-4, while fine-tuning commonly uses 1e-5 to 5e-5. section 3.1.2
During fine-tuning, set the learning rate alongside related controls:
- Use a lower rate than pre-training when preserving the foundation model’s learned representations matters; fine-tuning should make careful adaptations rather than dramatic changes. section 1.4.2
- Balance it with batch size and gradient accumulation, which affect update stability, efficiency, and resource needs. section 1.4.2
- Manage epochs and use evaluation or early stopping to avoid overfitting training examples instead of learning generalizable patterns. section 1.4.2
Analyst-report summarization configuration
A fine-tuning run formats each analyst report as an input and its verified summary as the target, then configures training with a learning rate of 2e-4, batch size 2, gradient accumulation of 4, warmup steps, a linear scheduler, logging, and evaluation. Together, these settings govern how quickly weights move toward producing useful summaries without simply memorizing report phrasing. section 2.5.3 section 2.5.3
Figure 3.5 Gradient descent principle illustrated with a single weight variable. For LLMs, the same principle applies to billions of parameters.

Use the single-weight landscape illustration to connect a numerical learning rate to the practical choice between overshooting, slow progress, and steady descent. section 3.1.2






















