Learning rate schedule
A learning rate schedule is a rule or function that determines how an optimizer’s learning rate changes during training.
LLMs in Production
Learning rate scheduling is the practice of changing an optimizer's learning rate during training according to a predefined or adaptive schedule.
- How it is used
- Stabilizes language-model training and supports convergence while contributing to generalization alongside monitoring and other techniques.
- How deep it goes
- Defines the purpose and demonstrates CosineAnnealingLR configuration, but does not develop scheduling theory or compare multiple schedules.
- Examples
- Uses AdamW with CosineAnnealingLR in a SimpleLlama training configuration spanning 1,000 epochs.
- What you can do after
- Apply learning rate scheduling through PyTorch’s CosineAnnealingLR alongside an AdamW optimizer.
Read how LLMs in Production explains learning rate schedule →
Machine Learning Algorithms in Depth
A learning rate schedule is a rule that specifies how an optimization learning rate changes across iterations.
- How it is used
- Controls optimization step sizes during stochastic gradient descent and neural-network training.
- How deep it goes
- Derives a Robbins–Monro schedule, interprets its parameters, and implements iteration- and epoch-based decay.
- Examples
- Uses SGD logistic regression on a synthetic Gaussian mixture and neural-network training on CIFAR-100.
- What you can do after
- Apply it via SGD code or a TensorFlow/Keras LearningRateScheduler callback.
Read how Machine Learning Algorithms in Depth explains learning rate schedule →
Related concepts
- learning rateconfigures · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
- ReduceLROnPlateauincludes · ReduceLROnPlateau is a learning-rate scheduler that lowers the learning rate when a monitored quantity stops improving.
- cosine annealinghas part · Cosine annealing is a learning-rate scheduling method that varies the learning rate over training according to a cosine-shaped curve.
- optimizationuses · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.