Cosine annealing
Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve.
Inside Deep Learning: Math, Algorithms, Models
A learning-rate schedule that changes the learning rate according to a cosine-shaped pattern over training.
- How it is used
- Schedules learning rates to alternate between higher and lower values during neural-network training.
- How deep it goes
- Explains the schedule’s parameters, motivation, tradeoff, and PyTorch setup without detailing scheduler internals.
- Examples
- Compares scheduled and unscheduled AdamW and Nesterov SGD in fully connected network experiments.
- What you can do after
- Apply it via PyTorch’s CosineAnnealingLR by configuring an optimizer, cycle length, and minimum learning rate.
Read how Inside Deep Learning: Math, Algorithms, Models explains cosine annealing →
Quantization and Fast Inference
Cosine annealing is a schedule that smoothly decreases the learning rate along a cosine-shaped curve.
- How it is used
- Schedules learning-rate decay during quantization-aware training so weights can settle onto quantization-grid points.
- How deep it goes
- Explains the warmup-and-decay rationale and gives schedule parameters and implementation, without comparing broader scheduling methods quantitatively.
- Examples
- Uses a three-epoch warmup and cosine decay to 1% of the base learning rate as an implementation example.
- What you can do after
- Apply it through PyTorch's LambdaLR in a warmup-plus-cosine QAT schedule.
Read how Quantization and Fast Inference explains cosine annealing →
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- epochAn epoch is one complete pass through the training data during model training.
Related concepts
- warmupused by · Learning-rate warmup is an initial training schedule that gradually increases the learning rate from a low value toward its target value.
- learning rateconfigures · A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
- CosineAnnealingLRimplemented by · A PyTorch learning-rate scheduler class that implements cosine annealing.
- learning rate schedulepart of · A learning rate schedule is a rule that determines how an optimizer’s learning rate changes during training according to progress or other conditions.
- quantization-aware trainingused by · Quantization-aware training optimizes a model while simulating or incorporating quantization effects so it can retain performance with reduced-precision representations.
Questions about cosine annealing
What is cosine annealing?
Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve.
Which Manning books explain cosine annealing?
Inside Deep Learning: Math, Algorithms, Models and Quantization and Fast Inference.
What should I know before learning cosine annealing?
Manning books that teach cosine annealing usually introduce epoch first.
What is the difference between cosine annealing and step decay?
Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve. Step decay is a learning-rate schedule that reduces the rate abruptly at specified training milestones.
What is cosine annealing part of?
Cosine annealing is part of learning rate schedule. A learning rate schedule is a rule that determines how an optimizer’s learning rate changes during training according to progress or other conditions.
What is cosine annealing also called?
Cosine annealing is also called cosine decay.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

