liveBook · Manning

Cosine annealing

Also called cosine decay

Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve.

How 2 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

A learning-rate schedule that changes the learning rate according to a cosine-shaped pattern over training.

How it is used
Schedules learning rates to alternate between higher and lower values during neural-network training.
How deep it goes
Explains the schedule’s parameters, motivation, tradeoff, and PyTorch setup without detailing scheduler internals.
Examples
Compares scheduled and unscheduled AdamW and Nesterov SGD in fully connected network experiments.
What you can do after
Apply it via PyTorch’s CosineAnnealingLR by configuring an optimizer, cycle length, and minimum learning rate.

Read how Inside Deep Learning: Math, Algorithms, Models explains cosine annealing →

Quantization and Fast Inference

Cosine annealing is a schedule that smoothly decreases the learning rate along a cosine-shaped curve.

How it is used
Schedules learning-rate decay during quantization-aware training so weights can settle onto quantization-grid points.
How deep it goes
Explains the warmup-and-decay rationale and gives schedule parameters and implementation, without comparing broader scheduling methods quantitatively.
Examples
Uses a three-epoch warmup and cosine decay to 1% of the base learning rate as an implementation example.
What you can do after
Apply it through PyTorch's LambdaLR in a warmup-plus-cosine QAT schedule.

Read how Quantization and Fast Inference explains cosine annealing →

Also covered in

What to know first

Related concepts

Questions about cosine annealing

What is cosine annealing?

Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve.

Which Manning books explain cosine annealing?

Inside Deep Learning: Math, Algorithms, Models and Quantization and Fast Inference.

What should I know before learning cosine annealing?

Manning books that teach cosine annealing usually introduce epoch first.

What is the difference between cosine annealing and step decay?

Cosine annealing is a learning-rate schedule that adjusts the learning rate over training according to a cosine-shaped curve. Step decay is a learning-rate schedule that reduces the rate abruptly at specified training milestones.

What is cosine annealing part of?

Cosine annealing is part of learning rate schedule. A learning rate schedule is a rule that determines how an optimizer’s learning rate changes during training according to progress or other conditions.

What is cosine annealing also called?

Cosine annealing is also called cosine decay.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.