liveBook · Manning

CosineAnnealingLR

As explained in Inside Deep Learning: Math, Algorithms, Models

A PyTorch learning-rate scheduler class that implements cosine annealing.

CosineAnnealingLR schedules the learning rate by repeatedly decreasing it toward a minimum and increasing it again. The oscillation can escape sub-optimal minima in non-convex neural-network optimization, but it is less stabilizing than purely decreasing schedules. section 5.2.3

Configure the schedule around these quantities:

  • Set η0 as the initial learning rate and ηmin as the minimum rate. section 5.2.3
  • Choose Tmax as the number of epochs between cosine cycles; for S dips across T epochs, use Tmax = T/(S⋅2−1). section 5.2.3
  • Expect a tradeoff: oscillation may improve the final result while reducing stability on poorly behaved datasets or networks. section 5.2.3

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Defines cosine parameters and cycle selection, then demonstrates scheduler construction without detailing scheduler internals.
Examples
Uses a cosine-rate plot, a three-phase schedule, and an SGD training example with eta_min=0.0001.
What you can do after
Apply it via PyTorch by attaching CosineAnnealingLR to an optimizer and selecting cycle length and minimum rate.
How it connects
Connects learning-rate scheduling with SGD, gradient descent, non-convex optimization, local minima, and convergence.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.