Cosine annealing changes the learning rate down and up over training instead of only decreasing it. This can help optimization escape a poor local minimum, although the extra movement reduces stability on poorly behaved datasets or networks. section 5.2.3
Set an initial rate η0, minimum rate ηmin, and cycle length Tmax. The cosine term oscillates between these bounds; Tmax controls the number of epochs between schedule dips. A practical choice aims for roughly 10–50 dips and ends training on a dip. section 5.2.3
PyTorch implementation
For three schedule phases across training, attach CosineAnnealingLR with epochs//3: scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, epochs//3, eta_min=0.0001). Pass the scheduler to train_network; the same pattern works with SGD or AdamW. section 5.2.3 section 5.3.2
cos_lr = eta_min +
0.5*(eta_init-eta_min)*(1+np.cos(epochs_input/(T/3.0)*np.pi)) #1
sns.lineplot(x=epochs_input, y=eta_init, color=’red’, label="$\eta_0$")
sns.lineplot(x=epochs_input, y=cos_lr, color=’purple’, label="$\cos$")
sns.lineplot(x=epochs_input, y=[eta_init]*18+[eta_init/3.16]*16 + *16, color=’green’,
label="StepLR") % ax = sns.lineplot(x=epochs_input, y=effective_learning_rate, color=’blue’,
label="$\eta_0 \cdot \gamma^t$")
ax.set_xlabel(’Epoch’)
ax.set_ylabel(’Learning Rate’)
[17]: Text(0, 0.5, 'Learning Rate')
#1 Computes the cosine schedule <span class="math"><i>η</i><i><sub>t</sub></i></span> for every value of <span class="math"><i>t</i></span>
Compare the cosine curve with constant, step, and exponentially decaying learning rates. section 5.2.3
fc_model.apply(weight_reset)
optimizer = torch.optim.SGD(fc_model.parameters(), lr=eta_0)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, epochs//3,
eta_min=0.0001) #1
fc_results_coslr = train_network(fc_model, loss_func, train_loader,
test_loader=test_loader, epochs=epochs, optimizer=optimizer,
lr_schedule=scheduler, score_funcs={’Accuracy’: accuracy_score},
device=device)
#1 Tells the cosine to go down, then up, and then down (that’s three). If we were doing more than 10 epochs, I would push this higher.
Inspect a complete training configuration using SGD, CosineAnnealingLR, eta_min, and a three-phase schedule. section 5.2.3
