Cosine annealing smoothly decays the learning rate during quantization-aware training (QAT), giving weights time to settle onto quantization-grid points. Paired with warmup, it avoids abrupt learning-rate changes that can cause weights to overshoot. section 5.3.1 section 5.2.2
A practical QAT schedule:
- Warm up the learning rate while quantization observers calibrate their statistics. section 5.3.1
- Then decay it smoothly with a cosine schedule so weights can make fine adjustments as the quantization grid stabilizes. section 5.3.1
Warmup followed by cosine decay
One implementation uses three warmup epochs, then cosine decay to 1% of the base learning rate by the end of training. section 5.3.1
Listing 5.7 QAT learning rate schedule with warmup
def create_qat_lr_schedule(optimizer, num_epochs, warmup_epochs=3,
min_lr_ratio=0.01):
"""
QAT-specific LR schedule: linear warmup + cosine decay.
The warmup phase lets observer statistics stabilize before
weights begin large updates. The cosine decay gives weights
time to settle onto grid points.
"""
def lr_lambda(epoch):
if epoch < warmup_epochs:
# Linear warmup: 10% -> 100% of base LR
return 0.1 + 0.9 * (epoch / warmup_epochs)
else:
# Cosine decay to min_lr_ratio of base LR
progress = (epoch - warmup_epochs) / (num_epochs - warmup_epochs)
return min_lr_ratio + (1 - min_lr_ratio) * 0.5 * (
1 + math.cos(math.pi * progress)
)
return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)See the PyTorch learning-rate schedule implementation, including the warmup and cosine-decay phases. section 5.3.1

