In quantization-aware training (QAT), warmup starts with a smaller learning rate while fake-quantization observers calibrate. This limits large weight updates against inaccurate quantization parameters; afterward, cosine decay gives weights time to settle onto quantization grid points. section 5.3.1
A QAT learning-rate schedule
With three warmup epochs, the schedule ramps from 10% to 100% of the base learning rate, then cosine-decays toward 1%. section 5.3.1
Listing 5.7 QAT learning rate schedule with warmup
def create_qat_lr_schedule(optimizer, num_epochs, warmup_epochs=3,
min_lr_ratio=0.01):
"""
QAT-specific LR schedule: linear warmup + cosine decay.
The warmup phase lets observer statistics stabilize before
weights begin large updates. The cosine decay gives weights
time to settle onto grid points.
"""
def lr_lambda(epoch):
if epoch < warmup_epochs:
# Linear warmup: 10% -> 100% of base LR
return 0.1 + 0.9 * (epoch / warmup_epochs)
else:
# Cosine decay to min_lr_ratio of base LR
progress = (epoch - warmup_epochs) / (num_epochs - warmup_epochs)
return min_lr_ratio + (1 - min_lr_ratio) * 0.5 * (
1 + math.cos(math.pi * progress)
)
return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)Inspect the PyTorch schedule implementation, including its configurable warmup duration and minimum learning-rate ratio. section 5.3.1
