liveBook · Manning

Cosine annealing

As explained in Quantization and Fast Inference

Cosine annealing is a schedule that smoothly decreases the learning rate along a cosine-shaped curve.

Cosine annealing smoothly decays the learning rate during quantization-aware training (QAT), giving weights time to settle onto quantization-grid points. Paired with warmup, it avoids abrupt learning-rate changes that can cause weights to overshoot. section 5.3.1 section 5.2.2

A practical QAT schedule:

  • Warm up the learning rate while quantization observers calibrate their statistics. section 5.3.1
  • Then decay it smoothly with a cosine schedule so weights can make fine adjustments as the quantization grid stabilizes. section 5.3.1

Warmup followed by cosine decay

One implementation uses three warmup epochs, then cosine decay to 1% of the base learning rate by the end of training. section 5.3.1

Listing 5.7 QAT learning rate schedule with warmup
def create_qat_lr_schedule(optimizer, num_epochs, warmup_epochs=3,
                           min_lr_ratio=0.01):
    """
    QAT-specific LR schedule: linear warmup + cosine decay.
    
    The warmup phase lets observer statistics stabilize before
    weights begin large updates. The cosine decay gives weights
    time to settle onto grid points.
    """
    def lr_lambda(epoch):
        if epoch < warmup_epochs:
            # Linear warmup: 10% -> 100% of base LR
            return 0.1 + 0.9 * (epoch / warmup_epochs)
        else:
            # Cosine decay to min_lr_ratio of base LR
            progress = (epoch - warmup_epochs) / (num_epochs - warmup_epochs)
            return min_lr_ratio + (1 - min_lr_ratio) * 0.5 * (
                1 + math.cos(math.pi * progress)
            )
    
    return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

See the PyTorch learning-rate schedule implementation, including the warmup and cosine-decay phases. section 5.3.1

How Quantization and Fast Inference covers it

How deep it goes
Explains the warmup-and-decay rationale and gives schedule parameters and implementation, without comparing broader scheduling methods quantitatively.
Examples
Uses a three-epoch warmup and cosine decay to 1% of the base learning rate as an implementation example.
What you can do after
Apply it through PyTorch's LambdaLR in a warmup-plus-cosine QAT schedule.
How it connects
Connects learning-rate warmup with observer calibration, quantization-aware training, and quantization-grid alignment.

Also explained in

Compare how 2 books explain cosine annealing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.