liveBook · Manning

Warmup

As explained in Quantization and Fast Inference

Learning-rate warmup gradually increases the learning rate from a smaller initial value during an early phase of training.

In quantization-aware training (QAT), warmup starts with a smaller learning rate while fake-quantization observers calibrate. This limits large weight updates against inaccurate quantization parameters; afterward, cosine decay gives weights time to settle onto quantization grid points. section 5.3.1

A QAT learning-rate schedule

With three warmup epochs, the schedule ramps from 10% to 100% of the base learning rate, then cosine-decays toward 1%. section 5.3.1

Listing 5.7 QAT learning rate schedule with warmup
def create_qat_lr_schedule(optimizer, num_epochs, warmup_epochs=3,
                           min_lr_ratio=0.01):
    """
    QAT-specific LR schedule: linear warmup + cosine decay.
    
    The warmup phase lets observer statistics stabilize before
    weights begin large updates. The cosine decay gives weights
    time to settle onto grid points.
    """
    def lr_lambda(epoch):
        if epoch < warmup_epochs:
            # Linear warmup: 10% -> 100% of base LR
            return 0.1 + 0.9 * (epoch / warmup_epochs)
        else:
            # Cosine decay to min_lr_ratio of base LR
            progress = (epoch - warmup_epochs) / (num_epochs - warmup_epochs)
            return min_lr_ratio + (1 - min_lr_ratio) * 0.5 * (
                1 + math.cos(math.pi * progress)
            )
    
    return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

Inspect the PyTorch schedule implementation, including its configurable warmup duration and minimum learning-rate ratio. section 5.3.1

How Quantization and Fast Inference covers it

How deep it goes
Defines a linear ramp followed by cosine decay, with a PyTorch implementation and configurable durations.
Examples
Uses a three-epoch ramp from 10% to full learning rate before cosine decay.
What it assumes
Assumes QAT with fake quantization and observers, and explains how observer calibration affects early weight updates.
What you can do after
Apply via PyTorch LambdaLR to build a configurable warmup-plus-cosine learning-rate schedule.
How it connects
Coordinates learning-rate scheduling with fake quantization, observer calibration, and cosine decay.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.