A warm start initializes training with weights believed to be closer to the desired solution than random values. In transfer learning, those weights commonly come from a pretrained model, but warm starting also applies to unrelated optimization problems. section 13.2.3
Use warm starts to:
- Continue training a pretrained model on new data while updating its weights. section 13.2.3
- Reuse a nearby solution during hyperparameter optimization, such as initializing a model for regularization λ + ϵ with the solution for λ. section 13.2.3
- Compare trainable warm weights with frozen pretrained weights when training data is scarce; freezing reduces the number of parameters that can change. section 13.3
Warm-started image classifier
Create a pretrained ResNet-18, replace its final layer for two classes, normalize its inputs, and train it with the existing weights rather than random initialization. Compare its learning curves with regular and frozen variants. section 13.3
warmstart_results = train_network(model_pretrained, loss, train_loader,
epochs=10, device=device, test_loader=test_loader, score_funcs={’Accuracy’:
accuracy_score})
Inspect the PyTorch training call that passes a pretrained model into train_network and records warm-start results. section 13.2.3
