A training technique that averages model parameters collected at multiple points during training, often from learning-rate-cycle minima.
Stochastic weight averaging (SWA) combines parameters collected at multiple learning-rate minima into one model. It approximates an ensemble’s accuracy without training or storing many separate networks. section 5.2.3
The cosine-annealing workflow is:
Run one model through repeated cosine learning-rate cycles. section 5.2.3
At each learning-rate minimum, copy the model parameters Θ. section 5.2.3
Average the collected parameter sets to produce one final model. section 5.2.3
Twenty-cycle ensemble
For an intended ensemble of 20 networks over T training epochs, choose a cosine schedule with T_max = T/(20·2−1), producing 20 learning-rate dips and one averaged model. section 5.2.3
How Inside Deep Learning: Math, Algorithms, Models covers it
How deep it goes
Defines the averaging strategy and schedule relationship but stops short of showing SWA API calls or parameter-update code.
Examples
Uses twenty cosine learning-rate dips as a concrete approximation to a twenty-network ensemble.
What it assumes
Assumes ensembling and cosine learning-rate schedules, while linking their benefits through parameter averaging.
What you can do after
Apply it via PyTorch to obtain ensemble-like accuracy with one model’s training and storage cost.
How it connects
Connects stochastic weight averaging with ensembling, cosine annealing, learning rates, and gradient descent.
ensembleequivalent to · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.
