Optimizer state

Q: How deeply does Deep Learning with PyTorch, Second Edition cover optimizer state?

Shows SGD construction and parameter updates, but does not inspect internal optimizer state or demonstrate saving and restoration.

As explained in Deep Learning with PyTorch, Second Edition

The internal state of an optimization algorithm, such as momentum buffers, that can be saved to resume training exactly.

Optimizer state is information an optimizer keeps while training. It is distinct from a model’s weights and biases. The PyTorch examples show how an optimizer receives parameters and updates them, but do not inspect its internal state or show how to save it. section 6.2.2

An optimizer in a training loop

For a linear model, create `optimizer = optim.SGD(linear_model.parameters(), lr=1e-2)`. Each training iteration calls `optimizer.zero_grad()`, computes a loss, calls `loss.backward()`, then calls `optimizer.step()`. Backward fills the parameters’ `grad` attributes; `step()` uses those gradients to change the parameters. The example does not expose any separate optimizer state. section 6.2.2

A model with multiple linear layers follows the same pattern: `model.parameters()` supplies their weights and biases to the optimizer. AdamW and SGD are also suggested for comparison during segmentation training, but no procedure for saving or restoring either optimizer is shown. section 6.3.2 section 15.10

How Deep Learning with PyTorch, Second Edition covers it

Cover of Deep Learning with PyTorch, Second Edition

liveBook manning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Related concepts

learning rateconfigures · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
momentumconfigures · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
trainingused by · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.
adamused by · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.