Stochastic gradient descent (SGD) is an optimization method for training models. It adjusts model parameters to reduce a loss: after a forward pass and loss calculation, backpropagation computes gradients, and the optimizer updates the parameters in the direction that lowers the loss. section 5.8
In PyTorch, SGD works with autograd, which calculates how each parameter contributes to the final output. A typical training loop then clears old gradients with optimizer.zero_grad(), performs the backward pass, and applies the update with the optimizer step. This prevents gradients from previous updates from being incorrectly accumulated. section 5.8
A practical SGD setup usually follows these choices:
- Start with torch.optim.SGD when choosing an optimizer; it is generally considered a safe starting point. section 13.3.1
- Choose a learning rate to control update size and momentum to influence how updates use prior movement. section 13.3.1
- Treat the initial values as practical starting points rather than guaranteed best settings, then search systematically when larger issues are resolved. section 13.3.1
Initializing SGD for a PyTorch model
Move the model to its target device before constructing the optimizer, then pass the model parameters to SGD: optimizer = SGD(model.parameters(), lr=0.001, momentum=0.99). For a CUDA-enabled application, this keeps the optimizer connected to the GPU-based parameter objects. section 13.3.1
Listing 13.4 LunaTrainingApp class initialization
class LunaTrainingApp:
def __init__(self, sys_argv=None):
# ... line 70
self.use_cuda = torch.cuda.is_available()
self.device = torch.device("cuda" if self.use_cuda else "cpu")
self.model = self.initModel()
self.optimizer = self.initOptimizer()
def initModel(self):
model = LunaModel()
if self.use_cuda:
log.info(f"Using CUDA; {torch.cuda.device_count()} devices.")
if torch.cuda.device_count() > 1: #1
model = nn.DataParallel(model) #2
model = model.to(self.device) #3
return model
def initOptimizer(self):
return SGD(self.model.parameters(), lr=0.001, momentum=0.99)
Inspect a complete initialization listing showing device selection, model placement, and construction of torch.optim.SGD with learning rate 0.001 and momentum 0.99. section 13.3.1
