Stochastic gradient descent (SGD) minimizes model error by updating parameters from one training example or a mini-batch at a time, rather than the entire dataset. Randomly reshuffling examples changes the path through the error surface and can help the model avoid settling permanently at a local minimum. section 5.2.5 section 5.3.2
Learning rate controls how far each update moves: larger values may converge faster but can overshoot the minimum. Momentum accelerates movement in a promising direction and slows movement away from the target; both are hyperparameters to tune. section 5.2.5 section 5.3.2
Training a PyTorch name classifier
A PyTorch logistic-regression model predicts whether names are labeled female or male. The workflow converts vectorized names and labels into tensors, defines binary cross-entropy loss, configures SGD with momentum 0.001 and learning rate 0.02, then repeats zeroing gradients, computing outputs and loss, backpropagating, and stepping the optimizer. Testing loss and binary accuracy can be inspected during training, while frequency-weighted loss gives more influence to common names. section 5.2.5 section 5.2.5
>>> from torch.optim import SGD
>>> hyperparams = {'momentum': 0.001, 'lr': 0.02} #1
>>> optimizer = SGD(
... model.parameters(), **hyperparams) #2
>>> optimizer
SGD (
Parameter Group 0
dampening: 0
differentiable: False
foreach: None
lr: 0.02
maximize: False
momentum: 0.001
nesterov: False
weight_decay: 0
)
Inspect the PyTorch SGD configuration, including learning rate, momentum, dampening, and weight decay. section 5.2.5
>>> from tqdm import tqdm >>> num_epochs = 200 >>> pbar_epochs = tqdm(range(num_epochs), desc='Epoch:', total=num_epochs, ascii=' =') >>> for epoch in pbar_epochs: ... optimizer.zero_grad() #1 ... outputs = model(X_train) ... loss_train = loss_func_train(outputs, y_train) #2 ... loss_train.backward() #3 ... optimizer.step() #4 Epoch:: 100%|================================| 200/200 [00:04<00:00, 42.84it/s] 96.26it/s]
Inspect the compact training loop that computes loss, backpropagates gradients, and updates model parameters across epochs. section 5.2.5
