A model learns by changing its parameters so its predictions better match known answers. Gradient descent makes those changes by measuring how the loss responds to each parameter, then stepping in the opposite direction. Picture a model with two adjustment knobs, weight w and bias b: if turning one knob raises the loss, turn it the other way. Repeat the adjustments as training proceeds. section 5.4 section 5.4.1
A step from a measured slope
For a simple model, estimate the slope for w by evaluating the loss at w + delta and w - delta, then dividing their difference by 2 × delta. Update with `w = w - learning_rate * slope_w`; do the same for b. The learning rate keeps each step modest: a large adjustment can pass the low-loss region and make the loss rise again. Recalculate the slopes after each step because they describe the loss near the current parameter values. section 5.4.1 section 5.4
In a PyTorch training loop, the same idea becomes:
- Make predictions and calculate a loss, such as mean squared error. section 6.2.2
- Clear old gradients with `optimizer.zero_grad()`, then calculate current gradients with `loss.backward()`. section 6.2.2
- Call `optimizer.step()` to update the model’s parameters from their gradients. Repeat across epochs while checking training and validation loss. section 6.2.2
Fitting thermometer readings
Suppose a thermometer’s unknown readings are paired with Celsius temperatures. Fit `C = w * reading + b` by repeatedly reducing prediction loss. Scaling the readings with `t_un = 0.1 * t_u` helps convergence in this example. After 5,000 epochs, the fitted parameters for the scaled readings are approximately w = 5.3671 and b = -17.3012, close to a Fahrenheit-to-Celsius conversion after accounting for the scaling. The remaining loss does not reach zero; the readings may be noisy or may not fall exactly on a line. section 5.4.4
The approach also applies to neural networks. Differentiable linear layers and activations let gradients guide many weights and biases, not just two knobs. In PyTorch, `nn.Module.parameters()` supplies those trainable parameters to `optim.SGD`; `backward()` populates their gradients, and `step()` changes their values. An activation such as `nn.Tanh` transforms values but has no parameters of its own to update. section 6.1.6 section 6.2.2 section 6.3.2
# In[13]:
def training_loop(n_epochs, optimizer, model, loss_fn, t_u_train, t_u_val,
t_c_train, t_c_val):
for epoch in range(1, n_epochs + 1):
t_p_train = model(t_u_train) #1
loss_train = loss_fn(t_p_train, t_c_train)
t_p_val = model(t_u_val)
loss_val = loss_fn(t_p_val, t_c_val)
optimizer.zero_grad()
loss_train.backward() #2
optimizer.step()
if epoch == 1 or epoch % 1000 == 0:
print(f"Epoch {epoch}, Training loss {loss_train.item():.4f},"
f" Validation loss {loss_val.item():.4f}")
Inspect a complete training loop showing prediction, training and validation losses, gradient clearing, backpropagation, and the optimizer step. section 6.2.2
