A learning rate controls how far model parameters move during each optimization update. In gradient descent, the update subtracts the gradient multiplied by a small learning rate. This lets parameters with a larger effect on the loss receive larger changes, while keeping each step cautious. section 5.4.1
# In[15]:
def training_loop(n_epochs, learning_rate, params, t_u, t_c):
for epoch in range(1, n_epochs + 1):
w, b = params
t_p = model(t_u, w, b) #1
loss = loss_fn(t_p, t_c)
grad = grad_fn(t_u, t_c, t_p, w, b) #2
params = params - learning_rate * grad
print('Epoch %d, Loss %f' % (epoch, float(loss))) #3
return params
The training loop makes the step size visible: `params = params - learning_rate * grad` runs once per epoch. section 5.4.3
When the step is too large—or too small
With a learning rate of `1e-2`, a weight-and-bias fitting loop overshoots repeatedly until its loss becomes `inf`. Lowering the rate to `1e-4` stabilizes training, but updates then become small and progress slows. Changing the learning rate is a form of hyperparameter tuning: it controls training rather than being a parameter fitted by that same optimization. section 5.4.3
The gradient matters as much as the chosen rate. If the weight’s gradient is much larger than the bias’s, one shared rate may be unstable for one parameter yet ineffective for the other. Rescaling the input by `0.1` brings the gradients closer together; the loop can then use `1e-2` without blowing up. section 5.4.4
An optimizer can also adjust learning rates during training. Adam maintains a separate rate for each parameter and updates it automatically. For a SegFormer segmentation model, an AdamW optimizer is configured with `torch.optim.AdamW(model.parameters(), lr=0.00006)`. A useful experiment is to compare learning rates and optimizers by their training dynamics and final performance. section 15.7.2 section 15.10
