A learning rate is the optimizer’s step size: it controls how far model weights move in response to an error gradient. Think of it as ski velocity on a loss surface—larger steps can reach a minimum faster, but steps that are too large may repeatedly overshoot it. section 5.2.5 section 5.3.2
Use the learning rate as a practical training-control knob:
- Set it as an optimizer hyperparameter; PyTorch optimizers require a learning rate. section 5.2.5
- Lower it when updates are too aggressive or the model overshoots useful minima; raise it when progress is needlessly slow. section 5.2.5
- Tune it alongside momentum, batch size, dropout, model depth, and other hyperparameters rather than treating one value as universally best. section 5.2.5 section 8.4.4
- Use mini-batches to balance noisy individual-sample gradients with the larger progress of full-batch updates; batch sizes around 16–64 are suggested for many neural-network problems. section 5.3.2
- Consider adaptive optimizers such as RMSprop when estimating changing parameter slopes more effectively could speed CNN training. section 7.5.4
A PyTorch training setting
Configure SGD with a learning rate of 0.02 and momentum of 0.001, then update the model after backpropagation: from torch.optim import SGD hyperparams = {'momentum': 0.001, 'lr': 0.02} optimizer = SGD(model.parameters(), **hyperparams) for epoch in range(200): optimizer.zero_grad() outputs = model(X_train) loss = loss_func_train(outputs, y_train) loss.backward() optimizer.step() The same training pattern can be evaluated by recording training loss, test loss, and test accuracy across epochs. section 5.2.5 section 5.2.5
import pandas as pd
import jsonlines
with jsonlines.open('experiments.jsonl') as fin:
lines = list(fin)
df = pd.DataFrame(lines)
df.to_csv('experiments.csv')
cols = 'rnn_type epochs lr num_layers dropout epoch_time test_loss'
cols = cols.split()
df[cols].round(2).sort_values('test_loss').head(10)epochs lr num_layers dropout epoch_time test_loss
37 12 2.0 2 0.2 35.43 5.23
28 12 2.0 1 0.0 22.66 5.23
49 32 0.5 2 0.0 32.35 5.22
57 32 0.5 2 0.2 35.50 5.22
38 12 2.0 3 0.2 46.14 5.21
50 32 0.5 3 0.0 37.36 5.20
52 32 2.0 1 0.0 22.90 5.10
55 32 2.0 5 0.0 56.23 5.09
53 32 2.0 2 0.0 32.49 5.06
54 32 2.0 3 0.0 38.78 5.04
Inspect recorded RNN experiments sorted by test loss to compare learning rates with depth, dropout, epochs, and training time. section 8.4.4
