liveBook · Manning

Learning rate

As explained in Fight Fraud with Machine Learning

A learning rate is a training setting that controls the step size used when updating model parameters.

A learning rate is a training setting that controls how far model parameters move during each update. In gradient descent, the optimizer scales each parameter’s gradient by this coefficient before adjusting the weights. section 4.1.1

Choose the value with training stability and convergence in mind:

  • Set it too high and updates may overshoot the optimum or fail to converge. section 4.1.1
  • Set it too low and convergence may become slow, or optimization may become stuck in a local minimum. section 4.1.1
  • Treat it as an optimizer hyperparameter; Adam can use one learning rate while adaptive methods adjust update behavior across weights. section 9.2.3 section 4.1.1

Fraud-model training configuration

A PyTorch MLP for licit-versus-illicit Bitcoin transactions sets LR = 0.01 and passes it to Adam. The training loop backpropagates loss, calls optimizer.step(), and tracks validation metrics across 1,000 epochs. A separate audio-transformer example uses Adam with a learning rate of 0.001 for 10 epochs. section 9.2.3 section 8.2.2

WDECAY = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=LR, weight_decay=WDECAY)  #A

Inspect the optimizer configuration that passes the learning rate and weight decay to Adam. section 9.2.3

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains gradient-scaled updates, risks of extreme values, and optimizer usage, but does not prescribe a tuning procedure.
Examples
Uses Adam configurations with learning rates of 0.01 for an MLP and 0.001 for an audio transformer.
What it assumes
Assumes gradients, loss, weights, and backpropagation, while introducing the learning-rate role within iterative optimization.
What you can do after
Apply it via a PyTorch optimizer and observe its effect through training loss and validation metrics.
How it connects
Connects gradient descent, stochastic gradient descent, Adam, Adagrad, backpropagation, loss functions, and model convergence.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.