liveBook · Manning

Learning rate

As explained in Timeless Algorithms, The Foundational Ideas

A learning rate controls the size of parameter updates during iterative model training.

A learning rate, η, sets how far parameters move in response to their gradients during gradient descent. It is selected before training, not calculated by the network. section 11.3.5

θnew=θold-η∂L∂θ

The update subtracts the gradient scaled by η from the current parameter. section 11.3.5

One update

With η = 0.1, a parameter at 1.2 with gradient −0.24 becomes 1.224. A negative gradient increases the parameter; a positive gradient decreases it. The example updates seven network parameters from one observation. section 11.3.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the update equation and works through one-observation updates, then relates them to mini-batches and epochs.
Examples
Uses a seven-parameter network update with learning rate 0.1 and gradients from one observation.
What you can do after
Understand how a selected learning rate scales gradient-based parameter changes during training.
How it connects
Connects gradient descent and backpropagation with loss functions, mini-batches, and epochs.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.