liveBook · Manning

Learning rate

As explained in Grokking Machine Learning, Second Edition

A learning rate is a small positive step size that controls the magnitude of parameter updates during iterative learning.

A learning rate is a small positive number chosen before training. It controls how far a model moves when it updates its parameters, so learning happens in small steps rather than sudden jumps. section 3.13 section 10.16

Moving a house-price line

Suppose a line predicts house prices from the number of rooms. If the actual price p is above the prediction p̂, the difference p − p̂ is positive, so the line moves upward. Multiplying that difference by the learning rate keeps the adjustment small. The same idea updates the slope using h times the number of rooms and the prediction error. section 3.13

def square_trick(base_price, price_per_room, num_rooms, price, learning_rate):
    predicted_price = base_price + price_per_room*num_rooms               #A
    base_price += learning_rate*(price-predicted_price) #B
    price_per_room += learning_rate*num_rooms*(price-predicted_price)     #C
    return price_per_room, base_price                 

See the learning rate applied separately to the base price and price-per-room parameter in a small update function. section 3.13

A learning rate works with repeated updates. In linear regression, many small steps gradually move a line closer to the data; in a perceptron, it is one of the two main training hyperparameters alongside the number of epochs. section 3.17 section 5.15

The learning rate appears in several update rules:

  • Square trick: add h(y − ŷ) to the bias and hxi(y − ŷ) to each weight. section 3.19
  • Absolute trick: add or subtract h and h times the feature, depending on whether the point lies above or below the line. section 3.14
  • Neural networks: use the learning rate as the size of each training step and tune it with other hyperparameters. section 10.16

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains the intuition, update equations, pseudocode, and repeated training behavior, but does not develop a systematic tuning strategy.
Examples
Uses house-price line fitting, square and absolute tricks, a Python update function, and a 0.01 learning rate in linear regression.
What it assumes
Assumes familiarity with predictions, parameters, and iterative training while building the update intuition from points, lines, and price differences.
What you can do after
Apply it by inserting a small step size into linear-regression, perceptron, and neural-network parameter updates.
How it connects
Connects learning rate to linear regression, the perceptron algorithm, neural networks, epochs, and hyperparameters.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.