Learning rate

As explained in Math and Architectures of Deep Learning

The step-size setting that controls how far model parameters move on each optimization update, possibly chosen separately for each parameter dimension.

A learning rate is the step-size setting used when optimization moves neural-network parameters such as weights and biases. Training repeatedly adjusts these parameters to reduce a loss and bring predictions closer to known targets. In gradient descent, the update moves opposite the gradient; the learning rate controls how far that move goes. section 8.4 section 8.4.3

The gradient supplies the direction of steepest local decrease, while the learning rate supplies the scale. A larger value can make optimization progress faster, but it can also overshoot the minimum. The rate is therefore a tunable hyperparameter rather than a learned model parameter. section 8.4.3

A typical PyTorch training step applies the learning rate through the optimizer:

• Create a loss function and an SGD optimizer, setting the rate with `optim.SGD(model.parameters(), lr=0.02)`. section 9.2.3
• For each minibatch, compute model scores and loss, clear old gradients, backpropagate with `loss.backward()`, and update parameters with `optimizer.step()`. section 9.2.3
• Repeat the loop across epochs while monitoring whether the loss decreases and predictions improve. section 9.2.3

Classifying Statsville residents

A neural network receives height and weight and predicts whether each resident is a man, woman, or child. Its SGD optimizer uses a learning rate of 0.02; successive epochs reduce the loss, with most training examples classified correctly after the first epoch. section 9.2.3

Equation 8.15

Equation 8.15 connects the learning rate to the local gradient-descent step and the direction producing the largest decrease. section 8.4.3

How Math and Architectures of Deep Learning covers it

How deep it goes

Explains gradient direction, Taylor approximation, Hessian context, and PyTorch configuration, while deferring fuller optimization details.

Examples

Uses a Statsville height-and-weight classifier, an SGD setting of 0.02, and epoch-by-epoch loss reduction.

What you can do after

Apply it via PyTorch by configuring SGD and interpreting its effect on training progress and overshooting risk.

How it connects

Connects gradient descent, loss functions, backpropagation, stochastic gradient descent, and Hessian matrices.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA learning rate is a small positive training parameter that controls the size of parameter updates made during model optimization.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersA learning rate is a training hyperparameter that controls the size of parameter updates made while fitting a model.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA learning rate is a hyperparameter that scales the size of successive updates made while fitting a model.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsThe step-size parameter controlling how far an optimization update moves in the direction suggested by the gradient.
• 

  liveBookmanning.com
  Ensemble Methods for Machine Learninglearning_rate is a boosting hyperparameter that scales the contribution of each successive weak learner and controls the rate at which ensemble complexity grows.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA learning rate is a numerical step-size setting that controls how far an optimization algorithm moves parameters in response to a gradient.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA learning rate is the step-size hyperparameter controlling how far model parameters move during an optimization update.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe step-size hyperparameter that controls how much model parameters are changed per update during optimization.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA learning rate is a training hyperparameter that controls the step size used to update model parameters during optimization.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA learning rate is the step-size parameter that determines how strongly model parameters are updated during training.
• 

  liveBookmanning.com
  Transformers in ActionLearning rate is an optimization hyperparameter that controls the magnitude of parameter updates during training.
• 

  liveBookmanning.com
  Data Without LabelsA learning rate is a scalar hyperparameter that controls the size of parameter updates during iterative training.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA learning rate is a training hyperparameter that controls the step size used to update model parameters.
• 

  liveBookmanning.com
  LLMs in ProductionA learning rate is a training hyperparameter that controls the magnitude of parameter updates during optimization.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA learning rate is a hyperparameter that controls the step size used to update model parameters during optimization.

Compare how 16 books explain learning rate →

Related concepts

• hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
• gradient descentconfigures · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• learning rate scheduleconfigured by · A learning rate schedule is a rule or function that determines how an optimizer’s learning rate changes during training.
• stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• cosine annealingconfigured by · Cosine annealing is a learning-rate scheduling method that varies the learning rate over training according to a cosine-shaped curve.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.