Learning rate
A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
Grokking Machine Learning
A learning rate is a small positive training parameter that controls the size of parameter updates made during model optimization.
- How it is used
- Controls how far linear-regression parameters or boosting predictions move during each training update.
- How deep it goes
- Defines the update-scaling role and demonstrates it in formulas and code, without deriving a general optimization theory.
- Examples
- Uses house-price line fitting with 0.01 and app-engagement gradient boosting with 0.8.
- What you can do after
- Apply it via Python linear-regression and Scikit-Learn gradient-boosting implementations.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
A learning rate is a training hyperparameter that controls the size of parameter updates made while fitting a model.
- How it is used
- Controls weight-update size during backpropagation training of a feed-forward neural network.
- How deep it goes
- Shows the executable update using cached outputs and neuron deltas, while deferring the partial-derivative justification.
- Examples
- Uses iris classification with a learning rate of 0.3 and wine classification with 0.9.
- What you can do after
- Build it from scratch in Swift and tune the learning rate against held-out classification results.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A learning rate is a hyperparameter that scales the size of successive updates made while fitting a model.
- How it is used
- Controls update size while fitting SGD and gradient-boosting models, and serves as a central manual-tuning parameter.
- How deep it goes
- Explains update-size effects, boosting-tree contributions, and practical tree-count and depth trade-offs without deriving optimization mathematics.
- Examples
- Uses out-of-core logistic regression batches, a custom gradient-boosting classifier, and a 1,000-tree learning-rate search.
- What you can do after
- Apply it by tuning update size alongside tree count and depth, then evaluating prediction performance on unseen data.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains learning rate →
Inside Deep Learning: Math, Algorithms, Models
The step-size parameter controlling how far an optimization update moves in the direction suggested by the gradient.
- How it is used
- Controls optimization step size when minimizing functions and training neural-network parameters.
- How deep it goes
- Defines the update rule, explains overshooting and divergence, and maps manual updates to PyTorch optimizers; it stops short of scheduling methods.
- Examples
- Uses scalar minimization from −3.5 to 2.0, PyTorch SGD, and GAN training with AdamW at η = 0.0001.
- What you can do after
- Apply it via PyTorch by configuring SGD, AdamW, or manual updates and selecting values through Optuna hyperparameter searches.
Read how Inside Deep Learning: Math, Algorithms, Models explains learning rate →
Ensemble Methods for Machine Learning
learning_rate is a boosting hyperparameter that scales the contribution of each successive weak learner and controls the rate at which ensemble complexity grows.
- How it is used
- Controls each tree or weak learner’s contribution while tuning boosted ensembles for generalization.
- How deep it goes
- Explains additive scaling, weight effects, tradeoffs, and cross-validation selection without deriving boosting objectives.
- Examples
- Uses breast-cancer classification with AdaBoost, GradientBoostingClassifier, LightGBM, and XGBoost validation sweeps.
- What you can do after
- Apply it via scikit-learn, LightGBM, or XGBoost and select candidate values using cross-validation.
Read how Ensemble Methods for Machine Learning explains learning rate →
Natural Language Processing in Action, Second Edition
A learning rate is a numerical step-size setting that controls how far an optimization algorithm moves parameters in response to a gradient.
- How it is used
- Controls parameter-update size during neural-network training and serves as a principal optimizer hyperparameter.
- How deep it goes
- Explains update intuition, convergence, overshooting, batch effects, and optimizer choices, but defers mathematical derivation.
- Examples
- Uses PyTorch SGD training with lr 0.02 and compares learning rates in documented RNN experiments.
- What you can do after
- Tune optimizer behavior by comparing convergence, loss, accuracy, and training speed across learning-rate settings.
Read how Natural Language Processing in Action, Second Edition explains learning rate →
Deep Learning with R, Third Edition
A learning rate is the step-size hyperparameter controlling how far model parameters move during an optimization update.
- How it is used
- Configures the magnitude of parameter updates during mini-batch gradient descent and helps diagnose stalled training.
- How deep it goes
- Defines the update role, gives the weight-update equation, and demonstrates tuning, but defers systematic schedules and advanced optimization details.
- Examples
- Uses manual weight updates, Keras optimizers, and MNIST runs comparing learning rates of 1 and 1e-2.
- What you can do after
- Operate/configure training by setting the learning rate and adjusting it when updates overshoot or optimization appears stalled.
Read how Deep Learning with R, Third Edition explains learning rate →
Deep Learning with PyTorch, Second Edition
The step-size hyperparameter that controls how much model parameters are changed per update during optimization.
- How it is used
- Controls parameter-update size while fitting a model by gradient descent.
- How deep it goes
- Works through update code and contrasting training runs, but does not detail the mechanisms of Adam variants.
- Examples
- Contrasts divergent and slow weight-and-bias training, then configures AdamW for SegFormer segmentation.
- What you can do after
- Diagnose/debug with it by reducing oversize updates and checking whether input scaling permits a useful shared learning rate.
Read how Deep Learning with PyTorch, Second Edition explains learning rate →
Machine Learning Platform Engineering
A learning rate is a training hyperparameter that controls the step size used to update model parameters during optimization.
- How it is used
- Balances training efficiency, resource utilization, accuracy, and training time in production model configuration.
- How deep it goes
- Explains learning-rate tuning as an iterative response to convergence problems but does not detail optimization internals.
- Examples
- Uses an Ultralytics YOLO ID card detector and revises learning rate, batch size, or epochs when training fails to converge.
- What you can do after
- Diagnose non-converging training by adjusting the learning rate alongside batch size and epochs.
Read how Machine Learning Platform Engineering explains learning rate →
Math and Architectures of Deep Learning
The step-size setting that controls how far model parameters move on each optimization update, possibly chosen separately for each parameter dimension.
- How it is used
- Controls parameter-update size while training neural networks with gradient descent.
- How deep it goes
- Explains gradient direction, Taylor approximation, Hessian context, and PyTorch configuration, while deferring fuller optimization details.
- Examples
- Uses a Statsville height-and-weight classifier, an SGD setting of 0.02, and epoch-by-epoch loss reduction.
- What you can do after
- Apply it via PyTorch by configuring SGD and interpreting its effect on training progress and overshooting risk.
Read how Math and Architectures of Deep Learning explains learning rate →
Evolutionary Deep Learning: Genetic algorithms and neural networks
A learning rate is the step-size parameter that determines how strongly model parameters are updated during training.
- How it is used
- Guides training-speed and model-quality experiments within broader hyperparameter tuning.
- How deep it goes
- Demonstrates configuration and comparative reruns, emphasizing observed training effects rather than update-equation derivations.
- Examples
- Uses notebook values from 3.5e-06 to 3.5e-01, GAN generator and discriminator settings, and agent-learning exercises.
- What you can do after
- Diagnose training behavior by changing learning-rate settings and observing effects on models, GANs, or agents.
Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains learning rate →
Transformers in Action
Learning rate is an optimization hyperparameter that controls the magnitude of parameter updates during training.
- How it is used
- Controls parameter-update magnitude while configuring and evaluating language-model training.
- How deep it goes
- Explains gradient-scaled updates and high-versus-low trade-offs, then demonstrates configuration with scheduling but stops short of tuning procedures.
- Examples
- Uses convergence figures at 0.1 and 0.01 plus Gemma 3 270M English–Spanish fine-tuning configuration.
- What you can do after
- Operate/configure training with a selected learning rate, scheduler, warmup ratio, and resource-aware fine-tuning setup.
Data Without Labels
A learning rate is a scalar hyperparameter that controls the size of parameter updates during iterative training.
- How it is used
- Uses the learning rate to control corrective weight updates during neural-network training.
- How deep it goes
- Defines update size and training tradeoffs, then applies the parameter to a Boltzmann weight-update rule.
- Examples
- Illustrates corrective network steps and visible–hidden correlation differences in Boltzmann weight updates.
- What you can do after
- Evaluate how update size affects accuracy and training time in gradient-based neural-network training.
Grokking AI Algorithms, Second Edition
A learning rate is a training hyperparameter that controls the step size used to update model parameters.
- How it is used
- Controls parameter-update size during neural-network and language-model training.
- How deep it goes
- Explains update-size effects and gradient-descent placement, then stops short of detailed optimization theory.
- Examples
- Uses candidate values 0.01, 0.001, and 0.0001 with manual, grid, and random search.
- What you can do after
- Operate/configure training by selecting and adjusting a learning rate through observed results or systematic search.
Read how Grokking AI Algorithms, Second Edition explains learning rate →
LLMs in Production
A learning rate is a training hyperparameter that controls the magnitude of parameter updates during optimization.
- How it is used
- Uses the learning rate to control training progress and recover from misleading early convergence.
- How deep it goes
- Explains parameter-update magnitude and gives a checkpoint-based adjustment procedure, stopping short of optimizer internals.
- Examples
- Uses a T5 XXL run that repeatedly appeared converged before completing its dataset.
- What you can do after
- Diagnose suspicious early convergence and adjust the learning rate through checkpointed recovery.
Designing Deep Learning Systems: A guide for software engineers
A learning rate is a hyperparameter that controls the step size used to update model parameters during optimization.
- How it is used
- Controls parameter-update behavior within the model training loop and serves as a tunable input to the optimization algorithm.
- How deep it goes
- Explains its workflow position and effects on training speed and accuracy, but does not derive the update rule or internal optimizer mechanics.
- Examples
- Demonstrates tuning through TensorFlow Playground and a spiral-classification configuration with specified layers, activation, batch size, and regularization.
- What you can do after
- Apply it via TensorFlow Playground experiments that compare learning behavior, training time, and classification performance.
Read how Designing Deep Learning Systems: A guide for software engineers explains learning rate →
Related concepts
- hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
- gradient descentconfigures · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
- learning rate scheduleconfigured by · A learning rate schedule is a rule or function that determines how an optimizer’s learning rate changes during training.
- stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
- cosine annealingconfigured by · Cosine annealing is a learning-rate scheduling method that varies the learning rate over training according to a cosine-shaped curve.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.