Learning rate
A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
Build AI Drug Discovery Pipelines
A learning rate is an optimization step-size configuration that controls the magnitude of parameter updates during iterative learning.
- How it is used
- Uses learning rates to control iterative model updates and shrink each tree's contribution in gradient boosting.
- How deep it goes
- Explains gradient-scaled updates, compares scheduler choices, and describes the tradeoff between boosting learning rate and tree count.
- Examples
- Illustrates rate selection with solubility regression and an Adam-trained binding-affinity surrogate for docking.
- What you can do after
- Apply learning-rate settings with scikit-learn's SGDRegressor and PyTorch's Adam, and monitor losses to inform tuning.
Read how Build AI Drug Discovery Pipelines explains learning rate →
Grokking Machine Learning
A learning rate is a small positive training parameter that controls the size of parameter updates made during model optimization.
- How it is used
- Controls how far linear-regression parameters or boosting predictions move during each training update.
- How deep it goes
- Defines the update-scaling role and demonstrates it in formulas and code, without deriving a general optimization theory.
- Examples
- Uses house-price line fitting with 0.01 and app-engagement gradient boosting with 0.8.
- What you can do after
- Apply it via Python linear-regression and Scikit-Learn gradient-boosting implementations.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
A learning rate is a training hyperparameter that controls the size of parameter updates made while fitting a model.
- How it is used
- Controls weight-update size during backpropagation training of a feed-forward neural network.
- How deep it goes
- Shows the executable update using cached outputs and neuron deltas, while deferring the partial-derivative justification.
- Examples
- Uses iris classification with a learning rate of 0.3 and wine classification with 0.9.
- What you can do after
- Build it from scratch in Swift and tune the learning rate against held-out classification results.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A learning rate is a hyperparameter that scales the size of successive updates made while fitting a model.
- How it is used
- Controls update size while fitting SGD and gradient-boosting models, and serves as a central manual-tuning parameter.
- How deep it goes
- Explains update-size effects, boosting-tree contributions, and practical tree-count and depth trade-offs without deriving optimization mathematics.
- Examples
- Uses out-of-core logistic regression batches, a custom gradient-boosting classifier, and a 1,000-tree learning-rate search.
- What you can do after
- Apply it by tuning update size alongside tree count and depth, then evaluating prediction performance on unseen data.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains learning rate →
Building Specialized AI Systems
A learning rate is the optimization step-size parameter that controls how much model weights change during training.
- How it is used
- Controls how quickly model weights adapt during pre-training and especially domain-specific fine-tuning.
- How deep it goes
- Explains the gradient-descent step, numerical ranges, and too-large versus too-small failure modes without deriving gradients.
- Examples
- Configures 2e-4 for LoRA-based analyst-report summarization with batch size, accumulation, warmup, and a linear scheduler.
- What you can do after
- Operate/configure fine-tuning by selecting a rate that balances adaptation speed, stability, and preservation of learned representations.
Read how Building Specialized AI Systems explains learning rate →
Inside Deep Learning: Math, Algorithms, Models
The step-size parameter controlling how far an optimization update moves in the direction suggested by the gradient.
- How it is used
- Controls gradient-update size when minimizing a function or training neural-network parameters.
- How deep it goes
- Works through a manual PyTorch update, an SGD loop, and scheduling strategies without deriving an optimal initial rate.
- Examples
- Uses a scalar minimization loop, an exponential-decay calculation, a validation-plateau scheduler, and separate GAN optimizer settings.
- What you can do after
- Operate/configure it through PyTorch optimizer learning rates and schedulers, choosing smaller steps when updates overshoot or validation loss plateaus.
Read how Inside Deep Learning: Math, Algorithms, Models explains learning rate →
Applied Reinforcement Learning
A learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.
- How it is used
- Controls the strength of error-driven parameter updates in reinforcement learning and language-model fine-tuning.
- How deep it goes
- Explains error scaling and a linear TD update, then compares learning rates across fine-tuning stages and diagnostic runs.
- Examples
- Compares three pipeline rates and reports LoRA and GRPO runs with nearly unchanged KL and no accuracy improvement.
- What you can do after
- Diagnose insufficient learning by checking KL movement and comparing the learning rate with the update method.
Read how Applied Reinforcement Learning explains learning rate →
Ensemble Methods for Machine Learning
learning_rate is a boosting hyperparameter that scales the contribution of each successive weak learner and controls the rate at which ensemble complexity grows.
- How it is used
- Controls each tree or weak learner’s contribution while tuning boosted ensembles for generalization.
- How deep it goes
- Explains additive scaling, weight effects, tradeoffs, and cross-validation selection without deriving boosting objectives.
- Examples
- Uses breast-cancer classification with AdaBoost, GradientBoostingClassifier, LightGBM, and XGBoost validation sweeps.
- What you can do after
- Apply it via scikit-learn, LightGBM, or XGBoost and select candidate values using cross-validation.
Read how Ensemble Methods for Machine Learning explains learning rate →
Natural Language Processing in Action, Second Edition
A learning rate is a numerical step-size setting that controls how far an optimization algorithm moves parameters in response to a gradient.
- How it is used
- Controls parameter-update size during neural-network training and serves as a principal optimizer hyperparameter.
- How deep it goes
- Explains update intuition, convergence, overshooting, batch effects, and optimizer choices, but defers mathematical derivation.
- Examples
- Uses PyTorch SGD training with lr 0.02 and compares learning rates in documented RNN experiments.
- What you can do after
- Tune optimizer behavior by comparing convergence, loss, accuracy, and training speed across learning-rate settings.
Read how Natural Language Processing in Action, Second Edition explains learning rate →
Grokking Machine Learning, Second Edition
A learning rate is a small positive step size that controls the magnitude of parameter updates during iterative learning.
- How it is used
- Controls the size of parameter updates while iterative algorithms fit lines, classify points, or train neural networks.
- How deep it goes
- Explains the intuition, update equations, pseudocode, and repeated training behavior, but does not develop a systematic tuning strategy.
- Examples
- Uses house-price line fitting, square and absolute tricks, a Python update function, and a 0.01 learning rate in linear regression.
- What you can do after
- Apply it by inserting a small step size into linear-regression, perceptron, and neural-network parameter updates.
Read how Grokking Machine Learning, Second Edition explains learning rate →
Deep Learning with R, Third Edition
A learning rate is the step-size hyperparameter controlling how far model parameters move during an optimization update.
- How it is used
- Configures the magnitude of parameter updates during mini-batch gradient descent and helps diagnose stalled training.
- How deep it goes
- Defines the update role, gives the weight-update equation, and demonstrates tuning, but defers systematic schedules and advanced optimization details.
- Examples
- Uses manual weight updates, Keras optimizers, and MNIST runs comparing learning rates of 1 and 1e-2.
- What you can do after
- Operate/configure training by setting the learning rate and adjusting it when updates overshoot or optimization appears stalled.
Read how Deep Learning with R, Third Edition explains learning rate →
Deep Learning with PyTorch, Second Edition
The step-size hyperparameter that controls how much model parameters are changed per update during optimization.
- How it is used
- Controls the size of parameter updates when fitting a model and provides a tuning lever when training is unstable or slow.
- How deep it goes
- Works through gradient-scaled updates, divergence, slow progress, and input normalization; introduces adaptive optimizers without detailing their variants.
- Examples
- Compares training runs at 1e-2 and 1e-4, then rescales inputs and retries 1e-2.
- What you can do after
- Diagnose training instability by inspecting loss and gradient scales, adjusting the learning rate, and normalizing inputs.
Read how Deep Learning with PyTorch, Second Edition explains learning rate →
LLM Customization and Fine-Tuning
The learning rate is a training hyperparameter that controls the size of parameter updates during optimization.
- How it is used
- Controls update size to balance adaptation, training stability, and retention of pretrained capabilities during fine-tuning.
- How deep it goes
- Provides starting ranges, a one-variable sweep, loss-curve diagnostics, and stopping criteria for hyperparameter iteration.
- Examples
- Compares an illustrative six-run SFT log and a DPO run using 270 training preference pairs.
- What you can do after
- Operate/configure learning rates for fine-tuning and diagnose underfitting or instability using training and validation loss.
Read how LLM Customization and Fine-Tuning explains learning rate →
Machine Learning Platform Engineering
A learning rate is a training hyperparameter that controls the step size used to update model parameters during optimization.
- How it is used
- Balances training efficiency, resource utilization, accuracy, and training time in production model configuration.
- How deep it goes
- Explains learning-rate tuning as an iterative response to convergence problems but does not detail optimization internals.
- Examples
- Uses an Ultralytics YOLO ID card detector and revises learning rate, batch size, or epochs when training fails to converge.
- What you can do after
- Diagnose non-converging training by adjusting the learning rate alongside batch size and epochs.
Read how Machine Learning Platform Engineering explains learning rate →
Math and Architectures of Deep Learning
The step-size setting that controls how far model parameters move on each optimization update, possibly chosen separately for each parameter dimension.
- How it is used
- Controls parameter-update size while training neural networks with gradient descent.
- How deep it goes
- Explains gradient direction, Taylor approximation, Hessian context, and PyTorch configuration, while deferring fuller optimization details.
- Examples
- Uses a Statsville height-and-weight classifier, an SGD setting of 0.02, and epoch-by-epoch loss reduction.
- What you can do after
- Apply it via PyTorch by configuring SGD and interpreting its effect on training progress and overshooting risk.
Read how Math and Architectures of Deep Learning explains learning rate →
Evolutionary Deep Learning: Genetic algorithms and neural networks
A learning rate is the step-size parameter that determines how strongly model parameters are updated during training.
- How it is used
- Guides training-speed and model-quality experiments within broader hyperparameter tuning.
- How deep it goes
- Demonstrates configuration and comparative reruns, emphasizing observed training effects rather than update-equation derivations.
- Examples
- Uses notebook values from 3.5e-06 to 3.5e-01, GAN generator and discriminator settings, and agent-learning exercises.
- What you can do after
- Diagnose training behavior by changing learning-rate settings and observing effects on models, GANs, or agents.
Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains learning rate →
Fight Fraud with Machine Learning
A learning rate is a training setting that controls the step size used when updating model parameters.
- How it is used
- Configures the step size used while training fraud-classification neural networks.
- How deep it goes
- Explains gradient-scaled updates, risks of extreme values, and optimizer usage, but does not prescribe a tuning procedure.
- Examples
- Uses Adam configurations with learning rates of 0.01 for an MLP and 0.001 for an audio transformer.
- What you can do after
- Apply it via a PyTorch optimizer and observe its effect through training loss and validation metrics.
Read how Fight Fraud with Machine Learning explains learning rate →
Transformers in Action
Learning rate is an optimization hyperparameter that controls the magnitude of parameter updates during training.
- How it is used
- Controls parameter-update magnitude while configuring and evaluating language-model training.
- How deep it goes
- Explains gradient-scaled updates and high-versus-low trade-offs, then demonstrates configuration with scheduling but stops short of tuning procedures.
- Examples
- Uses convergence figures at 0.1 and 0.01 plus Gemma 3 270M English–Spanish fine-tuning configuration.
- What you can do after
- Operate/configure training with a selected learning rate, scheduler, warmup ratio, and resource-aware fine-tuning setup.
Data Without Labels
A learning rate is a scalar hyperparameter that controls the size of parameter updates during iterative training.
- How it is used
- Uses the learning rate to control corrective weight updates during neural-network training.
- How deep it goes
- Defines update size and training tradeoffs, then applies the parameter to a Boltzmann weight-update rule.
- Examples
- Illustrates corrective network steps and visible–hidden correlation differences in Boltzmann weight updates.
- What you can do after
- Evaluate how update size affects accuracy and training time in gradient-based neural-network training.
Grokking AI Algorithms, Second Edition
A learning rate is a training hyperparameter that controls the step size used to update model parameters.
- How it is used
- Controls parameter-update size during neural-network and language-model training.
- How deep it goes
- Explains update-size effects and gradient-descent placement, then stops short of detailed optimization theory.
- Examples
- Uses candidate values 0.01, 0.001, and 0.0001 with manual, grid, and random search.
- What you can do after
- Operate/configure training by selecting and adjusting a learning rate through observed results or systematic search.
Read how Grokking AI Algorithms, Second Edition explains learning rate →
LLMs in Production
A learning rate is a training hyperparameter that controls the magnitude of parameter updates during optimization.
- How it is used
- Uses the learning rate to control training progress and recover from misleading early convergence.
- How deep it goes
- Explains parameter-update magnitude and gives a checkpoint-based adjustment procedure, stopping short of optimizer internals.
- Examples
- Uses a T5 XXL run that repeatedly appeared converged before completing its dataset.
- What you can do after
- Diagnose suspicious early convergence and adjust the learning rate through checkpointed recovery.
Designing Deep Learning Systems: A guide for software engineers
A learning rate is a hyperparameter that controls the step size used to update model parameters during optimization.
- How it is used
- Controls parameter-update behavior within the model training loop and serves as a tunable input to the optimization algorithm.
- How deep it goes
- Explains its workflow position and effects on training speed and accuracy, but does not derive the update rule or internal optimizer mechanics.
- Examples
- Demonstrates tuning through TensorFlow Playground and a spiral-classification configuration with specified layers, activation, batch size, and regularization.
- What you can do after
- Apply it via TensorFlow Playground experiments that compare learning behavior, training time, and classification performance.
Read how Designing Deep Learning Systems: A guide for software engineers explains learning rate →
Timeless Algorithms, The Foundational Ideas
A learning rate controls the size of parameter updates during iterative model training.
- How it is used
- Controls the size of gradient-descent parameter updates during network training.
- How deep it goes
- Defines the update equation and works through one-observation updates, then relates them to mini-batches and epochs.
- Examples
- Uses a seven-parameter network update with learning rate 0.1 and gradients from one observation.
- What you can do after
- Understand how a selected learning rate scales gradient-based parameter changes during training.
Read how Timeless Algorithms, The Foundational Ideas explains learning rate →
Also covered in
- Machine Learning Algorithms in Depth
- Learn Generative AI with PyTorch
- Build a Robo-Advisor with Python (From Scratch)
- Sutskever's List
- Machine Learning System Design
- Deep Learning with JAX
- Reinforcement Learning from Human Feedback
- Quantization and Fast InferenceIn early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- Software Testing with Generative AI
- Advanced Algorithms and Data Structures
- Transfer Learning for Natural Language Processing
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- warmupconfigured by · Learning-rate warmup is an initial training schedule that gradually increases the learning rate from a low value toward its target value.
- hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
- gradient descentconfigures · Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
- learning rate scheduleconfigured by · A learning rate schedule is a rule that determines how an optimizer’s learning rate changes during training according to progress or other conditions.
- stochastic gradient descentconfigures · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.
Questions about learning rate
What is learning rate?
A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
Which Manning books explain learning rate?
Build AI Drug Discovery Pipelines, Grokking Machine Learning, Classic Computer Science Problems in Swift: Essential techniques for practicing programmers, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Building Specialized AI Systems, Inside Deep Learning: Math, Algorithms, Models, Applied Reinforcement Learning, Ensemble Methods for Machine Learning and 15 more.
What should I know before learning learning rate?
Manning books that teach learning rate usually introduce objective function, accuracy, gradient and language model first.
What is learning rate a kind of?
Learning rate is a kind of hyperparameter. A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
What is learning rate also called?
Learning rate is also called LR or step size.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






















