liveBook · Manning

Learning rate

Also called LR, step size

A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.

How 23 Manning books explain it

Build AI Drug Discovery Pipelines

A learning rate is an optimization step-size configuration that controls the magnitude of parameter updates during iterative learning.

How it is used
Uses learning rates to control iterative model updates and shrink each tree's contribution in gradient boosting.
How deep it goes
Explains gradient-scaled updates, compares scheduler choices, and describes the tradeoff between boosting learning rate and tree count.
Examples
Illustrates rate selection with solubility regression and an Adam-trained binding-affinity surrogate for docking.
What you can do after
Apply learning-rate settings with scikit-learn's SGDRegressor and PyTorch's Adam, and monitor losses to inform tuning.

Read how Build AI Drug Discovery Pipelines explains learning rate →

Grokking Machine Learning

A learning rate is a small positive training parameter that controls the size of parameter updates made during model optimization.

How it is used
Controls how far linear-regression parameters or boosting predictions move during each training update.
How deep it goes
Defines the update-scaling role and demonstrates it in formulas and code, without deriving a general optimization theory.
Examples
Uses house-price line fitting with 0.01 and app-engagement gradient boosting with 0.8.
What you can do after
Apply it via Python linear-regression and Scikit-Learn gradient-boosting implementations.

Read how Grokking Machine Learning explains learning rate →

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

A learning rate is a training hyperparameter that controls the size of parameter updates made while fitting a model.

How it is used
Controls weight-update size during backpropagation training of a feed-forward neural network.
How deep it goes
Shows the executable update using cached outputs and neuron deltas, while deferring the partial-derivative justification.
Examples
Uses iris classification with a learning rate of 0.3 and wine classification with 0.9.
What you can do after
Build it from scratch in Swift and tune the learning rate against held-out classification results.

Read how Classic Computer Science Problems in Swift: Essential techniques for practicing programmers explains learning rate →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

A learning rate is a hyperparameter that scales the size of successive updates made while fitting a model.

How it is used
Controls update size while fitting SGD and gradient-boosting models, and serves as a central manual-tuning parameter.
How deep it goes
Explains update-size effects, boosting-tree contributions, and practical tree-count and depth trade-offs without deriving optimization mathematics.
Examples
Uses out-of-core logistic regression batches, a custom gradient-boosting classifier, and a 1,000-tree learning-rate search.
What you can do after
Apply it by tuning update size alongside tree count and depth, then evaluating prediction performance on unseen data.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains learning rate →

Building Specialized AI Systems

A learning rate is the optimization step-size parameter that controls how much model weights change during training.

How it is used
Controls how quickly model weights adapt during pre-training and especially domain-specific fine-tuning.
How deep it goes
Explains the gradient-descent step, numerical ranges, and too-large versus too-small failure modes without deriving gradients.
Examples
Configures 2e-4 for LoRA-based analyst-report summarization with batch size, accumulation, warmup, and a linear scheduler.
What you can do after
Operate/configure fine-tuning by selecting a rate that balances adaptation speed, stability, and preservation of learned representations.

Read how Building Specialized AI Systems explains learning rate →

Inside Deep Learning: Math, Algorithms, Models

The step-size parameter controlling how far an optimization update moves in the direction suggested by the gradient.

How it is used
Controls gradient-update size when minimizing a function or training neural-network parameters.
How deep it goes
Works through a manual PyTorch update, an SGD loop, and scheduling strategies without deriving an optimal initial rate.
Examples
Uses a scalar minimization loop, an exponential-decay calculation, a validation-plateau scheduler, and separate GAN optimizer settings.
What you can do after
Operate/configure it through PyTorch optimizer learning rates and schedulers, choosing smaller steps when updates overshoot or validation loss plateaus.

Read how Inside Deep Learning: Math, Algorithms, Models explains learning rate →

Applied Reinforcement Learning

A learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.

How it is used
Controls the strength of error-driven parameter updates in reinforcement learning and language-model fine-tuning.
How deep it goes
Explains error scaling and a linear TD update, then compares learning rates across fine-tuning stages and diagnostic runs.
Examples
Compares three pipeline rates and reports LoRA and GRPO runs with nearly unchanged KL and no accuracy improvement.
What you can do after
Diagnose insufficient learning by checking KL movement and comparing the learning rate with the update method.

Read how Applied Reinforcement Learning explains learning rate →

Ensemble Methods for Machine Learning

learning_rate is a boosting hyperparameter that scales the contribution of each successive weak learner and controls the rate at which ensemble complexity grows.

How it is used
Controls each tree or weak learner’s contribution while tuning boosted ensembles for generalization.
How deep it goes
Explains additive scaling, weight effects, tradeoffs, and cross-validation selection without deriving boosting objectives.
Examples
Uses breast-cancer classification with AdaBoost, GradientBoostingClassifier, LightGBM, and XGBoost validation sweeps.
What you can do after
Apply it via scikit-learn, LightGBM, or XGBoost and select candidate values using cross-validation.

Read how Ensemble Methods for Machine Learning explains learning rate →

Natural Language Processing in Action, Second Edition

A learning rate is a numerical step-size setting that controls how far an optimization algorithm moves parameters in response to a gradient.

How it is used
Controls parameter-update size during neural-network training and serves as a principal optimizer hyperparameter.
How deep it goes
Explains update intuition, convergence, overshooting, batch effects, and optimizer choices, but defers mathematical derivation.
Examples
Uses PyTorch SGD training with lr 0.02 and compares learning rates in documented RNN experiments.
What you can do after
Tune optimizer behavior by comparing convergence, loss, accuracy, and training speed across learning-rate settings.

Read how Natural Language Processing in Action, Second Edition explains learning rate →

Grokking Machine Learning, Second Edition

A learning rate is a small positive step size that controls the magnitude of parameter updates during iterative learning.

How it is used
Controls the size of parameter updates while iterative algorithms fit lines, classify points, or train neural networks.
How deep it goes
Explains the intuition, update equations, pseudocode, and repeated training behavior, but does not develop a systematic tuning strategy.
Examples
Uses house-price line fitting, square and absolute tricks, a Python update function, and a 0.01 learning rate in linear regression.
What you can do after
Apply it by inserting a small step size into linear-regression, perceptron, and neural-network parameter updates.

Read how Grokking Machine Learning, Second Edition explains learning rate →

Deep Learning with R, Third Edition

A learning rate is the step-size hyperparameter controlling how far model parameters move during an optimization update.

How it is used
Configures the magnitude of parameter updates during mini-batch gradient descent and helps diagnose stalled training.
How deep it goes
Defines the update role, gives the weight-update equation, and demonstrates tuning, but defers systematic schedules and advanced optimization details.
Examples
Uses manual weight updates, Keras optimizers, and MNIST runs comparing learning rates of 1 and 1e-2.
What you can do after
Operate/configure training by setting the learning rate and adjusting it when updates overshoot or optimization appears stalled.

Read how Deep Learning with R, Third Edition explains learning rate →

Deep Learning with PyTorch, Second Edition

The step-size hyperparameter that controls how much model parameters are changed per update during optimization.

How it is used
Controls the size of parameter updates when fitting a model and provides a tuning lever when training is unstable or slow.
How deep it goes
Works through gradient-scaled updates, divergence, slow progress, and input normalization; introduces adaptive optimizers without detailing their variants.
Examples
Compares training runs at 1e-2 and 1e-4, then rescales inputs and retries 1e-2.
What you can do after
Diagnose training instability by inspecting loss and gradient scales, adjusting the learning rate, and normalizing inputs.

Read how Deep Learning with PyTorch, Second Edition explains learning rate →

LLM Customization and Fine-Tuning

The learning rate is a training hyperparameter that controls the size of parameter updates during optimization.

How it is used
Controls update size to balance adaptation, training stability, and retention of pretrained capabilities during fine-tuning.
How deep it goes
Provides starting ranges, a one-variable sweep, loss-curve diagnostics, and stopping criteria for hyperparameter iteration.
Examples
Compares an illustrative six-run SFT log and a DPO run using 270 training preference pairs.
What you can do after
Operate/configure learning rates for fine-tuning and diagnose underfitting or instability using training and validation loss.

Read how LLM Customization and Fine-Tuning explains learning rate →

Machine Learning Platform Engineering

A learning rate is a training hyperparameter that controls the step size used to update model parameters during optimization.

How it is used
Balances training efficiency, resource utilization, accuracy, and training time in production model configuration.
How deep it goes
Explains learning-rate tuning as an iterative response to convergence problems but does not detail optimization internals.
Examples
Uses an Ultralytics YOLO ID card detector and revises learning rate, batch size, or epochs when training fails to converge.
What you can do after
Diagnose non-converging training by adjusting the learning rate alongside batch size and epochs.

Read how Machine Learning Platform Engineering explains learning rate →

Math and Architectures of Deep Learning

The step-size setting that controls how far model parameters move on each optimization update, possibly chosen separately for each parameter dimension.

How it is used
Controls parameter-update size while training neural networks with gradient descent.
How deep it goes
Explains gradient direction, Taylor approximation, Hessian context, and PyTorch configuration, while deferring fuller optimization details.
Examples
Uses a Statsville height-and-weight classifier, an SGD setting of 0.02, and epoch-by-epoch loss reduction.
What you can do after
Apply it via PyTorch by configuring SGD and interpreting its effect on training progress and overshooting risk.

Read how Math and Architectures of Deep Learning explains learning rate →

Evolutionary Deep Learning: Genetic algorithms and neural networks

A learning rate is the step-size parameter that determines how strongly model parameters are updated during training.

How it is used
Guides training-speed and model-quality experiments within broader hyperparameter tuning.
How deep it goes
Demonstrates configuration and comparative reruns, emphasizing observed training effects rather than update-equation derivations.
Examples
Uses notebook values from 3.5e-06 to 3.5e-01, GAN generator and discriminator settings, and agent-learning exercises.
What you can do after
Diagnose training behavior by changing learning-rate settings and observing effects on models, GANs, or agents.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains learning rate →

Fight Fraud with Machine Learning

A learning rate is a training setting that controls the step size used when updating model parameters.

How it is used
Configures the step size used while training fraud-classification neural networks.
How deep it goes
Explains gradient-scaled updates, risks of extreme values, and optimizer usage, but does not prescribe a tuning procedure.
Examples
Uses Adam configurations with learning rates of 0.01 for an MLP and 0.001 for an audio transformer.
What you can do after
Apply it via a PyTorch optimizer and observe its effect through training loss and validation metrics.

Read how Fight Fraud with Machine Learning explains learning rate →

Transformers in Action

Learning rate is an optimization hyperparameter that controls the magnitude of parameter updates during training.

How it is used
Controls parameter-update magnitude while configuring and evaluating language-model training.
How deep it goes
Explains gradient-scaled updates and high-versus-low trade-offs, then demonstrates configuration with scheduling but stops short of tuning procedures.
Examples
Uses convergence figures at 0.1 and 0.01 plus Gemma 3 270M English–Spanish fine-tuning configuration.
What you can do after
Operate/configure training with a selected learning rate, scheduler, warmup ratio, and resource-aware fine-tuning setup.

Read how Transformers in Action explains learning rate →

Data Without Labels

A learning rate is a scalar hyperparameter that controls the size of parameter updates during iterative training.

How it is used
Uses the learning rate to control corrective weight updates during neural-network training.
How deep it goes
Defines update size and training tradeoffs, then applies the parameter to a Boltzmann weight-update rule.
Examples
Illustrates corrective network steps and visible–hidden correlation differences in Boltzmann weight updates.
What you can do after
Evaluate how update size affects accuracy and training time in gradient-based neural-network training.

Read how Data Without Labels explains learning rate →

Grokking AI Algorithms, Second Edition

A learning rate is a training hyperparameter that controls the step size used to update model parameters.

How it is used
Controls parameter-update size during neural-network and language-model training.
How deep it goes
Explains update-size effects and gradient-descent placement, then stops short of detailed optimization theory.
Examples
Uses candidate values 0.01, 0.001, and 0.0001 with manual, grid, and random search.
What you can do after
Operate/configure training by selecting and adjusting a learning rate through observed results or systematic search.

Read how Grokking AI Algorithms, Second Edition explains learning rate →

LLMs in Production

A learning rate is a training hyperparameter that controls the magnitude of parameter updates during optimization.

How it is used
Uses the learning rate to control training progress and recover from misleading early convergence.
How deep it goes
Explains parameter-update magnitude and gives a checkpoint-based adjustment procedure, stopping short of optimizer internals.
Examples
Uses a T5 XXL run that repeatedly appeared converged before completing its dataset.
What you can do after
Diagnose suspicious early convergence and adjust the learning rate through checkpointed recovery.

Read how LLMs in Production explains learning rate →

Designing Deep Learning Systems: A guide for software engineers

A learning rate is a hyperparameter that controls the step size used to update model parameters during optimization.

How it is used
Controls parameter-update behavior within the model training loop and serves as a tunable input to the optimization algorithm.
How deep it goes
Explains its workflow position and effects on training speed and accuracy, but does not derive the update rule or internal optimizer mechanics.
Examples
Demonstrates tuning through TensorFlow Playground and a spiral-classification configuration with specified layers, activation, batch size, and regularization.
What you can do after
Apply it via TensorFlow Playground experiments that compare learning behavior, training time, and classification performance.

Read how Designing Deep Learning Systems: A guide for software engineers explains learning rate →

Timeless Algorithms, The Foundational Ideas

A learning rate controls the size of parameter updates during iterative model training.

How it is used
Controls the size of gradient-descent parameter updates during network training.
How deep it goes
Defines the update equation and works through one-observation updates, then relates them to mini-batches and epochs.
Examples
Uses a seven-parameter network update with learning rate 0.1 and gradients from one observation.
What you can do after
Understand how a selected learning rate scales gradient-based parameter changes during training.

Read how Timeless Algorithms, The Foundational Ideas explains learning rate →

Also covered in

What to know first

Related concepts

Questions about learning rate

What is learning rate?

A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.

Which Manning books explain learning rate?

Build AI Drug Discovery Pipelines, Grokking Machine Learning, Classic Computer Science Problems in Swift: Essential techniques for practicing programmers, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Building Specialized AI Systems, Inside Deep Learning: Math, Algorithms, Models, Applied Reinforcement Learning, Ensemble Methods for Machine Learning and 15 more.

What should I know before learning learning rate?

Manning books that teach learning rate usually introduce objective function, accuracy, gradient and language model first.

What is learning rate a kind of?

Learning rate is a kind of hyperparameter. A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.

What is learning rate also called?

Learning rate is also called LR or step size.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.