Gradient descent
Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
Advanced Algorithms and Data Structures
Gradient descent is an iterative optimization technique that moves a candidate solution in the direction of decreasing objective-function value using gradient information.
- How it is used
- Optimizes differentiable cost functions for graph drawings and model training by iteratively adjusting solution coordinates.
- How deep it goes
- Develops partial-derivative updates, stopping conditions, learning-rate trade-offs, and pseudocode, then implements graph-drawing restarts.
- Examples
- Uses a squared-error line fit and a force-directed graph drawing with random starting positions.
- What you can do after
- Build it from scratch with coordinate updates, a stopping limit, and optional restarts; recognize when the cost function makes it unsuitable.
Read how Advanced Algorithms and Data Structures explains gradient descent →
Grokking Machine Learning
An iterative optimization technique that changes parameters in the direction of greatest local decrease of an objective function.
- How it is used
- Uses iterative parameter updates to reduce error in linear regression and improve perceptron classifications.
- How deep it goes
- Explains the descent intuition, error functions, repeated updates, learning rate, and iteration monitoring, without deriving calculus.
- Examples
- Illustrates mountain descent, lines fitted to points, RMSE iteration plots, and word-weight updates for classified sentences.
- What you can do after
- Apply it via iterative updates to lower a model’s error and improve a classifier’s treatment of misclassified examples.
Read how Grokking Machine Learning explains gradient descent →
Ensemble Methods for Machine Learning
Gradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.
- How it is used
- Frames model training as loss minimization and supplies the optimization intuition behind gradient boosting.
- How deep it goes
- Works through negative-gradient updates, normalization, line search, tolerance, and vectorized squared-loss training; defers numerical gradient alternatives.
- Examples
- Uses Branin-function contour descent and a noisy two-class linear classification task with squared loss.
- What you can do after
- Build it from scratch for a differentiable objective and apply it to train a simple linear classifier.
Read how Ensemble Methods for Machine Learning explains gradient descent →
Deep Learning with R, Third Edition
Gradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.
- How it is used
- Trains neural networks by updating weights to reduce prediction loss.
- How deep it goes
- Explains the forward–backward–update loop and a training-step implementation, while leaving backpropagation derivatives unimplemented.
- Examples
- Uses MNIST classification, a TensorFlow GradientTape training step, and a comparison of stalled versus successful learning rates.
- What you can do after
- Apply it via Keras optimizers and diagnose stalled training by adjusting learning rate or batch size.
Read how Deep Learning with R, Third Edition explains gradient descent →
Deep Learning with PyTorch, Second Edition
- How it is used
- Trains models on paired inputs and desired outputs by adjusting parameters to reduce prediction loss.
- How deep it goes
- Works from numerical slope estimates through repeated updates, then shows how PyTorch autograd and SGD perform the training steps.
- Examples
- Uses weight-and-bias adjustment knobs, a Fahrenheit-to-Celsius fit, and PyTorch linear-model training loops.
- What you can do after
- Build it from scratch with slope-based updates, then apply it via PyTorch autograd and `optim.SGD`.
Read how Deep Learning with PyTorch, Second Edition explains gradient descent →
Deep Learning with JAX
An optimization step that updates model parameters using gradients, typically in the opposite direction of the loss gradient.
- How it is used
- Uses gradient updates to train regression and image-classification models, including synchronized data-parallel training.
- How deep it goes
- Explains the loss-gradient relationship and update loop, then implements gradient aggregation and learning-rate decay, without deriving autodiff internals.
- Examples
- Uses synthetic time-series regression and replicated image batches with eight devices and 32 examples per device.
- What you can do after
- Builds a JAX training update that computes gradients, aggregates them across devices, and applies synchronized parameter changes.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Gradient descent is an optimization technique that iteratively adjusts parameters in the direction that reduces a differentiable objective function.
- How it is used
- Optimizes linear-model weights and motivates sequential error correction in gradient-boosted trees.
- How deep it goes
- Explains iterative weight updates through gradients and landscapes, then applies the idea to SGD batches and boosted-tree corrections.
- Examples
- Uses a mountain-and-valley error surface, out-of-core logistic regression, and Airbnb NYC binary classification with boosted trees.
- What you can do after
- Apply it via Scikit-learn SGDClassifier or implement boosted-tree updates using learning rate, estimators, and regression trees.
Sutskever's List
Gradient descent is an iterative optimization method that changes parameters in the direction opposite the gradient of a loss function.
- How it is used
- Uses gradient descent to explain neural-network weight updates and the training difficulties posed by narrow loss valleys.
- How deep it goes
- Explains local-slope updates, narrow-valley failures, and practical training settings without a formal derivation.
- Examples
- Illustrates noisy mini-batch updates with a foggy hiker, narrow-valley failures with contour diagrams, and training choices with AlexNet.
- What you can do after
- Evaluate/understand when efficient first-order updates may stall and how momentum and learning-rate adjustments shape training.
Deep Learning with Python, Third Edition
A parameter update step computed from gradients to reduce model error.
- How it is used
- Updates neural-network weights toward lower batch loss instead of testing each coefficient separately.
- How deep it goes
- Explains updates through differentiable operations and loss gradients, while avoiding mathematical notation rather than deriving the gradient.
- Examples
- Contrasts trial losses for one coefficient and MNIST training at learning rates of 1.0 and 1e-2.
- What you can do after
- Diagnose/debug with it by adjusting learning rate or batch size when training stalls.
Read how Deep Learning with Python, Third Edition explains gradient descent →
Data Without Labels
Gradient descent is an iterative optimization technique that updates parameters in the direction of decreasing objective-function value.
- How it is used
- Optimizes prediction loss by adjusting model parameters and offers a way to reduce stress in nonmetric MDS.
- How deep it goes
- Explains iterative negative-gradient movement and three update schedules, but gives no worked parameter calculation or code implementation.
- Examples
- Uses a mountain-descent analogy, an MDS distance-preservation objective, and diagrams of convergence and local minima.
- What you can do after
- Evaluate/understand it by distinguishing batch, per-instance, and mini-batch updates and recognizing that convergence can occur at a local minimum.
Math and Architectures of Deep Learning
- How it is used
- Trains classifiers, neural networks, and likelihood-based models by iteratively reducing an error or loss function.
- How deep it goes
- Develops geometric intuition, update equations, learning-rate effects, local minima, and PyTorch loops, then defers practical optimizer variants.
- Examples
- Uses neural-network loss surfaces, PyTorch MSE, Gaussian MLE, MAP regularization, and local-minimum trajectories.
- What you can do after
- Apply it via PyTorch to update model or statistical parameters from differentiable losses and inspect convergence problems.
Read how Math and Architectures of Deep Learning explains gradient descent →
Grokking AI Algorithms, Second Edition
Gradient descent is an optimization technique that iteratively changes parameters in the direction that reduces an objective function.
- How it is used
- Trains neural networks by iteratively reducing prediction cost through weight updates.
- How deep it goes
- Explains slope intuition, learning-rate updates, backpropagation, and Chain Rule calculations without pursuing deeper optimization theory.
- Examples
- Uses a hilly-landscape analogy, weight-versus-cost figures, ANN updates, and diffusion-model noise prediction.
- What you can do after
- Apply it to update neural-network weights and tune diffusion models toward more accurate predictions.
Read how Grokking AI Algorithms, Second Edition explains gradient descent →
How Large Language Models Work
Gradient descent is an iterative optimization algorithm that adjusts parameters in the direction that reduces a differentiable objective function.
- How it is used
- Explains how training algorithms improve neural-network predictions by repeatedly reducing a measured loss.
- How deep it goes
- Provides a high-level procedural explanation with analogies, while deliberately sparing mathematical derivation.
- Examples
- Uses a rolling-ball hill analogy and figures showing inputs, labels, parameter updates, global minima, and local minima.
- What you can do after
- Evaluate how iterative parameter updates train models and recognize why loss objectives do not guarantee optimal or broadly capable behavior.
Read how How Large Language Models Work explains gradient descent →
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
- How it is used
- Updates neural-network weights during training to move toward lower error.
- How deep it goes
- Describes the weight-update rule and learning-rate role, but stops short of explaining why backpropagation works.
- Examples
- Uses updates across hidden and output layers, followed by repeated training and testing on known answers.
- What you can do after
- Apply the update rule to adjust network weights and continue training with new input-output examples.
Reinforcement Learning from Human Feedback
A gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
- How it is used
- Optimizes post-training language models through RL updates that improve targeted behaviors while seeking to retain general capabilities.
- How deep it goes
- Explains PPO update mechanics through importance ratios, clipping, token log-probabilities, and a one-step simplification.
- Examples
- Uses math-only RL training, per-token PPO losses, and unit-test rewards for code-generation answers.
- What you can do after
- Apply it via PPO or GRPO by choosing one or multiple update steps and using clipping when policy drift must be constrained.
Read how Reinforcement Learning from Human Feedback explains gradient descent →
Bayesian Optimization in Action
Gradient descent is an iterative optimization technique that updates parameters in the direction opposite the objective gradient.
- How it is used
- Uses iterative parameter updates to train Gaussian processes by minimizing likelihood-based loss.
- How deep it goes
- Explains intuition, training-loop steps, parameter targeting, convergence, and numerical instability without deriving gradient mathematics.
- Examples
- Uses GP mean optimization, 500-iteration training loops, loss curves, and mountain-descent imagery.
- What you can do after
- Apply it via PyTorch and GPyTorch to optimize GP parameters and inspect convergence.
Read how Bayesian Optimization in Action explains gradient descent →
Optimization Algorithms: AI techniques for design, planning, and control problems
An iterative optimization algorithm that changes variables in the direction of the negative gradient to reduce a differentiable objective function.
- How it is used
- Trains machine-learning models and supports prediction by optimizing continuous differentiable objectives.
- How deep it goes
- Defines iterative negative-gradient movement and notes convexity and learning-rate conditions, but does not provide implementation steps.
- Examples
- Uses hiking down to the bottom of a valley as an intuition for following the steepest negative slope.
- What you can do after
- Evaluate optimization methods for differentiable objectives and distinguish them from hill climbing.
Related concepts
- gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- stochastic gradient descentincludes · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
- parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
- neural networkused by · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
- regularizationuses · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
- newton-raphson methodcontrasts with · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.