liveBook · Manning

Gradient descent

Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.

How 24 Manning books explain it

Advanced Algorithms and Data Structures

Gradient descent is an iterative optimization technique that moves a candidate solution in the direction of decreasing objective-function value using gradient information.

How it is used
Optimizes differentiable cost functions for graph drawings and model training by iteratively adjusting solution coordinates.
How deep it goes
Develops partial-derivative updates, stopping conditions, learning-rate trade-offs, and pseudocode, then implements graph-drawing restarts.
Examples
Uses a squared-error line fit and a force-directed graph drawing with random starting positions.
What you can do after
Build it from scratch with coordinate updates, a stopping limit, and optional restarts; recognize when the cost function makes it unsuitable.

Read how Advanced Algorithms and Data Structures explains gradient descent →

Grokking Machine Learning

An iterative optimization technique that changes parameters in the direction of greatest local decrease of an objective function.

How it is used
Uses iterative parameter updates to reduce error in linear regression and improve perceptron classifications.
How deep it goes
Explains the descent intuition, error functions, repeated updates, learning rate, and iteration monitoring, without deriving calculus.
Examples
Illustrates mountain descent, lines fitted to points, RMSE iteration plots, and word-weight updates for classified sentences.
What you can do after
Apply it via iterative updates to lower a model’s error and improve a classifier’s treatment of misclassified examples.

Read how Grokking Machine Learning explains gradient descent →

Ensemble Methods for Machine Learning

Gradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.

How it is used
Frames model training as loss minimization and supplies the optimization intuition behind gradient boosting.
How deep it goes
Works through negative-gradient updates, normalization, line search, tolerance, and vectorized squared-loss training; defers numerical gradient alternatives.
Examples
Uses Branin-function contour descent and a noisy two-class linear classification task with squared loss.
What you can do after
Build it from scratch for a differentiable objective and apply it to train a simple linear classifier.

Read how Ensemble Methods for Machine Learning explains gradient descent →

Deep Learning with R, Third Edition

Gradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.

How it is used
Explains how neural networks learn weight values that reduce a loss.
How deep it goes
Develops an intuitive update procedure, contrasts SGD variants, and demonstrates gradient computation and optimizer-based updates without dense derivations.
Examples
Uses MNIST digit classification and an excessively high learning rate to illustrate stalled training.
What you can do after
Apply it via Keras optimizers and TensorFlow GradientTape, and troubleshoot training with learning rate and batch size adjustments.

Read how Deep Learning with R, Third Edition explains gradient descent →

Deep Learning with PyTorch, Second Edition

How it is used
Uses gradient descent to estimate parameters of differentiable models from data.
How deep it goes
Builds intuition from knob adjustments and finite differences, then implements iterative updates with autograd and an optimizer.
Examples
Fits a linear temperature conversion and trains a multilayer network on temperature data.
What you can do after
Apply it via PyTorch autograd and optim.SGD in a model training loop.

Read how Deep Learning with PyTorch, Second Edition explains gradient descent →

Deep Learning with JAX

An optimization step that updates model parameters using gradients, typically in the opposite direction of the loss gradient.

How it is used
Uses gradient updates to train regression and image-classification models, including synchronized data-parallel training.
How deep it goes
Explains the loss-gradient relationship and update loop, then implements gradient aggregation and learning-rate decay, without deriving autodiff internals.
Examples
Uses synthetic time-series regression and replicated image batches with eight devices and 32 examples per device.
What you can do after
Builds a JAX training update that computes gradients, aggregates them across devices, and applies synchronized parameter changes.

Read how Deep Learning with JAX explains gradient descent →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Gradient descent is an optimization technique that iteratively adjusts parameters in the direction that reduces a differentiable objective function.

How it is used
Optimizes linear-model weights and motivates sequential error correction in gradient-boosted trees.
How deep it goes
Explains iterative weight updates through gradients and landscapes, then applies the idea to SGD batches and boosted-tree corrections.
Examples
Uses a mountain-and-valley error surface, out-of-core logistic regression, and Airbnb NYC binary classification with boosted trees.
What you can do after
Apply it via Scikit-learn SGDClassifier or implement boosted-tree updates using learning rate, estimators, and regression trees.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains gradient descent →

Sutskever's List

Gradient descent is an iterative optimization method that changes parameters in the direction opposite the gradient of a loss function.

How it is used
Uses gradient descent to explain neural-network weight updates and the training difficulties posed by narrow loss valleys.
How deep it goes
Explains local-slope updates, narrow-valley failures, and practical training settings without a formal derivation.
Examples
Illustrates noisy mini-batch updates with a foggy hiker, narrow-valley failures with contour diagrams, and training choices with AlexNet.
What you can do after
Evaluate/understand when efficient first-order updates may stall and how momentum and learning-rate adjustments shape training.

Read how Sutskever's List explains gradient descent →

Inside Deep Learning: Math, Algorithms, Models

A rule that updates parameters by subtracting a scaled gradient from their current values.

How it is used
Uses repeated gradient-based parameter updates to reduce loss when training models.
How deep it goes
Derives scalar update direction and parameter-update notation, then identifies gradient and learning rate without detailing their optimization strategies.
Examples
Uses f(x) = (x−2)², linear regression, and neural-network training as examples.
What you can do after
Apply gradient descent through PyTorch automatic differentiation and a training loop to update model parameters.

Read how Inside Deep Learning: Math, Algorithms, Models explains gradient descent →

Deep Learning with Python, Third Edition

A parameter update step computed from gradients to reduce model error.

How it is used
Explains how neural networks efficiently adjust weights to reduce training loss.
How deep it goes
Develops an intuitive gradient explanation and practical tuning guidance without heavy mathematical notation.
Examples
Uses a coefficient-and-loss comparison and MNIST training runs with contrasting learning rates.
What you can do after
Diagnose/debug stalled training by adjusting learning rate or batch size.

Read how Deep Learning with Python, Third Edition explains gradient descent →

Data Without Labels

Gradient descent is an iterative optimization technique that updates parameters in the direction of decreasing objective-function value.

How it is used
Optimizes prediction loss by adjusting model parameters and offers a way to reduce stress in nonmetric MDS.
How deep it goes
Explains iterative negative-gradient movement and three update schedules, but gives no worked parameter calculation or code implementation.
Examples
Uses a mountain-descent analogy, an MDS distance-preservation objective, and diagrams of convergence and local minima.
What you can do after
Evaluate/understand it by distinguishing batch, per-instance, and mini-batch updates and recognizing that convergence can occur at a local minimum.

Read how Data Without Labels explains gradient descent →

Grokking Bayes

Gradient descent is an iterative optimization method that updates parameters in the direction of decreasing objective value according to the gradient.

How it is used
Optimizes variational distributions and many machine-learning models by repeatedly updating parameters using objective gradients.
How deep it goes
Defines iterative gradient-based updates and convergence checks, but defers detailed technical treatment of the algorithm.
Examples
Uses Normal-to-Beta approximation updates, ELBO progress plots, and PyMC Bayesian linear regression with 200,000 fitting steps.
What you can do after
Operate/configure PyMC variational inference, choose optimization length, use minibatches, and diagnose convergence from objective-history plots.

Read how Grokking Bayes explains gradient descent →

Math and Architectures of Deep Learning

How it is used
Trains classifiers, neural networks, and likelihood-based models by iteratively reducing an error or loss function.
How deep it goes
Develops geometric intuition, update equations, learning-rate effects, local minima, and PyTorch loops, then defers practical optimizer variants.
Examples
Uses neural-network loss surfaces, PyTorch MSE, Gaussian MLE, MAP regularization, and local-minimum trajectories.
What you can do after
Apply it via PyTorch to update model or statistical parameters from differentiable losses and inspect convergence problems.

Read how Math and Architectures of Deep Learning explains gradient descent →

Grokking AI Algorithms, Second Edition

Gradient descent is an optimization technique that iteratively changes parameters in the direction that reduces an objective function.

How it is used
Trains neural networks by iteratively reducing prediction cost through weight updates.
How deep it goes
Explains slope intuition, learning-rate updates, backpropagation, and Chain Rule calculations without pursuing deeper optimization theory.
Examples
Uses a hilly-landscape analogy, weight-versus-cost figures, ANN updates, and diffusion-model noise prediction.
What you can do after
Apply it to update neural-network weights and tune diffusion models toward more accurate predictions.

Read how Grokking AI Algorithms, Second Edition explains gradient descent →

How Large Language Models Work

Gradient descent is an iterative optimization algorithm that adjusts parameters in the direction that reduces a differentiable objective function.

How it is used
Explains how training algorithms improve neural-network predictions by repeatedly reducing a measured loss.
How deep it goes
Provides a high-level procedural explanation with analogies, while deliberately sparing mathematical derivation.
Examples
Uses a rolling-ball hill analogy and figures showing inputs, labels, parameter updates, global minima, and local minima.
What you can do after
Evaluate how iterative parameter updates train models and recognize why loss objectives do not guarantee optimal or broadly capable behavior.

Read how How Large Language Models Work explains gradient descent →

Building Specialized AI Systems

Gradient descent is an optimization algorithm that iteratively adjusts model parameters in the direction that decreases a loss function.

How it is used
Drives training and fine-tuning by repeatedly reducing prediction loss through parameter updates.
How deep it goes
Explains parameters, loss, directional updates, backpropagation, and optimizers through a conceptual loop and code listing.
Examples
Uses sentiment classification, a hiking-in-fog analogy, a single-weight figure, and a basic training-loop listing.
What you can do after
Operate/configure a training loop that computes loss, backpropagates gradients, updates parameters, and monitors progress.

Read how Building Specialized AI Systems explains gradient descent →

Grokking Machine Learning, Second Edition

Gradient descent is an iterative optimization technique that repeatedly moves parameters in a direction that decreases an objective function.

How it is used
Uses repeated parameter adjustments to minimize model error in linear regression and perceptron training.
How deep it goes
Explains the descent intuition, error functions, gradients, and data-batch variants, while deferring full calculus derivations.
Examples
Uses a mountain descent, fitted lines, absolute or squared error, and separating lines for two point groups.
What you can do after
Apply gradient descent to reduce linear-model error using one point, a mini-batch, or the full dataset per iteration.

Read how Grokking Machine Learning, Second Edition explains gradient descent →

Fight Fraud with Machine Learning

Gradient descent is an iterative optimization method that updates parameters in the direction that reduces an objective function.

How it is used
Trains deep neural networks by iteratively reducing a loss function through gradient-based weight updates.
How deep it goes
Explains initialization, forward passes, loss, chain-rule backpropagation, and updates, while deferring detailed optimizer mechanics.
Examples
Uses a dog-versus-cat image classifier, binary cross entropy, network weights, and a gradient-descent training diagram.
What you can do after
Apply it to understand and train a neural network by calculating loss, propagating gradients, and updating weights.

Read how Fight Fraud with Machine Learning explains gradient descent →

Timeless Algorithms, The Foundational Ideas

Gradient descent is an iterative optimization technique that updates model parameters in the direction that reduces an objective function.

How it is used
Updates model parameters incrementally during training to reduce a loss function.
How deep it goes
Works through a seven-parameter update and mini-batch training, then discusses optimization limits without guaranteeing a global minimum.
Examples
Uses a pedestrian-detection network to show seven parameter updates and a small improvement in predicted probability and loss.
What you can do after
Apply the update rule to parameters using gradients and a selected learning rate, then assess changes in prediction and loss.

Read how Timeless Algorithms, The Foundational Ideas explains gradient descent →

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

How it is used
Updates neural-network weights during training to move toward lower error.
How deep it goes
Describes the weight-update rule and learning-rate role, but stops short of explaining why backpropagation works.
Examples
Uses updates across hidden and output layers, followed by repeated training and testing on known answers.
What you can do after
Apply the update rule to adjust network weights and continue training with new input-output examples.

Read how Classic Computer Science Problems in Swift: Essential techniques for practicing programmers explains gradient descent →

Reinforcement Learning from Human Feedback

A gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.

How it is used
Uses gradient updates to optimize language-model policies during reinforcement-learning post-training.
How deep it goes
Explains data reuse and clipping in PPO, including the simplification available with one gradient step per sample.
Examples
Illustrates single-step PPO or GRPO and code-generation rewards based on passing unit tests.
What you can do after
Evaluate gradient-step choices in policy optimization, including data reuse and the single-step clipping simplification.

Read how Reinforcement Learning from Human Feedback explains gradient descent →

Bayesian Optimization in Action

Gradient descent is an iterative optimization technique that updates parameters in the direction opposite the objective gradient.

How it is used
Uses iterative parameter updates to train Gaussian processes by minimizing likelihood-based loss.
How deep it goes
Explains intuition, training-loop steps, parameter targeting, convergence, and numerical instability without deriving gradient mathematics.
Examples
Uses GP mean optimization, 500-iteration training loops, loss curves, and mountain-descent imagery.
What you can do after
Apply it via PyTorch and GPyTorch to optimize GP parameters and inspect convergence.

Read how Bayesian Optimization in Action explains gradient descent →

Build AI Drug Discovery Pipelines

Gradient descent is an iterative optimization method that updates parameters in the direction that decreases a differentiable objective function.

How it is used
Optimizes linear and logistic models for molecular solubility and cytochrome P450 inhibition prediction.
How deep it goes
Explains gradients, updates, learning rates, variants, stopping criteria, clipping, and scaling without deriving optimizer implementations.
Examples
Uses AqSolDB solubility modeling and cytochrome P450 inhibition classification as applied examples.
What you can do after
Apply it via batch, mini-batch, or stochastic updates while selecting step sizes and termination criteria for stable convergence.

Read how Build AI Drug Discovery Pipelines explains gradient descent →

Optimization Algorithms: AI techniques for design, planning, and control problems

An iterative optimization algorithm that changes variables in the direction of the negative gradient to reduce a differentiable objective function.

How it is used
Trains machine-learning models and supports prediction by optimizing continuous differentiable objectives.
How deep it goes
Defines iterative negative-gradient movement and notes convexity and learning-rate conditions, but does not provide implementation steps.
Examples
Uses hiking down to the bottom of a valley as an intuition for following the steepest negative slope.
What you can do after
Evaluate optimization methods for differentiable objectives and distinguish them from hill climbing.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains gradient descent →

Also covered in

What to know first

Related concepts

Questions about gradient descent

What is gradient descent?

Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.

Which Manning books explain gradient descent?

Advanced Algorithms and Data Structures, Grokking Machine Learning, Ensemble Methods for Machine Learning, Deep Learning with R, Third Edition, Deep Learning with PyTorch, Second Edition, Deep Learning with JAX, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Sutskever's List and 16 more.

What should I know before learning gradient descent?

Manning books that teach gradient descent usually introduce objective function, parameter, language model and gradient first.

What is the difference between gradient descent and newton-raphson method?

Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value. The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.

What kinds of gradient descent are there?

Kinds of gradient descent include stochastic gradient descent.

What is gradient descent a kind of?

Gradient descent is a kind of gradient-based optimization. A method that uses derivatives of a function with respect to variables to guide optimization or analyze how outputs respond to changes in those variables.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.