liveBook · Manning

Stochastic gradient descent

Also called SGD

Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.

How 17 Manning books explain it

Grokking Machine Learning

A gradient-descent variant that updates model parameters using one training data point at a time.

How it is used
Trains linear regression models and perceptron classifiers through repeated point-wise parameter adjustments.
How deep it goes
Defines point-wise updates and contrasts batch sizes, while deferring full mathematical details to gradient-descent appendices.
Examples
Uses a regression line, individual data points, and a perceptron line responding to correctly or incorrectly classified points.
What you can do after
Apply it conceptually by performing repeated single-point updates that reduce regression error or improve classification.

Read how Grokking Machine Learning explains stochastic gradient descent →

Bayesian Optimization in Action

Mini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.

How it is used
Scales loss optimization to datasets too large for full-batch Gaussian-process training.
How deep it goes
Defines batch approximation, explains its noisy descent behavior, and demonstrates the PyTorch training loop without deriving optimizer internals.
Examples
Uses loss-valley illustrations, PyTorch loops, variational Gaussian processes, and a housing-price timing exercise.
What you can do after
Apply it via PyTorch or GPyTorch to train models with repeated batch-level loss and gradient updates.

Read how Bayesian Optimization in Action explains stochastic gradient descent →

Deep Learning with Python, Third Edition

A gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.

How it is used
Trains neural networks by iteratively reducing loss on randomly drawn batches.
How deep it goes
Explains the forward pass, loss, backward pass, parameter update, learning-rate choice, and batch-size tradeoffs.
Examples
Uses a PyTorch linear classifier and a Keras handwritten-digit classifier to illustrate training updates.
What you can do after
Apply it via PyTorch’s SGD optimizer, including gradient calculation, parameter updates, and gradient resets.

Read how Deep Learning with Python, Third Edition explains stochastic gradient descent →

Machine Learning Algorithms in Depth

Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.

How it is used
Optimizes classifier and neural-network parameters through repeated gradient-based updates on sampled training data.
How deep it goes
Derives mini-batch updates and learning-rate effects, then relates perceptron updates to hinge-loss gradients and compares neural-network optimizers.
Examples
Uses an iris perceptron classifier and a LeNet-on-CIFAR-100 comparison of cross-entropy loss and validation accuracy.
What you can do after
Build it from scratch for perceptron training and apply it via Keras/TensorFlow to compare neural-network optimizers.

Read how Machine Learning Algorithms in Depth explains stochastic gradient descent →

Math and Architectures of Deep Learning

How it is used
Uses randomly sampled minibatches to estimate gradients and iteratively reduce neural-network loss.
How deep it goes
Explains dataset-scale motivation, geometric intuition, noisy estimates, and a complete PyTorch update loop.
Examples
Demonstrates height-weight classification with shuffled batches, cross-entropy loss, PyTorch SGD, and decreasing epoch loss.
What you can do after
Apply it via PyTorch by preparing a DataLoader, computing minibatch loss, backpropagating, and stepping an SGD optimizer.

Read how Math and Architectures of Deep Learning explains stochastic gradient descent →

Natural Language Processing in Action, Second Edition

Stochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.

How it is used
Deploys sampled parameter updates to train a PyTorch neural model while balancing convergence speed and update stability.
How deep it goes
Explains sampling, shuffling, gradients, learning rate, momentum, batches, and mini-batches, then demonstrates a runnable training loop.
Examples
Trains a name-gender classifier with vectorized names, binary cross-entropy, weighted examples, test accuracy, and PyTorch epoch logs.
What you can do after
Apply it via PyTorch by configuring SGD, running gradient updates, and monitoring training and test metrics.

Read how Natural Language Processing in Action, Second Edition explains stochastic gradient descent →

Deep Learning with R, Third Edition

Mini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.

How it is used
Optimizes neural-network weights during training by using loss gradients from randomly sampled data batches.
How deep it goes
Explains sampling, forward and backward passes, parameter updates, batch-size tradeoffs, and learning-rate risks without formal derivation.
Examples
Demonstrates MNIST digit classification with Keras Dense layers, Adam, five epochs, and batches of 128 samples.
What you can do after
Apply it via Keras by choosing an optimizer, loss, batch size, and training schedule for a neural-network model.

Read how Deep Learning with R, Third Edition explains stochastic gradient descent →

Inside Deep Learning: Math, Algorithms, Models

How it is used
Uses SGD to adjust parameters and reduce loss in a PyTorch optimization loop.
How deep it goes
Demonstrates optimizer setup and update steps, but defers the stochastic aspect and does not derive optimizer internals.
Examples
Uses a scalar squared-error minimization and a PyTorch neural-network training loop.
What you can do after
Apply SGD through PyTorch by configuring parameters and a learning rate, then calling the optimizer in a training loop.

Read how Inside Deep Learning: Math, Algorithms, Models explains stochastic gradient descent →

Building Specialized AI Systems

SGD is stochastic gradient descent, an optimization algorithm that updates parameters using gradients estimated from individual examples or minibatches.

How it is used
Optimizes fine-tuned language models when full-dataset gradient computation is impractical.
How deep it goes
Explains minibatch estimation, learning-rate trade-offs, noise, local minima, and the parameter-update loop.
Examples
Uses minibatches of 8, 16, or 32 examples and prompt-response pairs such as article summarization.
What you can do after
Apply it via a training loop that computes loss, backpropagates gradients, and updates parameters.

Read how Building Specialized AI Systems explains stochastic gradient descent →

Build AI Drug Discovery Pipelines

Mini-batch gradient descent is an optimization method that computes each parameter update from a small subset of the training data.

How it is used
Applies iterative parameter optimization to descriptor-based molecular solubility prediction.
How deep it goes
Explains sampling, convergence, stopping, clipping, and scaling, then demonstrates an SGDRegressor pipeline without developing optimization theory deeply.
Examples
Uses AqSolDB solubility prediction, molecular descriptors, SGDRegressor, loss curves, early stopping, and ESOL and y-scrambled baselines.
What you can do after
Apply it via Scikit-Learn’s SGDRegressor and assess convergence and generalization against solubility baselines.

Read how Build AI Drug Discovery Pipelines explains stochastic gradient descent →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Mini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.

How it is used
Handles large streamed datasets by optimizing model weights incrementally from batches or individual examples.
How deep it goes
Explains gradients, iterative updates, batch processing, and error landscapes, but does not derive the full optimization mathematics.
Examples
Uses streamed training batches, reserved test data, logistic regression code, optimization-landscape figures, and a reported accuracy of 0.818.
What you can do after
Apply SGD via scikit-learn’s SGDClassifier to train logistic regression incrementally on batches.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains stochastic gradient descent →

How Large Language Models Work

Stochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.

How it is used
Uses random training subsets to make neural-network parameter updates feasible when processing all training data is too expensive.
How deep it goes
Explains the subset-based update and its memory rationale, but skips detailed mathematics and implementation procedures.
Examples
Uses labeled inputs, parameter adjustments, a downhill ball, and loss-function graphs to illustrate training.
What you can do after
Evaluate/understand why training uses small data subsets and how those subsets approximate full-data gradient steps.

Read how How Large Language Models Work explains stochastic gradient descent →

Fight Fraud with Machine Learning

Stochastic gradient descent is an optimization method that updates model parameters using gradients estimated from individual examples or small batches.

How it is used
Trains deep neural networks by iteratively updating weights toward lower loss.
How deep it goes
Explains gradient-based updates, learning-rate trade-offs, and convergence, but stops short of optimizer implementation details.
Examples
Uses a deep-learning training workflow and contrasts overly high and low learning rates.
What you can do after
Build it from scratch by implementing iterative weight updates and checking whether the loss converges.

Read how Fight Fraud with Machine Learning explains stochastic gradient descent →

Deep Learning with PyTorch, Second Edition

An optimization method that updates parameters using stochastic gradient descent.

How it is used
Trains neural-network parameters by minimizing a loss during forward, backward, and optimizer steps.
How deep it goes
Explains gradient-based updates and PyTorch setup, but defers detailed trade-offs among alternative optimizers.
Examples
Demonstrates torch.optim.SGD in a tumor-classification application with CUDA placement, learning rate 0.001, and momentum 0.99.
What you can do after
Apply it via PyTorch by configuring SGD, connecting model parameters, and integrating it into a training loop.

Read how Deep Learning with PyTorch, Second Edition explains stochastic gradient descent →

Sutskever's List

How it is used
Trains large neural networks by updating weights from mini-batch gradients.
How deep it goes
Explains noisy gradient estimates, momentum, and learning-rate scheduling, while distinguishing first-order updates from curvature information.
Examples
Uses a foggy-hiker analogy and AlexNet's 128-image batches, 0.01 initial rate, and validation-triggered tenfold reductions.
What you can do after
Apply mini-batch updates with momentum and reduce the learning rate when validation accuracy flattens.

Read how Sutskever's List explains stochastic gradient descent →

Transformers in Action

How it is used
Optimizes language-model training by updating learned weights and biases to reduce loss.
How deep it goes
Explains gradients, update direction, learning-rate scaling, and batch-size effects, but does not detail stochastic sampling.
Examples
Compares convergence at learning rates of 0.1 and 0.01.
What you can do after
Configure training with a learning rate, batch size, and epoch count, and apply validation-based early stopping.

Read how Transformers in Action explains stochastic gradient descent →

Timeless Algorithms, The Foundational Ideas

Stochastic gradient descent is an optimization method that updates model parameters using gradients computed from individual observations or small batches.

How it is used
Trains neural networks through repeated gradient-based parameter updates on mini-batches.
How deep it goes
Explains the gradient update, learning-rate tradeoffs, mini-batch cycle, and epochs.
Examples
Traces network training through forward passes, loss calculation, backward gradient propagation, and mini-batch updates.
What you can do after
Apply mini-batch gradients to update network parameters across training epochs.

Read how Timeless Algorithms, The Foundational Ideas explains stochastic gradient descent →

Also covered in

What to know first

Related concepts

Questions about stochastic gradient descent

What is stochastic gradient descent?

Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.

Which Manning books explain stochastic gradient descent?

Grokking Machine Learning, Bayesian Optimization in Action, Deep Learning with Python, Third Edition, Machine Learning Algorithms in Depth, Math and Architectures of Deep Learning, Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Inside Deep Learning: Math, Algorithms, Models and 9 more.

What should I know before learning stochastic gradient descent?

Manning books that teach stochastic gradient descent usually introduce language model, objective function, parameter and gradient descent first.

What kinds of stochastic gradient descent are there?

Kinds of stochastic gradient descent include adam.

What is stochastic gradient descent a kind of?

Stochastic gradient descent is a kind of gradient descent. Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.

What is stochastic gradient descent also called?

Stochastic gradient descent is also called SGD.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.