Stochastic gradient descent
Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.
Grokking Machine Learning
A gradient-descent variant that updates model parameters using one training data point at a time.
- How it is used
- Trains linear regression models and perceptron classifiers through repeated point-wise parameter adjustments.
- How deep it goes
- Defines point-wise updates and contrasts batch sizes, while deferring full mathematical details to gradient-descent appendices.
- Examples
- Uses a regression line, individual data points, and a perceptron line responding to correctly or incorrectly classified points.
- What you can do after
- Apply it conceptually by performing repeated single-point updates that reduce regression error or improve classification.
Read how Grokking Machine Learning explains stochastic gradient descent →
Bayesian Optimization in Action
Mini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
- How it is used
- Scales loss optimization to datasets too large for full-batch Gaussian-process training.
- How deep it goes
- Defines batch approximation, explains its noisy descent behavior, and demonstrates the PyTorch training loop without deriving optimizer internals.
- Examples
- Uses loss-valley illustrations, PyTorch loops, variational Gaussian processes, and a housing-price timing exercise.
- What you can do after
- Apply it via PyTorch or GPyTorch to train models with repeated batch-level loss and gradient updates.
Read how Bayesian Optimization in Action explains stochastic gradient descent →
Deep Learning with Python, Third Edition
A gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
- How it is used
- Trains neural networks by iteratively reducing loss on randomly drawn batches.
- How deep it goes
- Explains the forward pass, loss, backward pass, parameter update, learning-rate choice, and batch-size tradeoffs.
- Examples
- Uses a PyTorch linear classifier and a Keras handwritten-digit classifier to illustrate training updates.
- What you can do after
- Apply it via PyTorch’s SGD optimizer, including gradient calculation, parameter updates, and gradient resets.
Read how Deep Learning with Python, Third Edition explains stochastic gradient descent →
Machine Learning Algorithms in Depth
Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
- How it is used
- Optimizes classifier and neural-network parameters through repeated gradient-based updates on sampled training data.
- How deep it goes
- Derives mini-batch updates and learning-rate effects, then relates perceptron updates to hinge-loss gradients and compares neural-network optimizers.
- Examples
- Uses an iris perceptron classifier and a LeNet-on-CIFAR-100 comparison of cross-entropy loss and validation accuracy.
- What you can do after
- Build it from scratch for perceptron training and apply it via Keras/TensorFlow to compare neural-network optimizers.
Read how Machine Learning Algorithms in Depth explains stochastic gradient descent →
Math and Architectures of Deep Learning
- How it is used
- Uses randomly sampled minibatches to estimate gradients and iteratively reduce neural-network loss.
- How deep it goes
- Explains dataset-scale motivation, geometric intuition, noisy estimates, and a complete PyTorch update loop.
- Examples
- Demonstrates height-weight classification with shuffled batches, cross-entropy loss, PyTorch SGD, and decreasing epoch loss.
- What you can do after
- Apply it via PyTorch by preparing a DataLoader, computing minibatch loss, backpropagating, and stepping an SGD optimizer.
Read how Math and Architectures of Deep Learning explains stochastic gradient descent →
Natural Language Processing in Action, Second Edition
Stochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
- How it is used
- Deploys sampled parameter updates to train a PyTorch neural model while balancing convergence speed and update stability.
- How deep it goes
- Explains sampling, shuffling, gradients, learning rate, momentum, batches, and mini-batches, then demonstrates a runnable training loop.
- Examples
- Trains a name-gender classifier with vectorized names, binary cross-entropy, weighted examples, test accuracy, and PyTorch epoch logs.
- What you can do after
- Apply it via PyTorch by configuring SGD, running gradient updates, and monitoring training and test metrics.
Deep Learning with R, Third Edition
Mini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
- How it is used
- Optimizes neural-network weights during training by using loss gradients from randomly sampled data batches.
- How deep it goes
- Explains sampling, forward and backward passes, parameter updates, batch-size tradeoffs, and learning-rate risks without formal derivation.
- Examples
- Demonstrates MNIST digit classification with Keras Dense layers, Adam, five epochs, and batches of 128 samples.
- What you can do after
- Apply it via Keras by choosing an optimizer, loss, batch size, and training schedule for a neural-network model.
Read how Deep Learning with R, Third Edition explains stochastic gradient descent →
Inside Deep Learning: Math, Algorithms, Models
- How it is used
- Uses SGD to adjust parameters and reduce loss in a PyTorch optimization loop.
- How deep it goes
- Demonstrates optimizer setup and update steps, but defers the stochastic aspect and does not derive optimizer internals.
- Examples
- Uses a scalar squared-error minimization and a PyTorch neural-network training loop.
- What you can do after
- Apply SGD through PyTorch by configuring parameters and a learning rate, then calling the optimizer in a training loop.
Read how Inside Deep Learning: Math, Algorithms, Models explains stochastic gradient descent →
Building Specialized AI Systems
SGD is stochastic gradient descent, an optimization algorithm that updates parameters using gradients estimated from individual examples or minibatches.
- How it is used
- Optimizes fine-tuned language models when full-dataset gradient computation is impractical.
- How deep it goes
- Explains minibatch estimation, learning-rate trade-offs, noise, local minima, and the parameter-update loop.
- Examples
- Uses minibatches of 8, 16, or 32 examples and prompt-response pairs such as article summarization.
- What you can do after
- Apply it via a training loop that computes loss, backpropagates gradients, and updates parameters.
Read how Building Specialized AI Systems explains stochastic gradient descent →
Build AI Drug Discovery Pipelines
Mini-batch gradient descent is an optimization method that computes each parameter update from a small subset of the training data.
- How it is used
- Applies iterative parameter optimization to descriptor-based molecular solubility prediction.
- How deep it goes
- Explains sampling, convergence, stopping, clipping, and scaling, then demonstrates an SGDRegressor pipeline without developing optimization theory deeply.
- Examples
- Uses AqSolDB solubility prediction, molecular descriptors, SGDRegressor, loss curves, early stopping, and ESOL and y-scrambled baselines.
- What you can do after
- Apply it via Scikit-Learn’s SGDRegressor and assess convergence and generalization against solubility baselines.
Read how Build AI Drug Discovery Pipelines explains stochastic gradient descent →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Mini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
- How it is used
- Handles large streamed datasets by optimizing model weights incrementally from batches or individual examples.
- How deep it goes
- Explains gradients, iterative updates, batch processing, and error landscapes, but does not derive the full optimization mathematics.
- Examples
- Uses streamed training batches, reserved test data, logistic regression code, optimization-landscape figures, and a reported accuracy of 0.818.
- What you can do after
- Apply SGD via scikit-learn’s SGDClassifier to train logistic regression incrementally on batches.
How Large Language Models Work
Stochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.
- How it is used
- Uses random training subsets to make neural-network parameter updates feasible when processing all training data is too expensive.
- How deep it goes
- Explains the subset-based update and its memory rationale, but skips detailed mathematics and implementation procedures.
- Examples
- Uses labeled inputs, parameter adjustments, a downhill ball, and loss-function graphs to illustrate training.
- What you can do after
- Evaluate/understand why training uses small data subsets and how those subsets approximate full-data gradient steps.
Read how How Large Language Models Work explains stochastic gradient descent →
Fight Fraud with Machine Learning
Stochastic gradient descent is an optimization method that updates model parameters using gradients estimated from individual examples or small batches.
- How it is used
- Trains deep neural networks by iteratively updating weights toward lower loss.
- How deep it goes
- Explains gradient-based updates, learning-rate trade-offs, and convergence, but stops short of optimizer implementation details.
- Examples
- Uses a deep-learning training workflow and contrasts overly high and low learning rates.
- What you can do after
- Build it from scratch by implementing iterative weight updates and checking whether the loss converges.
Read how Fight Fraud with Machine Learning explains stochastic gradient descent →
Deep Learning with PyTorch, Second Edition
An optimization method that updates parameters using stochastic gradient descent.
- How it is used
- Trains neural-network parameters by minimizing a loss during forward, backward, and optimizer steps.
- How deep it goes
- Explains gradient-based updates and PyTorch setup, but defers detailed trade-offs among alternative optimizers.
- Examples
- Demonstrates torch.optim.SGD in a tumor-classification application with CUDA placement, learning rate 0.001, and momentum 0.99.
- What you can do after
- Apply it via PyTorch by configuring SGD, connecting model parameters, and integrating it into a training loop.
Read how Deep Learning with PyTorch, Second Edition explains stochastic gradient descent →
Sutskever's List
- How it is used
- Trains large neural networks by updating weights from mini-batch gradients.
- How deep it goes
- Explains noisy gradient estimates, momentum, and learning-rate scheduling, while distinguishing first-order updates from curvature information.
- Examples
- Uses a foggy-hiker analogy and AlexNet's 128-image batches, 0.01 initial rate, and validation-triggered tenfold reductions.
- What you can do after
- Apply mini-batch updates with momentum and reduce the learning rate when validation accuracy flattens.
Read how Sutskever's List explains stochastic gradient descent →
Transformers in Action
- How it is used
- Optimizes language-model training by updating learned weights and biases to reduce loss.
- How deep it goes
- Explains gradients, update direction, learning-rate scaling, and batch-size effects, but does not detail stochastic sampling.
- Examples
- Compares convergence at learning rates of 0.1 and 0.01.
- What you can do after
- Configure training with a learning rate, batch size, and epoch count, and apply validation-based early stopping.
Read how Transformers in Action explains stochastic gradient descent →
Timeless Algorithms, The Foundational Ideas
Stochastic gradient descent is an optimization method that updates model parameters using gradients computed from individual observations or small batches.
- How it is used
- Trains neural networks through repeated gradient-based parameter updates on mini-batches.
- How deep it goes
- Explains the gradient update, learning-rate tradeoffs, mini-batch cycle, and epochs.
- Examples
- Traces network training through forward passes, loss calculation, backward gradient propagation, and mini-batch updates.
- What you can do after
- Apply mini-batch gradients to update network parameters across training epochs.
Read how Timeless Algorithms, The Foundational Ideas explains stochastic gradient descent →
Also covered in
- The AI Pocket Book
- Deep Learning with JAX
- Data Without Labels
- Grokking Machine Learning, Second EditionIn early access (MEAP)
- Grokking AI Algorithms, Second Edition
- Outlier Detection in Python
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
Related concepts
- gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
- adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- batchinguses · Batching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
- learning rateconfigured by · A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
- optimizationkind of · Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
Questions about stochastic gradient descent
What is stochastic gradient descent?
Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly sampled mini-batches.
Which Manning books explain stochastic gradient descent?
Grokking Machine Learning, Bayesian Optimization in Action, Deep Learning with Python, Third Edition, Machine Learning Algorithms in Depth, Math and Architectures of Deep Learning, Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Inside Deep Learning: Math, Algorithms, Models and 9 more.
What should I know before learning stochastic gradient descent?
Manning books that teach stochastic gradient descent usually introduce language model, objective function, parameter and gradient descent first.
What kinds of stochastic gradient descent are there?
Kinds of stochastic gradient descent include adam.
What is stochastic gradient descent a kind of?
Stochastic gradient descent is a kind of gradient descent. Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
What is stochastic gradient descent also called?
Stochastic gradient descent is also called SGD.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















