Stochastic gradient descent
Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
Grokking Machine Learning
A gradient-descent variant that updates model parameters using one training data point at a time.
- How it is used
- Trains linear regression models and perceptron classifiers through repeated point-wise parameter adjustments.
- How deep it goes
- Defines point-wise updates and contrasts batch sizes, while deferring full mathematical details to gradient-descent appendices.
- Examples
- Uses a regression line, individual data points, and a perceptron line responding to correctly or incorrectly classified points.
- What you can do after
- Apply it conceptually by performing repeated single-point updates that reduce regression error or improve classification.
Read how Grokking Machine Learning explains stochastic gradient descent →
Bayesian Optimization in Action
Mini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
- How it is used
- Scales loss optimization to datasets too large for full-batch Gaussian-process training.
- How deep it goes
- Defines batch approximation, explains its noisy descent behavior, and demonstrates the PyTorch training loop without deriving optimizer internals.
- Examples
- Uses loss-valley illustrations, PyTorch loops, variational Gaussian processes, and a housing-price timing exercise.
- What you can do after
- Apply it via PyTorch or GPyTorch to train models with repeated batch-level loss and gradient updates.
Read how Bayesian Optimization in Action explains stochastic gradient descent →
Deep Learning with Python, Third Edition
A gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
- How it is used
- Updates neural-network weights from randomly drawn training batches to reduce a loss function.
- How deep it goes
- Works through forward pass, loss, backward pass, and weight update; compares batch sizes and learning-rate choices without a dense derivation.
- Examples
- Uses a one-parameter loss curve, handwritten-digit training in Keras, and a PyTorch linear-classifier training step.
- What you can do after
- Apply it via Keras fit or torch.optim.SGD, or perform parameter updates and gradient resets in a PyTorch training step.
Read how Deep Learning with Python, Third Edition explains stochastic gradient descent →
Machine Learning Algorithms in Depth
Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
- How it is used
- Optimizes classifier and neural-network parameters through repeated gradient-based updates on sampled training data.
- How deep it goes
- Derives mini-batch updates and learning-rate effects, then relates perceptron updates to hinge-loss gradients and compares neural-network optimizers.
- Examples
- Uses an iris perceptron classifier and a LeNet-on-CIFAR-100 comparison of cross-entropy loss and validation accuracy.
- What you can do after
- Build it from scratch for perceptron training and apply it via Keras/TensorFlow to compare neural-network optimizers.
Read how Machine Learning Algorithms in Depth explains stochastic gradient descent →
Math and Architectures of Deep Learning
- How it is used
- Uses randomly sampled minibatches to estimate gradients and iteratively reduce neural-network loss.
- How deep it goes
- Explains dataset-scale motivation, geometric intuition, noisy estimates, and a complete PyTorch update loop.
- Examples
- Demonstrates height-weight classification with shuffled batches, cross-entropy loss, PyTorch SGD, and decreasing epoch loss.
- What you can do after
- Apply it via PyTorch by preparing a DataLoader, computing minibatch loss, backpropagating, and stepping an SGD optimizer.
Read how Math and Architectures of Deep Learning explains stochastic gradient descent →
Natural Language Processing in Action, Second Edition
Stochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
- How it is used
- Deploys sampled parameter updates to train a PyTorch neural model while balancing convergence speed and update stability.
- How deep it goes
- Explains sampling, shuffling, gradients, learning rate, momentum, batches, and mini-batches, then demonstrates a runnable training loop.
- Examples
- Trains a name-gender classifier with vectorized names, binary cross-entropy, weighted examples, test accuracy, and PyTorch epoch logs.
- What you can do after
- Apply it via PyTorch by configuring SGD, running gradient updates, and monitoring training and test metrics.
Deep Learning with R, Third Edition
Mini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
- How it is used
- Optimizes neural-network weights during training by using loss gradients from randomly sampled data batches.
- How deep it goes
- Explains sampling, forward and backward passes, parameter updates, batch-size tradeoffs, and learning-rate risks without formal derivation.
- Examples
- Demonstrates MNIST digit classification with Keras Dense layers, Adam, five epochs, and batches of 128 samples.
- What you can do after
- Apply it via Keras by choosing an optimizer, loss, batch size, and training schedule for a neural-network model.
Read how Deep Learning with R, Third Edition explains stochastic gradient descent →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Mini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
- How it is used
- Handles large streamed datasets by optimizing model weights incrementally from batches or individual examples.
- How deep it goes
- Explains gradients, iterative updates, batch processing, and error landscapes, but does not derive the full optimization mathematics.
- Examples
- Uses streamed training batches, reserved test data, logistic regression code, optimization-landscape figures, and a reported accuracy of 0.818.
- What you can do after
- Apply SGD via scikit-learn’s SGDClassifier to train logistic regression incrementally on batches.
How Large Language Models Work
Stochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.
- How it is used
- Uses random training subsets to make neural-network parameter updates feasible when processing all training data is too expensive.
- How deep it goes
- Explains the subset-based update and its memory rationale, but skips detailed mathematics and implementation procedures.
- Examples
- Uses labeled inputs, parameter adjustments, a downhill ball, and loss-function graphs to illustrate training.
- What you can do after
- Evaluate/understand why training uses small data subsets and how those subsets approximate full-data gradient steps.
Read how How Large Language Models Work explains stochastic gradient descent →
Deep Learning with PyTorch, Second Edition
An optimization method that updates parameters using stochastic gradient descent.
- How it is used
- Trains neural-network parameters by minimizing a loss during forward, backward, and optimizer steps.
- How deep it goes
- Explains gradient-based updates and PyTorch setup, but defers detailed trade-offs among alternative optimizers.
- Examples
- Demonstrates torch.optim.SGD in a tumor-classification application with CUDA placement, learning rate 0.001, and momentum 0.99.
- What you can do after
- Apply it via PyTorch by configuring SGD, connecting model parameters, and integrating it into a training loop.
Read how Deep Learning with PyTorch, Second Edition explains stochastic gradient descent →
Sutskever's List
- How it is used
- Trains large neural networks by updating weights from mini-batch gradients.
- How deep it goes
- Explains noisy gradient estimates, momentum, and learning-rate scheduling, while distinguishing first-order updates from curvature information.
- Examples
- Uses a foggy-hiker analogy and AlexNet's 128-image batches, 0.01 initial rate, and validation-triggered tenfold reductions.
- What you can do after
- Apply mini-batch updates with momentum and reduce the learning rate when validation accuracy flattens.
Read how Sutskever's List explains stochastic gradient descent →
Transformers in Action
- How it is used
- Optimizes model parameters during language-model training by moving them against the loss gradient.
- How deep it goes
- Explains parameter updates and learning-rate, batch-size, and epoch effects on training.
- Examples
- Compares gradient-descent figures using learning rates of 0.1 and 0.01.
- What you can do after
- Evaluate how learning rate, batch size, and epochs affect updates, convergence, and overfitting.
Read how Transformers in Action explains stochastic gradient descent →
Related concepts
- gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
- adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
- batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
- learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
- logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.