How deeply does How Large Language Models Work cover stochastic gradient descent?

Explains the subset-based update and its memory rationale, but skips detailed mathematics and implementation procedures.

Stochastic gradient descent

As explained in How Large Language Models Work

Stochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.

Stochastic gradient descent (SGD) updates a neural network using a small random subset of training data instead of the entire dataset. Each subset estimates the next small move toward a lower loss. section 4.1.2

Using less data at each step greatly reduces memory requirements and speeds training. The estimate is usually sufficient because gradient descent changes parameters only a little at a time, though it is not the same as evaluating all data. section 4.1.2

A small step toward lower loss

Imagine a ball moving downhill on a loss-function graph. Inputs and known labels suggest how to adjust parameters; repeating these small updates millions or billions of times transforms the network into something useful.

How How Large Language Models Work covers it

liveBook manning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking Machine LearningA gradient-descent variant that updates model parameters using one training data point at a time.
liveBook manning.com

Bayesian Optimization in ActionMini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
liveBook manning.com

Deep Learning with Python, Third EditionA gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
liveBook manning.com

Machine Learning Algorithms in DepthStochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
liveBook manning.com

Math and Architectures of Deep Learning
liveBook manning.com

Natural Language Processing in Action, Second EditionStochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
liveBook manning.com

Deep Learning with R, Third EditionMini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIMini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
liveBook manning.com

Deep Learning with PyTorch, Second EditionAn optimization method that updates parameters using stochastic gradient descent.
liveBook manning.com

Sutskever's List
liveBook manning.com

Transformers in Action

Compare how 12 books explain stochastic gradient descent →

Related concepts

gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.