Stochastic gradient descent

As explained in Math and Architectures of Deep Learning

Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.

Stochastic gradient descent (SGD) minimizes a neural network’s loss without processing the entire training dataset for every update. It estimates the gradient from a randomly sampled minibatch, then uses that estimate to update weights and biases. section 9.2.2 section 9.4

If B is a minibatch, SGD averages the losses or gradients for the examples in B and moves parameters in the negative-gradient direction. Geometrically, each update moves the model down its loss hypersurface, although the sampled estimate introduces noise. section 9.2.2 section 9.4

A PyTorch training step follows this sequence:

• Load a shuffled minibatch through a Dataset and DataLoader. section 9.2.3
• Run the model forward and compute the minibatch loss. section 9.2.3
• Clear old gradients, backpropagate the loss, and call optimizer.step() to update parameters. section 9.2.3

Classifying Statsville residents

A neural network receives height and weight and predicts man, woman, or child. A DataLoader supplies shuffled batches of 10 examples; CrossEntropyLoss measures each batch, and optim.SGD(model.parameters(), lr=0.02) updates the model. Across epochs, the loss decreases and predictions become more accurate. section 9.2.3

Figure 9.6 Height and weight of various Statsville residents. Class 0 (man) is represented by the right-most cluster, class 1 (woman) by the middle cluster, and class 2 (child) by the left-most cluster.

Inspect the height-weight clusters that provide the classifier’s training distribution and make minibatch optimization concrete. section 9.2.3

How Math and Architectures of Deep Learning covers it

How deep it goes

Explains dataset-scale motivation, geometric intuition, noisy estimates, and a complete PyTorch update loop.

Examples

Demonstrates height-weight classification with shuffled batches, cross-entropy loss, PyTorch SGD, and decreasing epoch loss.

What it assumes

Assumes neural-network weights, biases, model outputs, and loss functions while teaching minibatch-based optimization procedure.

What you can do after

Apply it via PyTorch by preparing a DataLoader, computing minibatch loss, backpropagating, and stepping an SGD optimizer.

How it connects

Connects minibatches and gradients to loss functions, optimization, neural-network parameters, and momentum-based optimization.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA gradient-descent variant that updates model parameters using one training data point at a time.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionMini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthStochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionStochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionMini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIMini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
• 

  liveBookmanning.com
  How Large Language Models WorkStochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization method that updates parameters using stochastic gradient descent.
• 

  liveBookmanning.com
  Sutskever's List
• 

  liveBookmanning.com
  Transformers in Action

Compare how 12 books explain stochastic gradient descent →

Related concepts

• gradient descentkind of · Gradient descent is an iterative optimization method that uses the negative gradient of an objective function to update parameters toward lower objective values.
• adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• batchinguses · Batching is the grouping of multiple data items or operations so they can be processed together as a unit.
• learning rateconfigured by · A learning rate is a parameter that controls the size or contribution of successive updates in an iterative learning or optimization process.
• logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.