Stochastic gradient descent

As explained in Grokking Machine Learning

A gradient-descent variant that updates model parameters using one training data point at a time.

How can a model improve without measuring every training example at once? Stochastic gradient descent answers by updating the model with one point per iteration, nudging its parameters to reduce that point’s error. It contrasts with mini-batch gradient descent, which uses several points, and batch gradient descent, which uses the entire dataset. section 3.4

The basic pattern is:

• Start with a model, such as a line or a perceptron separator. section 3.4 section 5.3
• Choose one training point and inspect how well the current model fits or classifies it. section 3.4 section 5.3
• Move the model slightly in the direction that improves its fit, repeating the process many times. section 3.4 section 5.3

A line that learns from individual points

Imagine a line separating cats from dogs. A correctly classified point leaves the line where it is; a misclassified point moves the line slightly toward that point. Repeating this with randomly selected points can produce a separator that classifies most examples correctly. For regression, the same point-by-point idea moves a line toward individual data points while reducing error. section 5.3 section 3.4

Figure 5.23 If we apply the perceptron trick many times, each time picking a random point, we can imagine that we’ll obtain a classifier that classifies most points correctly.

Visualizes repeated point-by-point updates and the gradual improvement of a classifier. section 5.3

How Grokking Machine Learning covers it

How deep it goes

Defines point-wise updates and contrasts batch sizes, while deferring full mathematical details to gradient-descent appendices.

Examples

Uses a regression line, individual data points, and a perceptron line responding to correctly or incorrectly classified points.

What you can do after

Apply it conceptually by performing repeated single-point updates that reduce regression error or improve classification.

How it connects

Connects gradient descent with error functions, linear regression, perceptron algorithms, mini-batch gradient descent, and batch gradient descent.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Bayesian Optimization in ActionMini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthStochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionStochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionMini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIMini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
• 

  liveBookmanning.com
  How Large Language Models WorkStochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization method that updates parameters using stochastic gradient descent.
• 

  liveBookmanning.com
  Sutskever's List
• 

  liveBookmanning.com
  Transformers in Action

Compare how 12 books explain stochastic gradient descent →

Related concepts

• gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
• logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.