liveBook · Manning

Stochastic gradient descent

As explained in Fight Fraud with Machine Learning

Stochastic gradient descent is an optimization method that updates model parameters using gradients estimated from individual examples or small batches.

Stochastic gradient descent (SGD) trains a model by repeatedly adjusting parameters in the direction indicated by estimated gradients. Individual examples or small batches can supply those estimates, while the learning rate controls update size. section 4.1.1

A practical SGD training loop:

  • Calculate the loss and gradients for the current model parameters. section 4.1.1
  • Update each weight using its gradient multiplied by the learning rate. section 4.1.1
  • Repeat updates until the loss reaches a minimum or stabilizes. section 4.1.1

Choosing the learning rate

A learning rate set too high can overshoot the optimum or prevent convergence; one set too low can make training slow or stall in local minima. section 4.1.1

Figure 4.6 Steps in the training of a deep learning model using gradient descent.

Shows the training steps of a deep learning model using gradient descent. section 4.1.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains gradient-based updates, learning-rate trade-offs, and convergence, but stops short of optimizer implementation details.
Examples
Uses a deep-learning training workflow and contrasts overly high and low learning rates.
What it assumes
Assumes model weights, gradients, loss, and neural-network training, while explaining the update loop from those elements.
What you can do after
Build it from scratch by implementing iterative weight updates and checking whether the loss converges.
How it connects
Connects SGD with gradient descent, loss functions, learning rate, Adagrad, momentum, Adam, and gradient-boosted trees.

Also explained in

Compare how 17 books explain stochastic gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.