liveBook · Manning

Stochastic gradient descent

As explained in Timeless Algorithms, The Foundational Ideas

Stochastic gradient descent is an optimization method that updates model parameters using gradients computed from individual observations or small batches.

Stochastic gradient descent trains a network through repeated parameter updates. For each mini-batch, the network computes predictions and loss, propagates gradients backward, and adjusts its parameters. The learning rate sets the adjustment size: too large can overshoot better values, while too small can slow training. section 11.2.5

θnew=θold-η∂L∂θ

The update subtracts the loss gradient, scaled by the learning rate, from the current parameter value. section 11.2.5

An epoch is one complete pass through the data and contains multiple mini-batch updates. Repeating updates across epochs gradually reduces loss and improves predictions; mini-batch updates also reduce the computational burden of each step and add variation to the search. section 11.2.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains the gradient update, learning-rate tradeoffs, mini-batch cycle, and epochs.
Examples
Traces network training through forward passes, loss calculation, backward gradient propagation, and mini-batch updates.
What you can do after
Apply mini-batch gradients to update network parameters across training epochs.
How it connects
Connects loss gradients and learning rate to mini-batches, backpropagation, and epochs.

Also explained in

Compare how 17 books explain stochastic gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.