Stochastic gradient descent

As explained in Deep Learning with R, Third Edition

Mini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.

Stochastic gradient descent (SGD) trains a model by repeatedly adjusting its parameters to reduce loss. Mini-batch SGD randomly selects a small batch, runs a forward pass, computes loss and gradients during the backward pass, then moves parameters slightly opposite the gradient. The learning rate controls the update size. section 2.4.3 section 2.6

A training update follows this loop:

• Draw a random batch of samples and targets. section 2.4.3
• Run the model on the batch to produce predictions. section 2.4.3
• Measure prediction error with a loss function and compute gradients with backpropagation. section 2.5 section 2.6
• Update the weights in the direction that reduces the batch loss. section 2.4.3

Training a digit classifier

A Keras model can classify normalized 28×28 MNIST images with two Dense layers. Compiling with Adam, sparse categorical cross-entropy, and accuracy selects the optimizer, feedback signal, and metric; fitting for five epochs with batches of 128 performs mini-batch updates. The same training process can be understood as repeatedly moving the network’s weight tensors toward lower loss. section 2.5

Mini-batch SGD balances two extremes: true SGD uses one sample per update, while batch gradient descent uses all available data. Larger batches can make each update more accurate but more expensive. A learning rate that is too small slows descent; one that is too large can send updates to unhelpful locations. Optimizers can specify variations such as Adam, RMSProp, or SGD with momentum. section 2.4.3 section 2.6

Figure 2.17 SGD down a 1D loss curve (one learnable parameter)

Use the one-parameter loss-curve figure to visualize how gradient-based updates descend toward lower loss. section 2.4.3

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains sampling, forward and backward passes, parameter updates, batch-size tradeoffs, and learning-rate risks without formal derivation.

Examples

Demonstrates MNIST digit classification with Keras Dense layers, Adam, five epochs, and batches of 128 samples.

What it assumes

Assumes basic models, data, and loss while teaching forward passes, gradients, backpropagation, and weight updates from the training loop.

What you can do after

Apply it via Keras by choosing an optimizer, loss, batch size, and training schedule for a neural-network model.

How it connects

Connects loss functions, optimizers, backpropagation, learning rate, batch gradient descent, and neural-network weights.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA gradient-descent variant that updates model parameters using one training data point at a time.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionMini-batch gradient descent is an optimization technique that updates model parameters using loss and gradient estimates computed from small subsets of the training data.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthStochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from random data subsets.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionStochastic gradient descent is an iterative optimization technique that updates model parameters using randomized or sampled training examples rather than the entire dataset at once.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIMini-batch gradient descent is an optimization algorithm that updates model parameters using gradients computed from small batches of examples.
• 

  liveBookmanning.com
  How Large Language Models WorkStochastic gradient descent is a gradient-descent technique that estimates parameter updates using a small random subset of training data.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization method that updates parameters using stochastic gradient descent.
• 

  liveBookmanning.com
  Sutskever's List
• 

  liveBookmanning.com
  Transformers in Action

Compare how 12 books explain stochastic gradient descent →

Related concepts

• gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
• batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• learning rateconfigured by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
• logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.