liveBook · Manning

Stochastic gradient descent

As explained in Building Specialized AI Systems

SGD is stochastic gradient descent, an optimization algorithm that updates parameters using gradients estimated from individual examples or minibatches.

Stochastic gradient descent (SGD) updates model parameters using a gradient estimated from a small random batch rather than the entire dataset. This makes each step computationally practical for millions or billions of examples, while introducing noise that can help explore the loss landscape and escape shallow local minima. section 3.1.2

Use SGD as part of an iterative training loop:

  • Process a small batch, commonly 8, 16, or 32 examples, and estimate its gradient. section 3.1.2
  • Choose the learning rate carefully: large values can overshoot and destabilize training, while very small values make progress painfully slow. section 3.1.2
  • Compute outputs and loss, clear previous gradients, backpropagate, and update parameters with the optimizer. section 3.1.2

LLM fine-tuning batch

A batch can contain pairs such as Input: “Summarize this article:” and Output: the expected response. SGD uses the resulting loss signal to adjust the model incrementally. section 3.1.2

Listing 3.1 Basic steps of the training process
for inputs, targets in dataloader: #A
    outputs = model(inputs) #B
    loss = loss_fn(outputs, targets) #C
 
    optimizer.zero_grad() #D
    loss.backward() #E
    optimizer.step() #F
    if step % 10 == 0:                                   #G
        print(f"Step {step}, Loss: {loss.item():.4f}")

Shows the practical loop from inputs and targets through loss calculation, backpropagation, and the parameter update. section 3.1.2

How Building Specialized AI Systems covers it

How deep it goes
Explains minibatch estimation, learning-rate trade-offs, noise, local minima, and the parameter-update loop.
Examples
Uses minibatches of 8, 16, or 32 examples and prompt-response pairs such as article summarization.
What it assumes
Builds from gradient descent, loss, and backpropagation rather than assuming advanced optimization knowledge.
What you can do after
Apply it via a training loop that computes loss, backpropagates gradients, and updates parameters.
How it connects
Connects gradient descent with loss functions, backpropagation, fine-tuning, Adam, AdamW, and Adafactor.

Also explained in

Compare how 17 books explain stochastic gradient descent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.