liveBook · Manning

Optimization

As explained in Building Specialized AI Systems

An optimizer is an algorithm that adjusts model parameters using computed gradients to minimize a training objective.

Optimization adjusts model parameters in the direction that reduces training loss. Gradient descent estimates that direction from computed gradients; the learning rate controls step size, with LLM fine-tuning commonly using rates between 1e-5 and 1e-3. section 3.1.2

Choose the optimizer according to dataset scale, model size, memory limits, and overfitting risk:

  • Use stochastic gradient descent (SGD) to estimate gradients from small random batches, accepting noisy steps that can explore the loss landscape. section 3.1.2
  • Use Adam when parameters need individually adapted learning rates; use AdamW when improved weight decay is needed to reduce overfitting. section 3.1.2
  • Use Adafactor when memory efficiency matters for very large models. section 3.1.2

One training step

For each input-target batch, the model performs a forward pass, computes cross-entropy loss, clears old gradients, backpropagates the loss, and calls optimizer.step() to update parameters. Logging the loss periodically helps track training progress. section 3.1.2

How Building Specialized AI Systems covers it

How deep it goes
Explains learning rates, mini-batch gradients, optimizer variants, and the update loop without deriving optimizer internals.
Examples
Uses language-model prompt-response batches and a training-loop listing with forward, loss, backward, and optimizer-step operations.
What it assumes
Assumes model inputs and targets, while introducing gradients, loss, learning rate, backpropagation, and parameter updates.
What you can do after
Operate/configure training with SGD, Adam, AdamW, or Adafactor and implement the optimizer step in a fine-tuning loop.
How it connects
Connects optimization to gradient descent, loss functions, backpropagation, stochastic gradient descent, and fine-tuning.

Also explained in

Compare how 38 books explain optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.