liveBook · Manning

Optimization

As explained in Build a DeepSeek Model (From Scratch)

An optimizer is an algorithm that updates model parameters using gradients to minimize a training objective.

Optimization turns computed gradients into parameter updates that reduce the training loss. In a PyTorch training script, the optimizer is configured after model initialization, receives the model parameters, and updates them after each backward pass. section 6.3.3

A practical optimizer step follows this sequence:

  • Fetch a mini-batch and run the model to calculate the loss. section 6.4.1
  • Call loss.backward() to calculate gradients for every model weight. section 6.4.1
  • Call optimizer.step() to nudge the weights in the direction that reduces loss, then clear gradients with optimizer.zero_grad(). section 6.4.1
  • Update the learning rate from a scheduler before the next iteration. section 6.3.3

Training-loop update

A typical iteration runs the model under mixed precision, obtains the combined loss, clears old gradients, computes new gradients with loss.backward(), and updates the weights with optimizer.step(). The optimizer state is saved alongside model weights in checkpoints so training can resume.

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains configuration and the forward-backward-update cycle, but does not detail AdamW’s internal update rule.
Examples
Uses PyTorch AdamW, optimizer.step(), optimizer.zero_grad(), and checkpoints containing optimizer state.
What it assumes
Assumes model parameters, loss, gradients, and forward/backward passes; grounds them in a mini-batch training loop.
What you can do after
Operate/configure it in a PyTorch training loop by scheduling its learning rate, applying gradients, and saving its state.
How it connects
Connects gradients, loss functions, learning-rate schedules, mixed precision, evaluation, and checkpointing.

Also explained in

Compare how 38 books explain optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.