liveBook · Manning

Total loss

As explained in Build a DeepSeek Model (From Scratch)

Total loss is an aggregate training error formed by summing the individual losses associated with multiple prediction targets.

Total loss aggregates multiple training errors into one objective. For multi-token prediction, it sums the individual cross-entropy losses across prediction depths, giving the model gradient signals for immediate and longer-term forecasting. section 5.3.3

Aggregation can also preserve distinct learning signals through weighting. Knowledge distillation combines hard-label cross-entropy with temperature-scaled KL divergence: L = αL_hard + (1−α)T²L_soft. The weights control the balance between ground-truth correctness and teacher guidance. section 8.4.4 section 8.6.2

A total loss supports several practical design choices:

  • Sum losses when multiple prediction targets should contribute directly to one training signal. section 5.3.3
  • Weight terms when different objectives require controlled influence, such as hard labels versus teacher-provided soft targets. section 8.4.4
  • Backpropagate the aggregate objective while updating only the intended trainable model; the frozen teacher contributes targets without receiving gradients. section 8.6.2

Numerical distillation loss

With α=0.3, temperature T=3, hard loss 0.275, and temperature-scaled soft loss 0.179, the total loss is 0.3×0.275 + 0.7×0.179 ≈ 0.208. section 8.6.2

Figure 5.14 The total loss for a single input token is the sum of the individual cross-entropy losses for each of the predicted future tokens.

Visualizes total loss as the sum of individual cross-entropy losses for future-token prediction depths. section 5.3.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines summation and weighted combination, then traces temperature scaling, KL divergence, and a numerical training example.
Examples
Uses multiple future-token targets and a numerical distillation example with logits, temperature three, and final loss 0.208.
What it assumes
Assumes individual prediction losses and targets, while grounding their aggregation in cross-entropy, KL divergence, and gradient behavior.
What you can do after
Apply it via PyTorch by summing or weighting component losses before backpropagation and the optimizer step.
How it connects
Connects cross-entropy loss, knowledge distillation, KL divergence, softmax, temperature scaling, and backpropagation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.