liveBook · Manning

Auxiliary loss

As explained in Build a DeepSeek Model (From Scratch)

An auxiliary loss is an additional penalty term optimized alongside a model's main training loss to encourage a desired property such as balanced expert selection.

An auxiliary loss adds a penalty to the main next-token prediction loss to encourage balanced expert selection in a mixture-of-experts model. It becomes large when expert importance is uneven and small when importance is balanced, guiding the router through backpropagation. section 4.3.1

Calculate the penalty from expert-selection statistics:

  • Sum each column of the expert selector weight matrix to obtain an expert's importance for the batch. section 4.3.1
  • Measure dispersion with the coefficient of variation, CV = σ / μ; higher CV indicates greater imbalance. section 4.3.1
  • Scale the squared variation by λ, giving Auxiliary Loss = λ × (CV)², then add it to the primary training objective. section 4.3.1

Importance can hide routing imbalance

For three experts, column sums of 1.4, 1.0, and 1.6 produce a CV of 0.187, exposing uneven importance. However, equal importance can still misrepresent workload: one expert may receive one token at probability 1.0 while another receives four tokens at probability 0.25 each. Their importance is equal, but the second expert processes four times as many tokens. section 4.3.1 section 4.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines column-summed importance, derives coefficient of variation, and gives a squared penalty, but stops short of implementation code.
Examples
Uses three expert importance scores of 1.4, 1.0, and 1.6, plus a two-expert case exposing unequal token loads.
What it assumes
Builds from an expert selector weight matrix and token-level routing probabilities before defining importance.
What you can do after
Apply it via a coefficient-of-variation penalty added to a mixture-of-experts training loss.
How it connects
Connects mixture-of-experts routing, expert importance, coefficient of variation, load balancing loss, and auxiliary-loss-free load balancing.

Also explained in

Compare how 2 books explain auxiliary loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.