An auxiliary loss adds a penalty to the main next-token prediction loss to encourage balanced expert selection in a mixture-of-experts model. It becomes large when expert importance is uneven and small when importance is balanced, guiding the router through backpropagation. section 4.3.1
Calculate the penalty from expert-selection statistics:
- Sum each column of the expert selector weight matrix to obtain an expert's importance for the batch. section 4.3.1
- Measure dispersion with the coefficient of variation, CV = σ / μ; higher CV indicates greater imbalance. section 4.3.1
- Scale the squared variation by λ, giving Auxiliary Loss = λ × (CV)², then add it to the primary training objective. section 4.3.1
Importance can hide routing imbalance
For three experts, column sums of 1.4, 1.0, and 1.6 produce a CV of 0.187, exposing uneven importance. However, equal importance can still misrepresent workload: one expert may receive one token at probability 1.0 while another receives four tokens at probability 0.25 each. Their importance is equal, but the second expert processes four times as many tokens. section 4.3.1 section 4.3.2

