A load-balancing loss is an auxiliary objective for mixture-of-experts routing. It penalizes uneven token dispatch by combining each expert’s dispatched-token fraction with the router’s mean selection probability. section 4.3.2
The loss separates two signals for every expert:
- p_i measures router probability: the expert’s total importance normalized by the number of tokens in the batch. section 4.3.2
- f_i measures the fraction of batch tokens actually dispatched after top-k selection. section 4.3.2
- The objective scales the sum of f_i multiplied by p_i across experts by the expert count N and hyperparameter λ. section 4.3.2
Why probability alone is insufficient
One expert can receive one token with probability 1.0, while another receives four tokens with probability 0.25 each. Both have importance 1.0, yet the second handles four times the workload; dispatched-token fractions expose this imbalance. section 4.3.2
Figure 4.16 An example demonstrating that equal expert importance does not guarantee a balanced token load.

Use the routing example and companion equation to inspect why equal expert importance does not guarantee balanced token loads. section 4.3.2
The practical trade-off is visible in comparison experiments: a conventional Standard MoE shows uneven expert selection, whereas an alternative DeepSeek-MoE uses dynamic balancing and achieves lower validation loss and higher throughput in the reported run. section 4.6
