liveBook · Manning

Load-balancing loss

As explained in Build a DeepSeek Model (From Scratch)

A load-balancing loss is an auxiliary objective that encourages routing assignments to be distributed evenly across mixture-of-experts modules.

A load-balancing loss is an auxiliary objective for mixture-of-experts routing. It penalizes uneven token dispatch by combining each expert’s dispatched-token fraction with the router’s mean selection probability. section 4.3.2

The loss separates two signals for every expert:

  • p_i measures router probability: the expert’s total importance normalized by the number of tokens in the batch. section 4.3.2
  • f_i measures the fraction of batch tokens actually dispatched after top-k selection. section 4.3.2
  • The objective scales the sum of f_i multiplied by p_i across experts by the expert count N and hyperparameter λ. section 4.3.2

Why probability alone is insufficient

One expert can receive one token with probability 1.0, while another receives four tokens with probability 0.25 each. Both have importance 1.0, yet the second handles four times the workload; dispatched-token fractions expose this imbalance. section 4.3.2

Figure 4.16 An example demonstrating that equal expert importance does not guarantee a balanced token load.

Use the routing example and companion equation to inspect why equal expert importance does not guarantee balanced token loads. section 4.3.2

The practical trade-off is visible in comparison experiments: a conventional Standard MoE shows uneven expert selection, whereas an alternative DeepSeek-MoE uses dynamic balancing and achieves lower validation loss and higher throughput in the reported run. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives p_i and f_i and gives their product-based objective, but stops short of implementation code.
Examples
Uses a two-expert workload counterexample, batch-level p_i and f_i calculations, and Standard-MoE selection-frequency comparisons.
What it assumes
Assumes mixture-of-experts routing, router probabilities, expert importance, batches, and top-k token selection.
What you can do after
Apply via an auxiliary objective that accounts for both expert probabilities and dispatched-token counts.
How it connects
Connects mixture-of-experts, router probability, top-k selection, expert importance, and auxiliary loss.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.