Auxiliary-loss-free balancing keeps Mixture-of-Experts routing balanced without adding a balancing term to the main training objective. Traditional auxiliary losses create a trade-off: weak balancing permits overloaded experts, while strong balancing can interfere with language-learning performance. section 4.4.4 section 4.7
The method intervenes directly in routing. It adds a non-backpropagated bias to each expert’s raw router score before top-k selection, then updates that bias with a fixed step size γ after every training step. section 4.4.4
Implement the self-correcting update loop as follows:
- Measure how many tokens each expert received in the current batch. section 4.4.4
- Compare each expert’s load with the average to identify its load violation. section 4.4.4
- Increase the bias of underloaded experts so the next router pass favors them. section 4.4.4
- Decrease the bias of overloaded experts so the next router pass favors them less. section 4.4.4
Fixed-step bias update
For expert i, update the routing bias with bᵢ_new = bᵢ_old + u × sign(load_violation_error). The bias changes after the training step rather than through backpropagation. section 4.4.4
