A routing bias is a persistent per-expert score adjustment added to raw router scores before top-k selection. It influences expert choice without being learned through backpropagation. section 4.4.4
Use the bias as a self-correcting load-balancing loop:
- Measure each expert’s token load after a training step. section 4.4.4
- Increase the bias of underloaded experts and decrease the bias of overloaded experts. section 4.4.4
- Apply the fixed-step update for the next training step, raising or lowering future selection scores. section 4.4.4
