Routing assigns each input token to a small subset of experts in a mixture-of-experts (MoE) layer. Unlike uniform processing, different tokens can follow different internal paths. section 4.4.2 section 4.4.3
For each token, a gating network scores the experts using the token’s current representation. It selects the top few and converts their scores into weights for their contributions to the layer output. Selection also determines which experts receive a direct learning signal for that token. Repeated assignments reinforce useful specializations, without requiring experts to be given explicit roles. section 4.4.2 section 4.4.2
Routing has to balance specialization against training coverage and system efficiency:
- Concentrating assignments too heavily can leave some experts with too little data; distributing them too evenly can weaken specialization. section 4.4.2
- Different routes create uneven workloads across devices, complicating batching, scheduling, and hardware utilization. section 4.4.3
MoE layers are generally used selectively, often in place of Transformer feedforward blocks, while attention layers remain unchanged. Routing can therefore support greater model capacity without increasing computation per token in proportion to the number of experts. section 4.4.3 section 4.4.3
A learned division of labor
A family doctor who improves at referring patients based on feedback illustrates the loop: routing patterns help specialists develop, and their improved performance makes similar referrals more useful. section 4.4.2

































