Expert collapse occurs when sparse MoE routing concentrates tokens on a small subset of experts, leaving others idle. This undermines specialization and can destabilize training. section 3.5.1
Common countermeasures include:
- Auxiliary load-balancing losses encourage a more uniform token distribution. section 3.5.1
- Entropy penalties discourage overly confident gating decisions. section 3.5.1
- Noisy gating adds stochasticity to improve training dynamics. section 3.5.1
- Expert-choice routing lets experts select tokens, improving balance by construction while risking coverage gaps. section 3.5.1
GShard-style top-2 routing
A sparse gate can select two experts per token, normalize their scores, and add load balancing, capacity constraints, and optional random routing to reduce degenerate usage patterns. section 3.5.1
Figure 3.4 Simplified MoE architecture. Only a few experts are active per input, making inference more efficient.
The architecture diagram shows sparse activation: only a few experts process each input, motivating routing while exposing the need to manage expert usage. section 3.5.1
