Sparse gating routes each input token to only the highest-scoring experts in a mixture-of-experts model. A lightweight gating layer scores experts from the token’s hidden representation, then applies top-k selection and normalizes the selected scores. section 3.5.1
Routing variants trade computation, differentiability, stability, and coverage:
- Sparse gating activates the top-k experts, reducing conditional computation. section 3.5.1
- Hard gating selects one expert, maximizing sparsity while potentially making training less stable. section 3.5.1
- Soft and dense gating involve all experts, while expert-choice routing lets experts select tokens but can leave coverage gaps. section 3.5.1
- Load balancing requires techniques such as auxiliary losses or expert-choice routing to limit expert collapse. section 3.5.1
Top-k implementation
A NativeGate applies a linear layer to each hidden vector, selects the top-k logits with torch.topk, reshapes the selected values, and applies softmax over the selected experts. A GshardGate extends this pattern with top-2 routing, capacity settings, random routing, and load-balancing behavior.
