Routing is the lightweight gating mechanism in a mixture-of-experts layer. It scores each token’s hidden representation, selects a small set of expert networks, and assigns their contribution. Replacing a transformer feed-forward network with sparsely activated experts enables conditional computation: total parameters can grow without activating them all for every token. section 3.5.1
A practical router typically performs these steps:
- Apply a lightweight linear layer to each token representation to produce expert scores. section 3.5.1
- Select the top-k experts, commonly one or two, rather than activating every expert. section 3.5.1
Top-2 gating
A NativeGate applies a linear projection, selects the two highest-scoring experts with topk, reshapes the selected scores, and applies softmax to obtain routing weights. A GshardGate builds on this pattern with capacity settings and optional random routing.
