Sparse expert routing reduces inference compute by selecting the top-k experts for each token. The router renormalizes weights across the selected experts, then groups tokens by expert so each module processes only its assigned tokens. section 9.3.1
A sparse forward pass follows these steps:
- Score each token against the available experts and select its top-k scores. section 9.3.1
- Recalculate softmax weights over selected experts; skip experts with no assigned tokens. section 9.3.1
- Run each expert on its assigned tokens and add its weighted results to the output. section 9.3.1
Listing 9.10 TrainableTopKMoE signature
class TrainableTopKMoE(nn.Module):
def __init__(self, original_mlp, hidden_size, num_experts=2, top_k=2,
extra_experts=None):
...
def forward(self, hidden_states):
...
The routing diagram shows top-k selection and the regrouping of tokens by expert. section 9.3.1


