Conditional computation activates only part of a model for each input. In transformer-based mixture-of-experts layers, multiple expert feed-forward networks replace one dense feed-forward network, while a lightweight router selects the most relevant experts for each token. section 3.5.1
The architecture changes computation in three connected ways:
- Replaces the transformer feed-forward network with many smaller expert networks, commonly activating only the top one or two experts per token. section 3.5.1
- Uses a gating network, or router, to score expert relevance from each token's hidden representation through a lightweight linear layer and softmax or top-k operation. section 3.5.1
- Scales total parameter capacity without increasing inference cost proportionally, because each forward pass activates only a small parameter subset. section 3.5.1
Top-k routing
A sparse gate applies a linear layer to the input, selects the highest-scoring experts with top-k, and normalizes those selected scores with softmax. A GShard-style variant adds top-two routing, capacity limits, load balancing, and optional random routing. section 3.5.1
Figure 3.4 Simplified MoE architecture. Only a few experts are active per input, making inference more efficient.
Use the simplified architecture figure to trace how a token is routed to only a few experts instead of every feed-forward component. section 3.5.1
