Sparsity activates only a subset of possible model components for each computation. In Mixture-of-Experts (MoE), it replaces one dense feed-forward network with parallel expert networks while keeping most experts dormant for each token. This preserves a large total parameter count without proportional training or inference cost. section 4.1.2 section 4.7
A sparse MoE computation proceeds through three linked decisions:
- A learnable routing layer scores the available experts for each token. section 4.7
- Top-k selection activates only the k highest-scoring experts; with k=2, one token uses two of three experts. section 4.2.2
- A softmax converts selected scores into weights for combining expert outputs into the single matrix expected by the next Transformer layer. section 4.2.1 section 4.7
- Specialization makes selective activation useful: experts can learn patterns for punctuation, verbs, domain information, or Python code while unrelated experts remain dormant. section 4.1.3 section 4.7
Routing “What is 1+1?”
A router may activate a punctuation expert for the question mark and a verb expert for “is,” while leaving noun and proper-noun experts inactive. The example illustrates how sparsity avoids processing every token through every expert. section 4.1.3
Figure 4.6 The principle of sparsity or load balancing. For each token, we decide to route it to only a subset (k=2) of the available experts.

Shows selective activation visually: each token is routed to only a subset of available experts, such as two of three. section 4.2.2





