A mixture of experts (MoE) expands a model’s capacity by letting different inputs use different subsets of its parameters. Unlike a dense layer, which applies the same full computation to every token, an MoE layer activates only selected subnetworks for each token. This can increase total capacity without increasing per-token computation proportionally. section 4.4.1
In a Transformer, an MoE layer can replace a feedforward block with a pool of independent subnetworks called experts. A gating network scores the token’s representation, selects a small subset of experts, and assigns weights to their contributions. The selected experts run in parallel, and their outputs are combined; unselected experts remain inactive for that token. section 4.4.1 section 4.4.2
Figure 4.17 Structure of a Mixture of Experts layer. A gating network selects which experts should process each input and assigns a weight to their contributions. The chosen experts run in parallel, and their outputs are combined into a weighted sum. Experts that are not selected for a given input, such as the grayed-out Expert 3, remain inactive. By activating only a subset of experts at a time, the model increases its capacity without increasing the computation required at runtime.

Follow a token through the gate, selected experts, and weighted output. section 4.4.1
Specialization is learned rather than assigned by hand. Selected experts receive a direct learning signal from their contribution to the loss, while the gate adjusts its routing as expert choices affect performance. Repeated assignments can make experts better at particular patterns and make those assignments more useful. section 4.4.2
Routing requires a balance. If too few experts receive most tokens, the others may not learn useful behavior; if tokens are spread too evenly, specialization can weaken. Selective computation also complicates execution: tokens in the same batch may take different paths, creating uneven device workloads and making batching and scheduling less efficient. section 4.4.2 section 4.4.3
MoE layers are typically used sparingly within Transformer stacks, while attention layers remain unchanged. This combines shared processing with selective additional capacity rather than replacing the underlying architecture. Mixtral is one example of a model using this approach. section 4.4.3 section 4.4.1









