An expert network is a subnetwork in a mixture-of-experts (MoE) layer. The layer replaces a standard feedforward block with a pool of experts; a gating network routes each token representation to a small subset, whose weighted outputs are combined. Unselected experts stay inactive for that token. section 4.4.1
Experts are not assigned explicit competencies. Their specializations emerge through training: selected experts receive learning signals, while the gate learns which routing choices improve performance. Repeated routing and feedback reinforce useful divisions of responsibility. section 4.4.2
Selective activation brings both capacity gains and operational trade-offs:
- Adding experts can increase model capacity without a proportional increase in computation per token. section 4.4.3
- Uneven routing can leave some experts undertrained; varied token paths can also complicate batching, scheduling, and hardware utilization. section 4.4.2 section 4.4.3
Specialist referral
A family doctor directs cases to specialists; repeated referrals help the doctor route more effectively, while specialists improve in the cases they see most often. section 4.4.2
Figure 4.17 Structure of a Mixture of Experts layer. A gating network selects which experts should process each input and assigns a weight to their contributions. The chosen experts run in parallel, and their outputs are combined into a weighted sum. Experts that are not selected for a given input, such as the grayed-out Expert 3, remain inactive. By activating only a subset of experts at a time, the model increases its capacity without increasing the computation required at runtime.

The figure shows the gate selecting experts, the selected subnetworks processing an input in parallel, and their weighted outputs being combined. section 4.4.1




