Specialization can mean adapting a pretrained model for a new objective or domain, or learning distinct competencies across subnetworks. Both build on prior learning rather than starting from scratch, but they work differently: adaptation changes or augments model parameters, while expert-based specialization develops through routing and training. section 4.3.1 section 4.4.2
Figure 4.13 Pretraining and model adaptation. Models are first trained on broad data, then specialized through full or partial parameter updates.

The diagram distinguishes broad pretraining from later specialization through full or partial parameter updates. section 4.3.1
In a mixture-of-experts (MoE) layer, a gating network scores experts for each token and selects a small subset. Selected experts contribute to the output and receive a learning signal. Across training, experts improve on patterns they repeatedly receive, while the gate learns which assignments reduce error. Competencies emerge from this feedback loop, not from explicit labels assigning each expert a role. section 4.4.2
MoE specialization depends on balancing routing:
- Concentrating assignments too heavily leaves some experts with too little data to learn useful behavior. section 4.4.2
- Spreading assignments too evenly mixes patterns across experts and weakens specialization. section 4.4.2
- Selective activation can increase model capacity without proportionally increasing computation per token, but uneven token paths make batching and hardware scheduling harder. section 4.4.3
Expert layers typically replace feedforward blocks sparingly in a Transformer, while attention remains shared. Specialization can expand capacity, but it does not remove the costs of routing or the risk that further adaptation weakens capabilities learned during pretraining. section 4.4.3 section 4.3.1
A family doctor and specialists
The gate resembles a doctor learning where to route patients from feedback, while specialists improve in the areas they see most often. section 4.4.2



