Sparse upcycling converts a pretrained dense model into a sparse mixture-of-experts (MoE) model by initializing experts with copies of trained modules, rather than random weights. This reuses learned representations and can speed specialization. section 9.2 section 9.6
A two-expert implementation follows this pattern:
- Replace selected Transformer-block MLPs with MoE blocks; leave attention unchanged. section 9.2
- Copy the pretrained MLP weights into two experts and add a router that assigns weights based on hidden states. section 9.2 section 9.6
- In the clinical-specialization setup, freeze the original general expert and the rest of the model; train the copied expert and routers. section 9.2
The practical setup differs from the cited research procedure: it freezes the general expert and adds Gaussian noise to router logits to encourage early routing exploration. Research results report lower training costs and gains over dense or from-scratch baselines, but those cost figures do not apply to this modified setup. section 9.4.1
Figure 9.1 The MoE block replaces the MLP module in selected transformer layers. A router receives the hidden states and assigns routing weights to two experts that process in parallel — expert_general, which holds the original MLP weights (frozen), and expert_clinical, the trainable copy. Attribute names in parentheses correspond directly to the model's code structure.
The diagram shows how a router and two parallel experts replace the MLP in selected Transformer layers. section 9.2
