liveBook · Manning

Sparse upcycling

As explained in Rearchitecting LLMs

Sparse Upcycling is a method for converting a pretrained dense model into a sparse mixture-of-experts model, commonly by duplicating feed-forward components as experts and further training the resulting architecture.

Sparse upcycling converts a pretrained dense model into a sparse mixture-of-experts (MoE) model by initializing experts with copies of trained modules, rather than random weights. This reuses learned representations and can speed specialization. section 9.2 section 9.6

A two-expert implementation follows this pattern:

  • Replace selected Transformer-block MLPs with MoE blocks; leave attention unchanged. section 9.2
  • Copy the pretrained MLP weights into two experts and add a router that assigns weights based on hidden states. section 9.2 section 9.6
  • In the clinical-specialization setup, freeze the original general expert and the rest of the model; train the copied expert and routers. section 9.2

The practical setup differs from the cited research procedure: it freezes the general expert and adds Gaussian noise to router logits to encourage early routing exploration. Research results report lower training costs and gains over dense or from-scratch baselines, but those cost figures do not apply to this modified setup. section 9.4.1

Figure 9.1 The MoE block replaces the MLP module in selected transformer layers. A router receives the hidden states and assigns routing weights to two experts that process in parallel — expert_general, which holds the original MLP weights (frozen), and expert_clinical, the trainable copy. Attribute names in parentheses correspond directly to the model's code structure.

The diagram shows how a router and two parallel experts replace the MLP in selected Transformer layers. section 9.2

How Rearchitecting LLMs covers it

How deep it goes
Details selected-layer MLP replacement, expert copying, freezing, and router training, while adapting rather than exactly replicating the research procedure.
Examples
Uses SmolLM2-1.7B-Instruct with two experts for clinical extraction and selects layers 13 through 23.
What it assumes
Requires a pretrained dense model with MLP weights that can initialize experts.
What you can do after
Build a two-expert MoE by copying pretrained MLP weights and training the copied expert and routers.
How it connects
Connects Mixture of Experts to Transformer MLPs, routers, hidden states, and soft or hard routing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.