A mixture-of-experts (MoE) block contains expert subnetworks and a router that assigns each token to some or all of them. It enables dynamic inference: different tokens can use different computation. In a Transformer, an MoE block can replace an MLP while leaving attention unchanged. section 9.1 section 9.2
The router projects each token’s hidden state to one score per expert. Because the hidden state carries context, the same token can be routed differently in different sequences. Expert outputs are combined into a single output using routing weights. section 9.2.1 section 9.6
Routing controls both the computation and the resulting behavior:
- Soft routing evaluates both experts in the two-expert design and combines their outputs by weight. It does not save expert computation in that implementation. section 9.2.2
- Hard routing selects one expert per token. Changing the forward method can produce different behavior even when the model weights remain the same. section 9.0 section 9.2.4
- Top-k routing generalizes selection to k experts per token. At inference, select the highest logits, normalize their weights, and group assigned tokens by expert. section 9.3.1
Figure 9.4 Two views of the same top-k routing decision. Left: the router produces one logit score per expert for each token; the two highest scores per row (highlighted) are selected. Right: the same assignments reorganized by expert, showing which tokens each expert will process and their renormalized weights — for each token, those weights sum to 1 across its assigned experts. The example uses four experts rather than two so that top_k = 2 leaves half the experts idle per token. The loop in TrainableTopKMoE.forward performs exactly this regrouping.
The diagram connects per-token top-k selection to the regrouping of tokens for expert execution. section 9.3.1
Upcycling a dense model
For SmolLM2-1.7B-Instruct, replace MLPs in layers 13–23 with two-expert blocks. Keep the original MLP as a frozen general expert, deep-copy it as a clinical expert, and train the copy and a new linear router. This initialization from trained weights is sparse upcycling. Check trainable parameter counts and inspect stored routing weights for general and clinical prompts. section 9.2 section 9.2.1 section 9.2.2
Listing 9.4 Preparing model weights
for param in model.parameters():
param.requires_grad_(False) #A
for layer_idx in LAYERS_TO_MODIFY:
for param in model.model.layers[layer_idx].mlp.expert_clinical.parameters():
param.requires_grad_(True) #B
for param in model.model.layers[layer_idx].mlp.router.parameters():
param.requires_grad_(True) #B
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Total parameters : {total:,}")
print(f"Trainable (Expert 1 + router) : {trainable:,} ({trainable/total:.2%})")
The freezing step makes the clinical experts and routers trainable while preserving the remaining model weights. section 9.2.1
A second approach transplants MLPs from a fine-tuned clinical model as experts. Train the routers first, then adapt the clinical expert’s final layers if needed: the transplanted MLPs originally worked with different, fine-tuned attention layers. section 9.3 section 9.3.2 section 9.3.3
On a 40-sample clinical test set, the copied-expert model achieved 97.5% schema compliance with soft routing. The transplanted model reached 90% after router-only training and 100% after tail adaptation, for both top-k values of 1 and 2. General benchmarks were also compared with the base model; individual tasks did not all improve. These results describe the tested setups, not a guarantee of generalization or inference speed. section 9.2.2 section 9.3.2 section 9.3.3









