During MoE specialization, freezing keeps the general expert fixed while selected clinical experts and routers learn. This limits which model weights training can update. section 9.2.1
A practical setup:
- Set every parameter to non-trainable. section 9.2.1
- For each modified MoE layer, enable training for the clinical expert and router; leave the general expert fixed. section 9.2.1
- Count trainable parameters to check the scope: this configuration trains 24.44% of the model weights. section 9.2.1
Listing 9.4 Preparing model weights
for param in model.parameters():
param.requires_grad_(False) #A
for layer_idx in LAYERS_TO_MODIFY:
for param in model.model.layers[layer_idx].mlp.expert_clinical.parameters():
param.requires_grad_(True) #B
for param in model.model.layers[layer_idx].mlp.router.parameters():
param.requires_grad_(True) #B
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Total parameters : {total:,}")
print(f"Trainable (Expert 1 + router) : {trainable:,} ({trainable/total:.2%})")
Use the parameter-freezing snippet to configure and verify which weights can update. section 9.2.1
Isolating an expert
An experiment freezes Expert 1 as well as Expert 0, then trains only Expert 2 tail layers and the router to compare routing and expert identity. section 9.5




