An expert is an independently parameterized network component in a mixture-of-experts (MoE) model. Experts process inputs separately; a router assigns weights based on each token’s hidden state, and the forward pass combines their outputs. Experts may specialize, but specialization need not follow domain boundaries. section 9.2 section 9.4.2
In a Transformer MoE, an expert can be an MLP replacing part of the original feed-forward path. Soft routing computes a weighted mixture of experts. Top-k routing instead selects a fixed number per token and renormalizes their weights, allowing unselected experts to be skipped for that token. The router uses contextual hidden states, so the same token can be routed differently in different contexts. section 9.3.1 section 9.6
Figure 9.4 Two views of the same top-k routing decision. Left: the router produces one logit score per expert for each token; the two highest scores per row (highlighted) are selected. Right: the same assignments reorganized by expert, showing which tokens each expert will process and their renormalized weights — for each token, those weights sum to 1 across its assigned experts. The example uses four experts rather than two so that top_k = 2 leaves half the experts idle per token. The loop in TrainableTopKMoE.forward performs exactly this regrouping.
Follow how token-level top-k selections are regrouped by expert and combined with renormalized weights. section 9.3.1
A practical two-expert construction uses sparse upcycling:
- Wrap selected original MLPs in MoE blocks. Keep each original MLP as the general expert and deep-copy it to initialize a separately trainable clinical expert. section 9.2.1
- Add a linear router from hidden-state size to two scores. Freeze the rest of the model, then train the clinical expert and router on clinical and general prompts. section 9.2.1
- Inspect per-token routing weights and test both general responses and clinical schema compliance. A routing preference alone does not establish task quality. section 9.2.2
Listing 9.1 Trainable2ExpertMoE
class Trainable2ExpertMoE(nn.Module):
def __init__(self, original_mlp, hidden_size):
super().__init__()
target_dtype = next(original_mlp.parameters()).dtype
target_device = next(original_mlp.parameters()).device
self.expert_general = original_mlp #A
self.expert_clinical = copy.deepcopy(original_mlp) #B
self.router = nn.Linear(hidden_size, 2,
bias=False,
dtype=target_dtype,
device=target_device) #C
self.last_routing = None
def forward(self, hidden_states):
router_logits = self.router(hidden_states)
if self.training:
noise = torch.randn_like(router_logits) * 0.35 #1
router_logits = router_logits + noise
routing_weights = F.softmax(
router_logits.float(), dim=-1
).to(hidden_states.dtype) #2
out_general = self.expert_general(hidden_states)
out_clinical = self.expert_clinical(hidden_states)
# Soft routing: each token receives a weighted blend of both experts
output = out_general * routing_weights[..., 0].unsqueeze(-1)
[CA]+ out_clinical * routing_weights[..., 1].unsqueeze(-1) #3
self.last_routing = routing_weights.detach() #D
return output
Inspect the two-expert module, including the deep copy, router initialization, and forward-pass setup. section 9.2.1
Clinical extraction expert
In SmolLM2-1.7B-Instruct, MLPs in layers 13–23 are upcycled into frozen general and trainable clinical experts; attention remains unchanged. On a 40-sample clinical test set, the soft-routed model reaches 97.5% schema compliance. On irrelevant notes, compliance rises from 58.3% for the base model to 91.7%; a general knowledge prompt also receives a coherent answer. section 9.2 section 9.2.2
An alternative is to transplant MLPs from an already fine-tuned model and initially train only the router. This is cheaper to train, but the transplanted MLPs may have adapted to different attention layers and need further adjustment. In one router-only test, two-expert soft routing achieved 90.0% clinical schema compliance; the irrelevant-note category fell to 66.7%. section 9.3 section 9.3.2
More experts do not guarantee clean domain separation. Mixtral 8x7B selects two of eight experts per token; its observed routing showed syntactic patterns rather than clear domain specialization. A three-expert exercise tests general, clinical, and code experts by varying training depth and freezing, then comparing routing across layers and prompts. section 9.4.2 section 9.5




