Hard routing selects experts for each token rather than continuously mixing all candidate expert outputs. In a two-expert mixture of experts (MoE), the router makes a binary choice and only the selected expert processes the token. This selective activation offers a path to greater total model capacity without inference cost growing in the same proportion, although it does not by itself establish a measured speedup. section 9.2.2 section 9.2.3
The router acts on contextual hidden states, so the same token can be assigned differently in different contexts. Soft and hard routing can use the same trained expert and router weights: switching the inference forward pass requires no second training run. section 9.6 section 9.2.3
A two-expert hard inference pass can be implemented as follows:
- Compute router logits from hidden states and take argmax over the expert dimension to choose expert 0 or 1 for each token. section 9.2.3
- Allocate an output tensor with the same shape as the hidden states. Build one Boolean mask for each expert; each token belongs to exactly one mask. section 9.2.3
- For each nonempty mask, run the corresponding expert on its selected hidden states and place the results back at their original positions. section 9.2.3
Listing 9.8 Trainable2ExpertMoE hard inference
class Trainable2ExpertMoE(nn.Module):
def __init__(self, original_mlp, hidden_size):
... #A
def forward(self, hidden_states):
router_logits = self.router(hidden_states)
routing_weights = F.softmax(
router_logits.float(),
dim=-1).to(hidden_states.dtype)
if self.training:
.... #A
else:
expert_choice = torch.argmax(
router_logits,
dim=-1) #B
output = torch.zeros_like(hidden_states)
mask_general = (expert_choice == 0)
mask_clinical = (expert_choice == 1)
if mask_general.any():
output[mask_general] = self.expert_general(
hidden_states[mask_general]) #C
if mask_clinical.any():
output[mask_clinical] = self.expert_clinical(
hidden_states[mask_clinical]) #C
hard_decisions = F.one_hot(expert_choice, num_classes=2).float()
self.last_routing = hard_decisions.detach()
return output
The evaluation branch uses argmax, masks, and indexed assignment to route tokens to two experts. section 9.2.3
To replace existing MoE blocks, construct blocks with the hard-routing forward method and copy the trained clinical expert and router state into them. The general expert is initialized from the existing general expert. Inspect per-token routing afterward to verify that the decisions are one-hot. section 9.2.3
General and clinical prompts
In a layer-23 inspection, a general-knowledge prompt routed every displayed token to the general expert. A clinical prompt routed a mixture of tokens to both experts, including “chronic” to the clinical expert. These are individual routing observations, not a guarantee that every clinical term selects the clinical expert. section 9.2.3
Routing can also change generated behavior. On one general-knowledge prompt, the hard model produced a shorter answer than the base and soft models; this single prompt is only a control test. Across five general benchmarks, both MoE variants improved over the base in four and declined slightly on PIQA. The small movements do not support definitive conclusions. section 9.2.4
